## Summary
Main ticket: https://github.com/elastic/kibana/issues/166545
This PR integrates the shared-ux `DefaultNavigation` component in the
Security Solution project for serverless.
These changes do not replace the original Security left navigation yet,
which is still the navigation component displayed by default. In order
to render the shared-ux `DefaultNavigation` this experimental flag
should be enabled:
`xpack.securitySolutionServerless.enableExperimental:
['platformNavEnabled']`
<img width="223" alt="Captura de pantalla 2023-09-25 a les 14 00 49"
src="91f247ce-ad9c-4093-a0f7-dbca63164b4a">
## Implementation
- Security navigation is still the default. Please enable the
`platformNavEnabled` experimental flag to render the shared navigation.
- We have two different formatters from the security navigation links
config to the navigation tree required in serverless:
- ChromeNavigationTree: registered directly to the serverless plugin for
the breadcrumbs to work when the navigation is customized with the
Security-specific nav. It will be removed after the migration to the
shared nav.
- NavigationTree: the format the shared nav uses, it already registers
the chromeNavigationTree for the breadcrumbs to the serverless plugin by
itself.
- Security plugin `deepLinks` needed to be formatted differently to make
this shared navigation work, since the `navLinkStatus: hidden` prevents
the links from being processed and displayed, this has been solved via
the `setDeepLinksFormatter` exposed on the plugin setup contract.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Part of #167467.
Adds support for text fields in log pattern analysis. Text fields will
be analysed using log categorization, similar to log pattern analysis.
Significant log patterns will be identified using the `chi2test`
package, similar to how we detect data drifts.
## Summary
This PR refactors the `assistantLangChain` code feature flag introduced
in https://github.com/elastic/kibana/pull/164908, to be a UI feature
toggle that users can enable/disable via the `Knowledge Base` assistant
advanced settings.
Left image shows the feature disabled, and the right image shows the
feature partly enabled. If ELSER is configured, the UI will attempt to
install all resources automatically for a one-click UX, however if ELSER
is not configured, or there are failures, the user can manually enable
the Knowledge Base or ES|QL base documentation:
<p align="center">
<img width="400"
src="be85522e-b2f5-4a39-9f0e-d359424caf37"
/> <img width="400"
src="d901c4f8-2184-4fb7-8c59-f2ff877118b9"
/>
</p>
Also, since this code feature flag was shared with the model evaluator
experimental feature, a `modelEvaluatorEnabled` flag has been plumbed to
fully decouple the two settings. Now _only the_ model evaluator is
enabled when setting security Solution Advanced setting:
```
xpack.securitySolution.enableExperimental: ['assistantModelEvaluation']
```
and the previous `assistantLangChain` code feature flag is now enabled
by simply toggling on the Knowledge Base in the settings shown above.
> [!NOTE]
> Even if ELSER isn't configured, and the knowledge base/docs aren't
setup, if the Knowledge Base is enabled, the LangChain code path will
still be enabled as intended, but we can change this behavior if testing
shows this is not ideal.
### Checklist
Delete any items that are not applicable to this PR.
- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)added
to match the most common scenarios
- [X] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
# [Security Solution] New Data Quality dashboard `Same family` category
This PR introduces a new `Same family` category to the [Data Quality dashboard](https://www.elastic.co/guide/en/security/current/data-quality-dash.html).
Fields with the `same family` tag that were previously counted in the `Incompatible fields` category are now counted in the new `Same family` category, per the annotated screenshot below:
_Above - Left: previously, fields with the `same family` tag were counted as `Incompatible fields`, Right: in this PR, those fields are counted in the new `Same family` category_
The annotations on the _Right_ side of the screenshot above highlight (in this example):
- The total `Incompatible fields` count was reduced
- Some patterns, like `auditbeat-*` have significant reductions
- The `Incompatible fields` count was reduced by one field
- The new `Same family` category has one field
- The `agent.type` field moved from the `Incompatible fields` category to the `Same family` category
## Details
### The new `Same family` category
Fields with mappings in the same [family](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html) have exactly the same search behavior as the type specified by ECS, but may have different space usage or performance characteristics.
The color of the `Same family` category is the same as the `Custom fields` category in badges and charts, per the screenshot below:
_Above: The `Same family` tab's badge and chart legend color is the same as the `Custom fields` category_
### The new `Same family` tab
This PR introduces a new `Same family` tab, as shown in the screenshot below:
_Above: The new `Same family` tab is selected_
In the screenshot above:
- The callout includes a description of fields in the same family (moved from the `Incompatible fields` tab)
- The `constant_keyword` text, yellow in previous versions (when it appered in the `Incompatible fields` tab), is blue
- Only one action, `Copy to clipboard` is available in the `Same family` tab. The remaining text in this _Details_ section is the markdown copied to the clipboard for the example above:
### auditbeat-custom-index-1
| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | auditbeat-custom-index-1 | 2 (0.0%) | 3 | `unmanaged` | 13.1KB |
### **Incompatible fields** `3` **Same family** `1` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `10`
#### 1 Same family field mapping
This field is defined by the Elastic Common Schema (ECS), version 8.6.1, but its mapping type doesn't exactly match.
Fields with mappings in the same family have exactly the same search behavior as the type specified by ECS, but may have different space usage or performance characteristics.
#### Same family field mappings - auditbeat-custom-index-1
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| agent.type | `keyword` | `constant_keyword` `same family` |
## Desk testing
1) Navigate to `Dev Tools` > `Console`
2) Execute the queries below:
<details>
<summary>Queries to create the `auditbeat-custom-index-1` example in this PR description</summary>
```
DELETE auditbeat-custom-index-1
PUT auditbeat-custom-index-1
PUT auditbeat-custom-index-1/_mapping
{
"properties": {
"@timestamp": {
"type": "date"
},
"agent.type": {
"type": "constant_keyword"
},
"event.category": {
"type": "constant_keyword"
}
}
}
POST auditbeat-custom-index-1/_doc
{
"@timestamp": "2023-02-06T09:41:49.668Z",
"host": {
"name": "foo"
},
"event": {
"category": "an_invalid_category"
},
"some.field": "this",
"source": {
"port": 90210,
"ip": "10.1.2.3"
}
}
POST auditbeat-custom-index-1/_doc
{
"@timestamp": "2023-02-06T09:42:22.123Z",
"host": {
"name": "bar"
},
"event": {
"category": "an_invalid_category"
},
"some.field": "space",
"source": {
"port": 867,
"ip": "10.9.8.7"
}
}
```
</details>
3) Navigate to `Security` > `Dashboards` > `Data Quality`
4) Expand the `auditbeat-custom-index-1` index
**Expected results**
- The sum of the category badge counts, `3 + 1 + 4 + 2`, equals the total number of fields in the `All fields` [`10`] category:
- `Incompatible fields` [`3`]
- `Same family` [`1`]
- `Custom fields` [`4`]
- `ECS compliant fields` [`2`]
- The `Incompatible fields` callout title is `3 incompatible fields`
- The `Incompatible fields` callout does NOT include a description of same family fields
- The `Incompatible fields` tab displays two fields with incompatible mappings
- `host.name` (`keyword` vs `text`)
- `source.ip` (`ip` vs `text`)
- The `Incompatible fields` tab displays one field with incompatible field values
- `event.category` (`an_invalid_category`)
5) Click the `Summary` tab
**Expected results**
- The `Summary` tab is focused
- The chart legend includes a `Same family` entry, with a count of `1`
- The `Same family` tab's badge color is the same as the `Custom fields` category
- The `Same family` chart legend color is the same as the `Custom fields` category
6) Click the `Same family` legend item
**Expected results**
- The `Same family` tab is focused
- One field, `agent.type`, is displayed
- The `constant_keyword` Index mapping type is blue
- The `same family` badge is yellow
7) Click `Copy to clipboard`
**Expected result**
- The expected markdown is copied to the clipboard:
```
### auditbeat-custom-index-1
| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | auditbeat-custom-index-1 | 2 (0.0%) | 3 | `unmanaged` | 12.9KB |
### **Incompatible fields** `3` **Same family** `1` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `10`
#### 1 Same family field mapping
This field is defined by the Elastic Common Schema (ECS), version 8.6.1, but its mapping type doesn't exactly match.
Fields with mappings in the same family have exactly the same search behavior as the type specified by ECS, but may have different space usage or performance characteristics.
#### Same family field mappings - auditbeat-custom-index-1
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| agent.type | `keyword` | `constant_keyword` `same family` |
```
## What this PR changes
Follow up of elastic/kibana/pull/164107/
For serverless ES/Kibana, it gates exception list API for endpoint
exceptions and restricts endpoint exceptions tab on Endpoint Security
rule details based on project PLIs. If no endpoint PLIs, endpoint
exceptions should not be accessible.
- [x] Add upselling to `app/security/exceptions/details/endpoint_list`
page
- [ ] Tests (WIP) - in a follow up PR
### How to review
Best to follow along commits for a code review. Below are details to
manually test the changes.
- Setup for _Servlerless_
- Run `yarn es serverless --kill --clean --license trial -E
xpack.security.authc.api_key.enabled=true` on a terminal window to start
ES.
- Copy `config/serverless.security.yml` to
`config/serverless.security.dev.yml`
- Run `yarn serverless-security --no-base-path` on another terminal
window to start kibana in serverless mode
- Log in using `serverless_security` user.
### Tests (Serverless)
This needs to be tested with a custom user/role and not
`elastic_serverless` which has `superuser` role.
1. ### PLI configs
`{ product_line: 'security', product_tier: 'essentials' }` or `{
product_line: 'security', product_tier: 'complete' }`
and
`{ product_line: 'endpoint', product_tier: 'essentials' }` or `{
product_line: 'endpoint', product_tier: 'complete' }`
- #### UX
1. Navigate to Rules via `http://localhost:5601/app/security/rules/`.
Click on `Add Elastic rules`.
2. Select and add `Endpoint Security` rule.
3. Click `Endpoint Security` and navigate to the rules details page, and
you should see `Endpoint exceptions` tab. The tabs visible are `Alerts`,
`Endpoint exceptions`, `Rule exceptions`, `Execution results`.
4. Navigate to Rules>Shared Exception Lists > Endpoint Security
Exception List via `app/security/exceptions/details/endpoint_list` and
you should be able to see the page with any added endpoint exceptions.
- #### API requests (with user `serverless_security`)
1. should get a status `200` on`POST api/exception_lists/items`
2. should get a status `200` on `POST
api/exception_lists/_export?id=endpoint_list&list_id=endpoint_list&namespace_type=agnostic&include_expired_exceptions=true`
3. should get a status `200` on `PUT api/exception_lists/items`
5. should get a status `200` on `DELETE api/exception_lists/items`
6. should get a status `200` on `GET
api/exception_lists/items/_find?list_id=endpoint_list&namespace_type=agnostic`
2. ### PLI configs
`{ product_line: 'security', product_tier: 'essentials' }` or `{
product_line: 'security', product_tier: 'complete' }`
- #### UX
1. Navigate to Rules via `http://localhost:5601/app/security/rules/`.
Click on `Add Elastic rules`.
2. Select and add `Endpoint Security` rule.
3. Click `Endpoint Security` and navigate to the rules details page, and
you should not see `Endpoint exceptions` tab. The only tabs visible are
`Alerts`, `Rule exceptions`, `Execution results`.
4. Navigate to Rules>Shared Exception Lists > Endpoint Security
Exception List via `app/security/exceptions/details/endpoint_list` and
you should see an upsell message.
- #### API requests
1. should get a status `403` on`POST api/exception_lists/items`
2. should get a status `403` on `POST
api/exception_lists/_export?id=endpoint_list&list_id=endpoint_list&namespace_type=agnostic&include_expired_exceptions=true`
3. should get a status `403` on `PUT api/exception_lists/items`
6. should get a status `403` on `DELETE api/exception_lists/items`
7. should get a status `403` on `GET
api/exception_lists/items/_find?list_id=endpoint_list&namespace_type=agnostic`
---
**Flaky FTRs**
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3248https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3255
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR adds tracking for Log Rate Analysis and Log Pattern Analysis
endpoints for AIOps.
- tracks type of analysis and source (where the analysis is being run
from)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR introduces a new `internal/elastic_assistant/evaluate` route and
`Evaluation` Advanced Setting within the Assistant for benchmarking and
testing models, agents, and other aspects of the Assistant
configuration.
Enable via the `assistantModelEvaluation` experimental feature in your
`kibana.dev.yml` (and better add `discoverInTimeline` for good measure
as well! :)
> xpack.securitySolution.enableExperimental:
['assistantModelEvaluation', 'discoverInTimeline']
Then access from within the `Advanced Settings` modal in the Assistant.
To use, first select your Connectors/Models, then corresponding Agent
configurations, then what model you would like to use for final
evaluation, the evaluation type, and if `custom`, you can specify the
evaluation prompt that is sent off to the evaluator model. Finally,
specify the `dataset`, and `output index` that the results should be
written to, then click `Perform evaluation`.
Sample datasets can be found in
`x-pack/plugins/elastic_assistant/server/lib/model_evaluator/datasets`,
and include:
* `esql_dataset.json`
* `query_dataset.json`
* `security_labs.json`
* `security_questions_dataset.json`
<p align="center">
<img width="500"
src="99f8e764-34bc-4eb7-bbd8-7038ab72117b"
/>
</p>
<p align="center">
<img width="500"
src="f48f91dc-45da-4cd6-9dc7-cb88105668b2"
/>
</p>
### Checklist
Delete any items that are not applicable to this PR.
- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
The `chi2test` utils so fare were only used within data comparison view.
We plan to use it with other plugins, so moving it so a separate package
in this PR. `SIGNIFICANCE_LEVELS` was updated to include some more
digits.
Removed `isServerless` flag which lived in our global context and had to
be passed about to the various components which create their own version
of the context using `getMlGlobalServices`
This PR adds a new context which contains flags for all of the features
which can be toggled when in serverless mode.
Flags added:
```
showNodeInfo
showMLNavMenu
showLicenseInfo
isADEnabled
isDFAEnabled
isNLPEnabled
```
The enabled features flags are now read from the config file client
side, rather than using capabilities.
Additional changes:
- Changes the wording of the awaiting ML node callout in serverless.
- In the search project, the default ML page is the trained models list
and not Overview
- Reenables the Memory Usage page for all projects
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR adds data drift detection workflow from Trained models to Data
comparison view. It also renames Data comparison to Data Drift.
**From the new map view in Trained model list:**
- Clicking on the index icon in the map view will give an option/action
to Analyze data drift
a68163ab-8a83-4378-8cf3-ea49f4480a06
- If model has detected related indices, it will also give an option to
Analyze data drift in the Transform actions
**From the data comparison/drift page:**
- Default screen with list of available data views and saved search will
be shown
<img width="1470" alt="Screen Shot 2023-09-07 at 00 22 01"
src="db13b8b7-9d90-4220-b03e-9f9d12ab53e9">
- But can also customize index patterns for the data sets to analyze.
Upon 'analyzing', a new data view will be created if needed (either
permanently or temporarily).
<img width="1271" alt="Screen Shot 2023-08-29 at 16 56 57"
src="e000e920-162b-4369-8762-70b6244e50e7">
<img width="1470" alt="Screen Shot 2023-09-07 at 00 22 49"
src="6577a530-c3b0-4ab9-95e4-d1d8fd1c9f0a">
- If there exists a data view with exact combination of index patterns
and time field, it will use that data view
- If there exists a data view with the same index patterns but different
time field, it will create a new data view with name
`{referencePattern},{comparisonPattern}-{timeField}`
- If no data view exists that matches, it will create a new data view
with name `{referencePattern},{comparisonPattern}`
## For reviewers:
- **appex-sharedux**: [Small change in the exported type interface for
BaseSavedObjectFinder](https://github.com/elastic/kibana/pull/162853/files#diff-5e2e62df8aba5ac9445962bfa00eee933a386110d0a24dfe6ac0f300a796ccc3)
to correctly list `children` as an accepted prop. This prop which is
used for the `toolsRight`.
- **security-solution**: Renaming of `Data comparison` to `Data Drift`
## Tests:
[Flaky test suite runner with Data Drift
test](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3216#018accc2-d33b-4cd6-a178-589e6698b675)
... successful after 50 runs✅
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## [Security Solution] [Elastic AI Assistant] LangChain Agents and Tools integration for ES|QL query generation via ELSER
This PR integrates [LangChain](https://www.langchain.com/) [Agents](https://js.langchain.com/docs/modules/agents/) and [Tools](https://js.langchain.com/docs/modules/agents/tools/) with the [Elastic AI Assistant](https://www.elastic.co/blog/introducing-elastic-ai-assistant).
These abstractions enable the LLM to dynamically choose whether or not to query, via [ELSER](https://www.elastic.co/guide/en/machine-learning/current/ml-nlp-elser.html), an [ES|QL](https://www.elastic.co/blog/elasticsearch-query-language-esql) knowledge base. Context from the knowledge base is used to generate `ES|QL` queries, or answer questions about `ES|QL`.
Registration of the tool occurs in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts`:
```typescript
const tools: Tool[] = [
new ChainTool({
name: 'esql-language-knowledge-base',
description:
'Call this for knowledge on how to build an ESQL query, or answer questions about the ES|QL query language.',
chain,
}),
];
```
The `tools` array above may be updated in future PRs to include, for example, an `ES|QL` query validator endpoint.
### Details
The `callAgentExecutor` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts`:
1. Creates a `RetrievalQAChain` from an `ELSER` backed `ElasticsearchStore`, which serves as a knowledge base for `ES|QL`:
```typescript
// ELSER backed ElasticsearchStore for Knowledge Base
const esStore = new ElasticsearchStore(esClient, KNOWLEDGE_BASE_INDEX_PATTERN, logger);
const chain = RetrievalQAChain.fromLLM(llm, esStore.asRetriever());
```
2. Registers the chain as a tool, which may be invoked by the LLM based on its description:
```typescript
const tools: Tool[] = [
new ChainTool({
name: 'esql-language-knowledge-base',
description:
'Call this for knowledge on how to build an ESQL query, or answer questions about the ES|QL query language.',
chain,
}),
];
```
3. Creates an Agent executor that combines the `tools` above, the `ActionsClientLlm` (an abstraction that calls `actionsClient.execute`), and memory of the previous messages in the conversation:
```typescript
const executor = await initializeAgentExecutorWithOptions(tools, llm, {
agentType: 'chat-conversational-react-description',
memory,
verbose: false,
});
```
Note: Set `verbose` above to `true` to for detailed debugging output from LangChain.
4. Calls the `executor`, kicking it off with `latestMessage`:
```typescript
await executor.call({ input: latestMessage[0].content });
```
### Changes to `x-pack/packages/kbn-elastic-assistant`
A client side change was required to the assistant, because the response returned from the agent executor is JSON. This response is parsed on the client in `x-pack/packages/kbn-elastic-assistant/impl/assistant/api.tsx`:
```typescript
return assistantLangChain ? getFormattedMessageContent(result) : result;
```
Client-side parsing of the response only happens when then `assistantLangChain` feature flag is `true`.
## Desk testing
Set
```typescript
assistantLangChain={true}
```
in `x-pack/plugins/security_solution/public/assistant/provider.tsx` to enable this experimental feature in development environments.
Also (optionally) set `verbose` to `true` in the following code in ``x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts``:
```typescript
const executor = await initializeAgentExecutorWithOptions(tools, llm, {
agentType: 'chat-conversational-react-description',
memory,
verbose: true,
});
```
After setting the feature flag and optionally enabling verbose debugging output, you may ask the assistant to generate an `ES|QL` query, per the example in the next section.
### Example output
When the Elastic AI Assistant is asked:
```
From employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. "September 2019". Only show the query
```
it replies:
```
Here is the query to get the employee number and the formatted hire date for the 5 earliest employees by hire_date:
FROM employees
| KEEP emp_no, hire_date
| EVAL month_year = DATE_FORMAT(hire_date, "MMMM YYYY")
| SORT hire_date
| LIMIT 5
```
Per the screenshot below:
The `verbose: true` output from LangChain logged to the console reveals that the prompt sent to the LLM includes text like the following:
```
Assistant can ask the user to use tools to look up information that may be helpful in answering the users original question. The tools the human can use are:\\n\\nesql-language-knowledge-base: Call this for knowledge on how to build an ESQL query, or answer questions about the ES|QL query language.
```
along with instructions for "calling" the tool like a function.
The debugging output also reveals the agent selecting the tool, and returning results from ESLR:
```
[agent/action] [1:chain:AgentExecutor] Agent selected action: {
"tool": "esql-language-knowledge-base",
"toolInput": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.",
"log": "```json\n{\n \"action\": \"esql-language-knowledge-base\",\n \"action_input\": \"Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\"\n}\n```"
}
[tool/start] [1:chain:AgentExecutor > 4:tool:ChainTool] Entering Tool run with input: "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
[chain/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain] Entering Chain run with input: {
"query": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
}
[retriever/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 6:retriever:VectorStoreRetriever] Entering Retriever run with input: {
"query": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
}
[retriever/end] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 6:retriever:VectorStoreRetriever] [115ms] Exiting Retriever run with output: {
"documents": [
{
"pageContent": "[[esql-date_format]]\n=== `DATE_FORMAT`\nReturns a string representation of a date in the provided format. If no format\nis specified, the `yyyy-MM-dd'T'HH:mm:ss.SSSZ` format is used.\n\n[source,esql]\n----\nFROM employees\n| KEEP first_name, last_name, hire_date\n| EVAL hired = DATE_FORMAT(hire_date, \"YYYY-MM-dd\")\n----\n",
```
The documents containing `ES|QL` examples, retrieved from ELSER, are sent back to the LLM to answer the original question, per the abridged output below:
```
[llm/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain > 8:chain:LLMChain > 9:llm:ActionsClientLlm] Entering LLM run with input: {
"prompts": [
"Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.\n\n[[esql-date_format]]\n=== `DATE_FORMAT`\nReturns a string representation of a date in the provided format. If no format\nis specified, the `yyyy-MM-dd'T'HH:mm:ss.SSSZ` format is used.\n\n[source,esql]\n----\nFROM employees\n| KEEP first_name, last_name, hire_date\n| EVAL hired = DATE_FORMAT(hire_date, \"YYYY-MM-dd\")\n----\n\n\n[[esql-date_trunc]]\n=== `DATE_TRUNC`\nRounds down a date to the closest interval. Intervals can be expressed using the\n<<esql-timespan-literals,timespan literal syntax>>.\n\n[source,esql]\n----\nFROM employees\n| EVAL year_hired = DATE_TRUNC(1 year, hire_date)\n| STATS count(emp_no) BY year_hired\n| SORT year_hired\n----\n\n\n[[esql-from]]\n=== `FROM`\n\nThe `FROM` source command returns a table with up to 10,000 documents from a\ndata stream, index,
```
### Complete (verbose) LangChain output from the example
The following `verbose: true` output from LangChain below was produced via the example in the previous section:
```
[chain/start] [1:chain:AgentExecutor] Entering Chain run with input: {
"input": "\n\n\n\nFrom employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. \"September 2019\". Only show the query",
"chat_history": []
}
[chain/start] [1:chain:AgentExecutor > 2:chain:LLMChain] Entering Chain run with input: {
"input": "\n\n\n\nFrom employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. \"September 2019\". Only show the query",
"chat_history": [],
"agent_scratchpad": [],
"stop": [
"Observation:"
]
}
[llm/start] [1:chain:AgentExecutor > 2:chain:LLMChain > 3:llm:ActionsClientLlm] Entering LLM run with input: {
"prompts": [
"[{\"lc\":1,\"type\":\"constructor\",\"id\":[\"langchain\",\"schema\",\"SystemMessage\"],\"kwargs\":{\"content\":\"Assistant is a large language model trained by OpenAI.\\n\\nAssistant is designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, Assistant is able to generate human-like text based on the input it receives, allowing it to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.\\n\\nAssistant is constantly learning and improving, and its capabilities are constantly evolving. It is able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. Additionally, Assistant is able to generate its own text based on the input it receives, allowing it to engage in discussions and provide explanations and descriptions on a wide range of topics.\\n\\nOverall, Assistant is a powerful system that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether you need help with a specific question or just want to have a conversation about a particular topic, Assistant is here to assist. However, above all else, all responses must adhere to the format of RESPONSE FORMAT INSTRUCTIONS.\",\"additional_kwargs\":{}}},{\"lc\":1,\"type\":\"constructor\",\"id\":[\"langchain\",\"schema\",\"HumanMessage\"],\"kwargs\":{\"content\":\"TOOLS\\n------\\nAssistant can ask the user to use tools to look up information that may be helpful in answering the users original question. The tools the human can use are:\\n\\nesql-language-knowledge-base: Call this for knowledge on how to build an ESQL query, or answer questions about the ES|QL query language.\\n\\nRESPONSE FORMAT INSTRUCTIONS\\n----------------------------\\n\\nOutput a JSON markdown code snippet containing a valid JSON object in one of two formats:\\n\\n**Option 1:**\\nUse this if you want the human to use a tool.\\nMarkdown code snippet formatted in the following schema:\\n\\n```json\\n{\\n \\\"action\\\": string, // The action to take. Must be one of [esql-language-knowledge-base]\\n \\\"action_input\\\": string // The input to the action. May be a stringified object.\\n}\\n```\\n\\n**Option #2:**\\nUse this if you want to respond directly and conversationally to the human. Markdown code snippet formatted in the following schema:\\n\\n```json\\n{\\n \\\"action\\\": \\\"Final Answer\\\",\\n \\\"action_input\\\": string // You should put what you want to return to use here and make sure to use valid json newline characters.\\n}\\n```\\n\\nFor both options, remember to always include the surrounding markdown code snippet delimiters (begin with \\\"```json\\\" and end with \\\"```\\\")!\\n\\n\\nUSER'S INPUT\\n--------------------\\nHere is the user's input (remember to respond with a markdown code snippet of a json blob with a single action, and NOTHING else):\\n\\n\\n\\n\\n\\nFrom employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. \\\"September 2019\\\". Only show the query\",\"additional_kwargs\":{}}}]"
]
}
[llm/end] [1:chain:AgentExecutor > 2:chain:LLMChain > 3:llm:ActionsClientLlm] [3.08s] Exiting LLM run with output: {
"generations": [
[
{
"text": "```json\n{\n \"action\": \"esql-language-knowledge-base\",\n \"action_input\": \"Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\"\n}\n```"
}
]
]
}
[chain/end] [1:chain:AgentExecutor > 2:chain:LLMChain] [3.09s] Exiting Chain run with output: {
"text": "```json\n{\n \"action\": \"esql-language-knowledge-base\",\n \"action_input\": \"Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\"\n}\n```"
}
[agent/action] [1:chain:AgentExecutor] Agent selected action: {
"tool": "esql-language-knowledge-base",
"toolInput": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.",
"log": "```json\n{\n \"action\": \"esql-language-knowledge-base\",\n \"action_input\": \"Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\"\n}\n```"
}
[tool/start] [1:chain:AgentExecutor > 4:tool:ChainTool] Entering Tool run with input: "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
[chain/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain] Entering Chain run with input: {
"query": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
}
[retriever/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 6:retriever:VectorStoreRetriever] Entering Retriever run with input: {
"query": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
}
[retriever/end] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 6:retriever:VectorStoreRetriever] [115ms] Exiting Retriever run with output: {
"documents": [
{
"pageContent": "[[esql-date_format]]\n=== `DATE_FORMAT`\nReturns a string representation of a date in the provided format. If no format\nis specified, the `yyyy-MM-dd'T'HH:mm:ss.SSSZ` format is used.\n\n[source,esql]\n----\nFROM employees\n| KEEP first_name, last_name, hire_date\n| EVAL hired = DATE_FORMAT(hire_date, \"YYYY-MM-dd\")\n----\n",
"metadata": {
"source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/functions/date_format.asciidoc"
}
},
{
"pageContent": "[[esql-date_trunc]]\n=== `DATE_TRUNC`\nRounds down a date to the closest interval. Intervals can be expressed using the\n<<esql-timespan-literals,timespan literal syntax>>.\n\n[source,esql]\n----\nFROM employees\n| EVAL year_hired = DATE_TRUNC(1 year, hire_date)\n| STATS count(emp_no) BY year_hired\n| SORT year_hired\n----\n",
"metadata": {
"source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/functions/date_trunc.asciidoc"
}
},
{
"pageContent": "[[esql-from]]\n=== `FROM`\n\nThe `FROM` source command returns a table with up to 10,000 documents from a\ndata stream, index, or alias. Each row in the resulting table represents a\ndocument. Each column corresponds to a field, and can be accessed by the name\nof that field.\n\n[source,esql]\n----\nFROM employees\n----\n\nYou can use <<api-date-math-index-names,date math>> to refer to indices, aliases\nand data streams. This can be useful for time series data, for example to access\ntoday's index:\n\n[source,esql]\n----\nFROM <logs-{now/d}>\n----\n\nUse comma-separated lists or wildcards to query multiple data streams, indices,\nor aliases:\n\n[source,esql]\n----\nFROM employees-00001,employees-*\n----\n",
"metadata": {
"source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/source_commands/from.asciidoc"
}
},
{
"pageContent": "[[esql-where]]\n=== `WHERE`\n\nUse `WHERE` to produce a table that contains all the rows from the input table\nfor which the provided condition evaluates to `true`:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=where]\n----\n\nWhich, if `still_hired` is a boolean field, can be simplified to:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereBoolean]\n----\n\n[discrete]\n==== Operators\n\nRefer to <<esql-operators>> for an overview of the supported operators.\n\n[discrete]\n==== Functions\n`WHERE` supports various functions for calculating values. Refer to\n<<esql-functions,Functions>> for more information.\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereFunction]\n----\n",
"metadata": {
"source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/processing_commands/where.asciidoc"
}
}
]
}
[chain/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain] Entering Chain run with input: {
"question": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.",
"input_documents": [
{
"pageContent": "[[esql-date_format]]\n=== `DATE_FORMAT`\nReturns a string representation of a date in the provided format. If no format\nis specified, the `yyyy-MM-dd'T'HH:mm:ss.SSSZ` format is used.\n\n[source,esql]\n----\nFROM employees\n| KEEP first_name, last_name, hire_date\n| EVAL hired = DATE_FORMAT(hire_date, \"YYYY-MM-dd\")\n----\n",
"metadata": {
"source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/functions/date_format.asciidoc"
}
},
{
"pageContent": "[[esql-date_trunc]]\n=== `DATE_TRUNC`\nRounds down a date to the closest interval. Intervals can be expressed using the\n<<esql-timespan-literals,timespan literal syntax>>.\n\n[source,esql]\n----\nFROM employees\n| EVAL year_hired = DATE_TRUNC(1 year, hire_date)\n| STATS count(emp_no) BY year_hired\n| SORT year_hired\n----\n",
"metadata": {
"source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/functions/date_trunc.asciidoc"
}
},
{
"pageContent": "[[esql-from]]\n=== `FROM`\n\nThe `FROM` source command returns a table with up to 10,000 documents from a\ndata stream, index, or alias. Each row in the resulting table represents a\ndocument. Each column corresponds to a field, and can be accessed by the name\nof that field.\n\n[source,esql]\n----\nFROM employees\n----\n\nYou can use <<api-date-math-index-names,date math>> to refer to indices, aliases\nand data streams. This can be useful for time series data, for example to access\ntoday's index:\n\n[source,esql]\n----\nFROM <logs-{now/d}>\n----\n\nUse comma-separated lists or wildcards to query multiple data streams, indices,\nor aliases:\n\n[source,esql]\n----\nFROM employees-00001,employees-*\n----\n",
"metadata": {
"source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/source_commands/from.asciidoc"
}
},
{
"pageContent": "[[esql-where]]\n=== `WHERE`\n\nUse `WHERE` to produce a table that contains all the rows from the input table\nfor which the provided condition evaluates to `true`:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=where]\n----\n\nWhich, if `still_hired` is a boolean field, can be simplified to:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereBoolean]\n----\n\n[discrete]\n==== Operators\n\nRefer to <<esql-operators>> for an overview of the supported operators.\n\n[discrete]\n==== Functions\n`WHERE` supports various functions for calculating values. Refer to\n<<esql-functions,Functions>> for more information.\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereFunction]\n----\n",
"metadata": {
"source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/processing_commands/where.asciidoc"
}
}
],
"query": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
}
[chain/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain > 8:chain:LLMChain] Entering Chain run with input: {
"question": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.",
"query": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.",
"context": "[[esql-date_format]]\n=== `DATE_FORMAT`\nReturns a string representation of a date in the provided format. If no format\nis specified, the `yyyy-MM-dd'T'HH:mm:ss.SSSZ` format is used.\n\n[source,esql]\n----\nFROM employees\n| KEEP first_name, last_name, hire_date\n| EVAL hired = DATE_FORMAT(hire_date, \"YYYY-MM-dd\")\n----\n\n\n[[esql-date_trunc]]\n=== `DATE_TRUNC`\nRounds down a date to the closest interval. Intervals can be expressed using the\n<<esql-timespan-literals,timespan literal syntax>>.\n\n[source,esql]\n----\nFROM employees\n| EVAL year_hired = DATE_TRUNC(1 year, hire_date)\n| STATS count(emp_no) BY year_hired\n| SORT year_hired\n----\n\n\n[[esql-from]]\n=== `FROM`\n\nThe `FROM` source command returns a table with up to 10,000 documents from a\ndata stream, index, or alias. Each row in the resulting table represents a\ndocument. Each column corresponds to a field, and can be accessed by the name\nof that field.\n\n[source,esql]\n----\nFROM employees\n----\n\nYou can use <<api-date-math-index-names,date math>> to refer to indices, aliases\nand data streams. This can be useful for time series data, for example to access\ntoday's index:\n\n[source,esql]\n----\nFROM <logs-{now/d}>\n----\n\nUse comma-separated lists or wildcards to query multiple data streams, indices,\nor aliases:\n\n[source,esql]\n----\nFROM employees-00001,employees-*\n----\n\n\n[[esql-where]]\n=== `WHERE`\n\nUse `WHERE` to produce a table that contains all the rows from the input table\nfor which the provided condition evaluates to `true`:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=where]\n----\n\nWhich, if `still_hired` is a boolean field, can be simplified to:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereBoolean]\n----\n\n[discrete]\n==== Operators\n\nRefer to <<esql-operators>> for an overview of the supported operators.\n\n[discrete]\n==== Functions\n`WHERE` supports various functions for calculating values. Refer to\n<<esql-functions,Functions>> for more information.\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereFunction]\n----\n"
}
[llm/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain > 8:chain:LLMChain > 9:llm:ActionsClientLlm] Entering LLM run with input: {
"prompts": [
"Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.\n\n[[esql-date_format]]\n=== `DATE_FORMAT`\nReturns a string representation of a date in the provided format. If no format\nis specified, the `yyyy-MM-dd'T'HH:mm:ss.SSSZ` format is used.\n\n[source,esql]\n----\nFROM employees\n| KEEP first_name, last_name, hire_date\n| EVAL hired = DATE_FORMAT(hire_date, \"YYYY-MM-dd\")\n----\n\n\n[[esql-date_trunc]]\n=== `DATE_TRUNC`\nRounds down a date to the closest interval. Intervals can be expressed using the\n<<esql-timespan-literals,timespan literal syntax>>.\n\n[source,esql]\n----\nFROM employees\n| EVAL year_hired = DATE_TRUNC(1 year, hire_date)\n| STATS count(emp_no) BY year_hired\n| SORT year_hired\n----\n\n\n[[esql-from]]\n=== `FROM`\n\nThe `FROM` source command returns a table with up to 10,000 documents from a\ndata stream, index, or alias. Each row in the resulting table represents a\ndocument. Each column corresponds to a field, and can be accessed by the name\nof that field.\n\n[source,esql]\n----\nFROM employees\n----\n\nYou can use <<api-date-math-index-names,date math>> to refer to indices, aliases\nand data streams. This can be useful for time series data, for example to access\ntoday's index:\n\n[source,esql]\n----\nFROM <logs-{now/d}>\n----\n\nUse comma-separated lists or wildcards to query multiple data streams, indices,\nor aliases:\n\n[source,esql]\n----\nFROM employees-00001,employees-*\n----\n\n\n[[esql-where]]\n=== `WHERE`\n\nUse `WHERE` to produce a table that contains all the rows from the input table\nfor which the provided condition evaluates to `true`:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=where]\n----\n\nWhich, if `still_hired` is a boolean field, can be simplified to:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereBoolean]\n----\n\n[discrete]\n==== Operators\n\nRefer to <<esql-operators>> for an overview of the supported operators.\n\n[discrete]\n==== Functions\n`WHERE` supports various functions for calculating values. Refer to\n<<esql-functions,Functions>> for more information.\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereFunction]\n----\n\n\nQuestion: Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\nHelpful Answer:"
]
}
[llm/end] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain > 8:chain:LLMChain > 9:llm:ActionsClientLlm] [2.23s] Exiting LLM run with output: {
"generations": [
[
{
"text": "FROM employees\n| KEEP emp_no, hire_date\n| EVAL month_year = DATE_FORMAT(hire_date, \"MMMM YYYY\")\n| SORT hire_date\n| LIMIT 5"
}
]
]
}
[chain/end] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain > 8:chain:LLMChain] [2.23s] Exiting Chain run with output: {
"text": "FROM employees\n| KEEP emp_no, hire_date\n| EVAL month_year = DATE_FORMAT(hire_date, \"MMMM YYYY\")\n| SORT hire_date\n| LIMIT 5"
}
[chain/end] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain] [2.23s] Exiting Chain run with output: {
"text": "FROM employees\n| KEEP emp_no, hire_date\n| EVAL month_year = DATE_FORMAT(hire_date, \"MMMM YYYY\")\n| SORT hire_date\n| LIMIT 5"
}
[chain/end] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain] [2.35s] Exiting Chain run with output: {
"text": "FROM employees\n| KEEP emp_no, hire_date\n| EVAL month_year = DATE_FORMAT(hire_date, \"MMMM YYYY\")\n| SORT hire_date\n| LIMIT 5"
}
[tool/end] [1:chain:AgentExecutor > 4:tool:ChainTool] [2.35s] Exiting Tool run with output: "FROM employees
| KEEP emp_no, hire_date
| EVAL month_year = DATE_FORMAT(hire_date, "MMMM YYYY")
| SORT hire_date
| LIMIT 5"
[chain/start] [1:chain:AgentExecutor > 10:chain:LLMChain] Entering Chain run with input: {
"input": "\n\n\n\nFrom employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. \"September 2019\". Only show the query",
"chat_history": [],
"agent_scratchpad": [
{
"lc": 1,
"type": "constructor",
"id": [
"langchain",
"schema",
"AIMessage"
],
"kwargs": {
"content": "```json\n{\n \"action\": \"esql-language-knowledge-base\",\n \"action_input\": \"Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\"\n}\n```",
"additional_kwargs": {}
}
},
{
"lc": 1,
"type": "constructor",
"id": [
"langchain",
"schema",
"HumanMessage"
],
"kwargs": {
"content": "TOOL RESPONSE:\n---------------------\nFROM employees\n| KEEP emp_no, hire_date\n| EVAL month_year = DATE_FORMAT(hire_date, \"MMMM YYYY\")\n| SORT hire_date\n| LIMIT 5\n\nUSER'S INPUT\n--------------------\n\nOkay, so what is the response to my last comment? If using information obtained from the tools you must mention it explicitly without mentioning the tool names - I have forgotten all TOOL RESPONSES! Remember to respond with a markdown code snippet of a json blob with a single action, and NOTHING else.",
"additional_kwargs": {}
}
}
],
"stop": [
"Observation:"
]
}
[llm/start] [1:chain:AgentExecutor > 10:chain:LLMChain > 11:llm:ActionsClientLlm] Entering LLM run with input: {
"prompts": [
"[{\"lc\":1,\"type\":\"constructor\",\"id\":[\"langchain\",\"schema\",\"SystemMessage\"],\"kwargs\":{\"content\":\"Assistant is a large language model trained by OpenAI.\\n\\nAssistant is designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, Assistant is able to generate human-like text based on the input it receives, allowing it to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.\\n\\nAssistant is constantly learning and improving, and its capabilities are constantly evolving. It is able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. Additionally, Assistant is able to generate its own text based on the input it receives, allowing it to engage in discussions and provide explanations and descriptions on a wide range of topics.\\n\\nOverall, Assistant is a powerful system that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether you need help with a specific question or just want to have a conversation about a particular topic, Assistant is here to assist. However, above all else, all responses must adhere to the format of RESPONSE FORMAT INSTRUCTIONS.\",\"additional_kwargs\":{}}},{\"lc\":1,\"type\":\"constructor\",\"id\":[\"langchain\",\"schema\",\"HumanMessage\"],\"kwargs\":{\"content\":\"TOOLS\\n------\\nAssistant can ask the user to use tools to look up information that may be helpful in answering the users original question. The tools the human can use are:\\n\\nesql-language-knowledge-base: Call this for knowledge on how to build an ESQL query, or answer questions about the ES|QL query language.\\n\\nRESPONSE FORMAT INSTRUCTIONS\\n----------------------------\\n\\nOutput a JSON markdown code snippet containing a valid JSON object in one of two formats:\\n\\n**Option 1:**\\nUse this if you want the human to use a tool.\\nMarkdown code snippet formatted in the following schema:\\n\\n```json\\n{\\n \\\"action\\\": string, // The action to take. Must be one of [esql-language-knowledge-base]\\n \\\"action_input\\\": string // The input to the action. May be a stringified object.\\n}\\n```\\n\\n**Option #2:**\\nUse this if you want to respond directly and conversationally to the human. Markdown code snippet formatted in the following schema:\\n\\n```json\\n{\\n \\\"action\\\": \\\"Final Answer\\\",\\n \\\"action_input\\\": string // You should put what you want to return to use here and make sure to use valid json newline characters.\\n}\\n```\\n\\nFor both options, remember to always include the surrounding markdown code snippet delimiters (begin with \\\"```json\\\" and end with \\\"```\\\")!\\n\\n\\nUSER'S INPUT\\n--------------------\\nHere is the user's input (remember to respond with a markdown code snippet of a json blob with a single action, and NOTHING else):\\n\\n\\n\\n\\n\\nFrom employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. \\\"September 2019\\\". Only show the query\",\"additional_kwargs\":{}}},{\"lc\":1,\"type\":\"constructor\",\"id\":[\"langchain\",\"schema\",\"AIMessage\"],\"kwargs\":{\"content\":\"```json\\n{\\n \\\"action\\\": \\\"esql-language-knowledge-base\\\",\\n \\\"action_input\\\": \\\"Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\\\"\\n}\\n```\",\"additional_kwargs\":{}}},{\"lc\":1,\"type\":\"constructor\",\"id\":[\"langchain\",\"schema\",\"HumanMessage\"],\"kwargs\":{\"content\":\"TOOL RESPONSE:\\n---------------------\\nFROM employees\\n| KEEP emp_no, hire_date\\n| EVAL month_year = DATE_FORMAT(hire_date, \\\"MMMM YYYY\\\")\\n| SORT hire_date\\n| LIMIT 5\\n\\nUSER'S INPUT\\n--------------------\\n\\nOkay, so what is the response to my last comment? If using information obtained from the tools you must mention it explicitly without mentioning the tool names - I have forgotten all TOOL RESPONSES! Remember to respond with a markdown code snippet of a json blob with a single action, and NOTHING else.\",\"additional_kwargs\":{}}}]"
]
}
[llm/end] [1:chain:AgentExecutor > 10:chain:LLMChain > 11:llm:ActionsClientLlm] [6.47s] Exiting LLM run with output: {
"generations": [
[
{
"text": "```json\n{\n \"action\": \"Final Answer\",\n \"action_input\": \"Here is the query to get the employee number and the formatted hire date for the 5 earliest employees by hire_date:\\n\\nFROM employees\\n| KEEP emp_no, hire_date\\n| EVAL month_year = DATE_FORMAT(hire_date, \\\"MMMM YYYY\\\")\\n| SORT hire_date\\n| LIMIT 5\"\n}\n```"
}
]
]
}
[chain/end] [1:chain:AgentExecutor > 10:chain:LLMChain] [6.47s] Exiting Chain run with output: {
"text": "```json\n{\n \"action\": \"Final Answer\",\n \"action_input\": \"Here is the query to get the employee number and the formatted hire date for the 5 earliest employees by hire_date:\\n\\nFROM employees\\n| KEEP emp_no, hire_date\\n| EVAL month_year = DATE_FORMAT(hire_date, \\\"MMMM YYYY\\\")\\n| SORT hire_date\\n| LIMIT 5\"\n}\n```"
}
[chain/end] [1:chain:AgentExecutor] [11.91s] Exiting Chain run with output: {
"output": "Here is the query to get the employee number and the formatted hire date for the 5 earliest employees by hire_date:\n\nFROM employees\n| KEEP emp_no, hire_date\n| EVAL month_year = DATE_FORMAT(hire_date, \"MMMM YYYY\")\n| SORT hire_date\n| LIMIT 5"
}
```
## Summary
Adds a shared service for elastic curated models. The first use case is
to provide a default/recommended ELSER version based on the hardware of
the current cluster.
#### Why?
In 8.11 we'll provide a platform-specific version of the ELSER v2
alongside the portable one. At the moment several solutions refer to
ELSER for download/inference purposes with a `.elser_model_1` constant.
Starting 8.11 the model ID will vary, so using the `ElastcModels`
service allows retrieving the recommended version of ELSER for the
current cluster without any changes by solution teams in future
releases. It is still possible to request an older version of the model
if necessary.
#### Implementation
- Adds a new Kibana API endpoint `/trained_models/model_downloads` that
provides a list of model definitions, with the `default` and
`recommended` flags.
- Adds a new Kibana API endpoint `/trained_models/elser_config` that
provides an ELSER configuration based on the cluster architecture.
- `getELSER` method is exposed from the plugin `setup` server-side as
part of our shared services and plugin `start` client-side.
### Checklist
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Advanced Settings to enable the Knowledge Base is currently behind the
same code toggle introduced in
https://github.com/elastic/kibana/pull/164908, please modify
`assistantLangChain` to be `true` to enable the Advanced Settings and
Knowledge Base options:
1dee16e061/x-pack/plugins/security_solution/public/assistant/provider.tsx (L55)
When the above modification is present, a new `Advanced Settings` UI is
available within the Assistant that enables the ability to turn on and
configure the Knowledge Base, and also add the requisite `ES|QL`
Knowledge Base resources.
<p align="center">
<img width="500"
src="3abab422-9a85-45f6-8b0b-854ed7383d1c"
/>
</p>
Once enabled, the Assistant will query the Knowledge Base using a
`ConversationalRetrievalQAChain`, by means of the ELSER specific
`ElasticsearchStore` LangChain `VectorStore` abstraction:
<p align="center">
<img width="500"
src="a529faab-948c-40f1-bc11-7a39b1766c7f"
/>
</p>
### Checklist
Delete any items that are not applicable to this PR.
- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
* TBD
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [X] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
issue: https://github.com/elastic/kibana/issues/163606
Fixes the bug on the Security tab navigation component scrolling to the
top of the page when clicked.
### Problem
As described in the React Router
[docs](https://v5.reactrouter.com/web/guides/scroll-restoration), all
browsers introduced scroll reset in the `history.pushState` API
([spec](https://majido.github.io/scroll-restoration-proposal/history-based-api.html#web-idl)),
which is the expected behavior on most of the regular navigation
actions, however in some cases such as the tabs navigation we don't want
the scrolling to be reset to the top after pushing the URL.
The spec also defines the `window.history.scrollRestoration` API to
configure this behavior ('manual' or 'auto'), but it does not work
correctly when using `react-route-dom` history object to navigate.
React Router v5 also provides a `ScrollRestoration` component to solve
this problem, but it only works with _"data router"_ type, we use
_"browser router"_ in Kibana, which is not supported by this tool.
There are other libraries that also solve this problem, implementing a
very similar approach, such as
[react-scroll-restoration](https://github.com/nanyang24/react-scroll-restoration).
### Solution
Instead of loading new external modules, I implemented the same approach
integrating it into our Security navigation tools:
The `navigateTo` function now accepts a new `restoreScroll` option prop,
when set to `true` it will prevent the scroll reset after navigating by
restoring the previous scroll position when it receives the event
triggered by the Browser.
In the case of using an old browser version that does not trigger the
scroll reset after the navigation, this change will have no implication
other than losing the first scroll event after the navigation, this will
be very hard to notice since many events are sent when a user scrolls
manually.
## Recordings
Before:
464f708a-f373-47b9-9826-a5cfa7473f5b
After:
46d99b12-69e8-410a-a700-ca9ef7f5c2a4
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
issue: https://github.com/elastic/kibana/issues/166436
### Bug
The bug was caused by the `AssistantOverlay` being rendered inside the
`SecuritySolutionPageWrapper`, which is a "styling" wrapper that is not
used by all the pages in the Security Solution application, the
serverless-specific pages for instance, use their own page wrapper for
styling.
### Changes
#### Fix
This PR fixes the bug by rendering the `AssistantOverlay` in the
`HomePage`, along with other similar components such as the `HelpMenu`
or the `TopValuesPopover`, which are components that need to be always
present in the application layout.
#### Cleaning
The assistant provider configuration logic has also been moved from the
`App` component to a new `provider.tsx` component in the
_public/assistant_ directory, this way we don't pollute the main
Security `App` component with assistant-specific logic.
The `isAssistantEnabled` prop has been removed from the `Assistant`
component. We don't need to check the availability and pass it by props
everywhere we use the assistant, it can be retrieved from the assistant
context.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
**ML Management page**
- Fixes general page loading issues.
- Ensures only enabled features are shown as tabs
- Ensures only jobs for enabled features can be exported and imported.
- Ensures only enabled features are listed in the saved object sync
output.
- On trained models tab:
- Only lists DFA models if NLP is disabled.
- Only lists non-DFA models if DFA is disabled.
**Anomaly Detection**
- Hides node information in anomaly detection jobs list.
- Hides the Exclude frozen data option in the Use full time range
selector in job wizards.
**Data frame analytics**
- Hides all node and license level information.
**Trained models**
- Only lists DFA models if NLP is disabled.
- Only lists non-DFA models if DFA is disabled.
- Hides all node and license level information.
- Hides DFA nodes
**Notifications and memory usage**
- Ensures only enabled features are mentioned. Including selectable
types in the search bar filters.
**Integrations with other plugins**
- Changes registration for integrations into other plugins so they only
happen if the relevant feature is enabled.
- Client side: UI actions, cases, embeddables, alerts, maps.
- Server side: Sample data sets, cases
**AIOPS**
- Hides the Exclude frozen data option in the Use full time range
selector on all pages
**Notes for non ML team reviewers**
**response-ops**
I've divided the
[persistable_state.ts](https://github.com/elastic/kibana/pull/163724#diff-e02dc0b6cb5b63965372b1f4a84d2287cba31a15ab525ab7983f02d09f23879f)
test into basic and trial version.
The ML cases attachments should only be registered if anomaly detection
is available in a trial or platinum license. This was a bug which I
noticed when making serverless changes.
**Observability**
I've made a few minor changes to the nav menu, fixing names of ML
features and adding the missing Change point detection AIOPs page.
**Security solution**
I've made a few minor changes to the nav menu, fixing names of ML
features and adding some missing ML features.
I think the icons being used will need to be revisited before release as
we have [official ML
icons](https://elastic.github.io/eui/#/display/icons#apps) but not for
every page. So we should probably either have new icons created or all
agree on which standard non-ML icons should be used for the ones which
are missing.
**Search**
The NLP feature is currently disabled in main, I believe this was an
attempt to stop ML anomaly detection alert rules from being registered.
I've reenabled NLP and changed the way we're registering the alerts.
They will now only be registered if the anomaly detection feature is
enabled.
Fixes https://github.com/elastic/kibana/issues/163372
This PR is a second part of Artifact Rollout Epic
(https://github.com/elastic/security-team/issues/3593) and it introduces
**Note** field as described in
https://github.com/elastic/security-team/issues/7238 ticket.
Changes:
1. Added a new SO, `policy-settings-protection-updates-note` which holds
non indexable `note` field of type **text** and reference to package
policy
2. Added `getPackagePolicyDeleteCallback` that cleans up SO on package
policy deletion
3. Exposed an API to interact with the SO (POST, GET) with
POST method accepting both creation and update if SO exists.
4. Integrated UI with API with `react-query` hooks.
Flow
359d59bd-1bde-417a-9449-467d08e81809
Read only access
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
…ed d4c manage policies link to the list of excluded links when user
does not have permission to read policies in security solution
## Summary
Fixes: https://github.com/elastic/kibana/issues/163562
## Summary
Fixes issue in descriptions of landing page icons which were always
displaying in white color. It only happens in Safari, other browsers
show the text in the correct color.
Fix: Remove misuse of `color` property in `EuiText` component
Before:
After:
### What this PR changes
branched from elastic/kibana/pull/163759
- Introduces new AppFeatures package `@kbn/security-solution-features`
with the common logic and `AppFeatureService` to apply offering specific
configurations for Security Solution features independently for
Serverless and ESS. This logic is replacing the earlier `AppFeatures` in
order to introduce new Kibana feature privileges for serverless PLIs so
that new Kibana privileges introduced for serverless PLIs do not
affect/show up as new Kibana feature privileges in ESS.
- Gates endpoint exceptions on alerts/rules based on serverless PLI
configurations. On serverless `Endpoint exceptions` should be
accessible/seen only on endpoint essentials/complete.
New AppFeatures logic architecture diagram:
**Note:** Corresponding API changes related to endpoint exceptions will
be in a new PR, along with the last set of UX changes for hiding the
`Endpoint exceptions` tab from the Rules details page.
### How to review
- Setup for _Servlerless_
- Run `yarn es snapshot` on a terminal window to start ES.
- Copy `config/serverless.security.yml` to
`config/serverless.security.dev.yml`
- Run `yarn serverless-security --no-base-path` on another terminal
window to start kibana in serverless mode
- Run `node
x-pack/plugins/security_solution/scripts/endpoint/endpoint_agent_emulator.js
--asSuperuser` on a new window and then select `1` to `Load Endoints`
and then `1` to `Run` the loader script. This will load some fake
agents/alerts data to test with.
### Tests (Serverless)
- with
`{ product_line: 'security', product_tier: 'essentials' }` or `{
product_line: 'security', product_tier: 'complete' }`
and
`{ product_line: 'endpoint', product_tier: 'essentials' }` or `{
product_line: 'endpoint', product_tier: 'complete' }`
1. Navigate to Rules>Shared exception lists via
`http://localhost:5601/app/security/exceptions`
2. Test that you can see `Endpoint Security Exception List` card on the
shared exception lists page.
3. Navigate to `Alerts` page via `app/security/alerts`, you should see
endpoint alerts. If not, then click on `Manage Rules` and then
disable/enable `Endpoint Security` rules. That should trigger alerts to
show up on the Alerts table.
4. Click on `View Details` button under `Actions` column. Once the
flyout is visible, click on `Take Action` and verify that `Add Endpoint
exception` is visible/enabled/clickable on the menu.
5. Click on `More actions` button under `Actions` column and verify that
`Add Endpoint exception` is visible/enabled/clickable on the menu.
6. Click on `Investigate in timeline` button under `Actions` column;
when the timeline view is visible and the alert item is displayed, click
on buttons mentioned in 4. and 5. above and verify the same.
7. Navigate to `Rules`>`DetectionRules`>`Endpoint Security` rule under
the `Rules` table. Select the `Alerts` tab.
8. Click and verify `View details`,`More actions` and `Investigate in
timeline` buttons same as in 4., 5., 6. above.
9. You should be able to see the `Endpoint exceptions` tab as well.
Click and verify that you can see the tab's content.
- with
`{ product_line: 'security', product_tier: 'essentials' }` or `{
product_line: 'security', product_tier: 'complete' }`
1. Edit `config/serverless.security.dev.yml` so that `endpoint` product
line item is commented out.
2. Test that you can not see `Endpoint Security Exception List` card on
the shared exception lists page.
3. Items 4. 5. 6. as above but the menu items should be disabled. This
can be verified with fake data only as with a real endpoint, endpoint
alerts are actually not visible at all.
### Tests (ESS)
On the ESS side, endpoint exceptions are not affected by this change and
work as usual based on index privileges.
---------
Co-authored-by: semd <sergi.massaneda@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: YulNaumenko <jo.naumenko@gmail.com>
Co-authored-by: Pablo Neves Machado <pablo.nevesmachado@elastic.co>
Co-authored-by: Pablo Machado <machadoum@gmail.com>
Resolves https://github.com/elastic/kibana/issues/159343
In this PR, I'm preparing the alerting rule task types for serverless by
defining an explicit task state schema for the framework level fields.
This schema is used to validate the task's state before saving but also
when reading. In the scenario an older Kibana node runs a task after a
newer Kibana node has stored additional task state, the unknown state
properties will be dropped. Additionally, this will prompt developers to
be aware that adding required fields to the task state is a breaking
change that must be handled with care. (see
https://github.com/elastic/kibana/issues/155764).
The PR also includes the following changes:
- Modifying the `@kbn/alerting-state-types` package so the types are
generated from `config-schema`, instead of `io-ts`. The schema is
re-used for exporting a new `stateSchemaByVersion` property.
- Removing `DateFromString` in favour of strings everywhere
(config-schema doesn't support this conversion)
- Add a v1 `up` migration that will ensure the types match the
TypeScript interface on any existing alerting task. The migration
assumes any type of data could exist and dropping parts that don't match
the type expectation. The TypeScript interface uses `schema.maybe()` in
a lot of places (as `io-ts` did), so safe if ever data gets dropped.
- Cleanup the `alerting/common/**` exports to reduce bundle size.
Because the `@kbn/alerting-state-types` package grew.
- Since the new TypeScript interfaces / types are `ReadOnly<...>`, I
created some `Mutable...` types for places that needed it (in order to
avoid code refactoring).
## To verify
Stack Monitoring:
- To make TypeScript happy with the new ReadOnly `RawAlertInstance`
type, I removed some redundant code and solved the issue.
Security Solution:
- Changes to the `alertInstanceFactoryStub` set the alert's date to a
`string` instead of a `Date` value. Note: The HTTP API response
converted `Date` objects to `string`, so the HTTP API response will look
the same with this change.
Response Ops:
- In a fresh Kibana install, create alerting rules and ensure they run.
- In a 8.9 version, create some rules, upgrade to this branch and ensure
they still run.
- Compare the `io-ts` definition with the new `config-schema`. They
should match 1:1.
- Look for ways the migration code could fail.
Note: The main changes are within the following areas:
- `x-pack/plugins/alerting/server/rule_type_registry.ts`
- `x-pack/packages/kbn-alerting-state-types/*`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes Preconfigured Connectors not working with the Assistant, and also
ensures default `model` from connector will be used first if available
(instead of defaulting to `gpt-3.5-turbo`).
<p align="center">
<img width="500"
src="637f5919-7560-40b0-a8db-681096e77ac0"
/>
</p>
Note how `Model` is not displayed even though this is an OpenAI
connector:
<p align="center">
<img width="500"
src="2c4bbe91-2851-48d7-8bfe-20e07db52155"
/>
</p>
Additionally, resolves issue with Detection Rule Assistant CTA not
displaying correctly on some platforms/browsers. It now shows as a
`Chat` button to the right of the table tabs, matching the other
assistant CTA's throughout the application.
<p align="center">
<img width="500"
src="9fcecd54-8e1a-423a-be05-7137632acbc4"
/>
</p>
And lastly removes `Beta` title from callout since we're going GA in
`8.10` 🎉
<p align="center">
<img width="500"
src="5beb379a-1bc7-4afc-b4bc-09f1d6085211"
/>
</p>
Resolves:
https://github.com/elastic/kibana/issues/163394#issuecomment-1693431066
Resolves: https://github.com/elastic/kibana/issues/164819
### Checklist
Delete any items that are not applicable to this PR.
- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
## [Security Solution] [Elastic AI Assistant] LangChain integration (experimental)
This PR integrates [LangChain](https://www.langchain.com/) with the [Elastic AI Assistant](https://www.elastic.co/blog/introducing-elastic-ai-assistant) as an experimental, alternative execution path.
### How it works
- There are virtually no client side changes to the assistant, apart from a new branch in `x-pack/packages/kbn-elastic-assistant/impl/assistant/api.tsx` that chooses a path based on the value of the `assistantLangChain` flag:
```typescript
const path = assistantLangChain
? `/internal/elastic_assistant/actions/connector/${apiConfig?.connectorId}/_execute`
: `/api/actions/connector/${apiConfig?.connectorId}/_execute`;
```
Execution of the LangChain chain happens server-side. The new route still executes the request via the `connectorId` in the route, but the connector won't execute the request exactly as it was sent by the client. Instead, the connector will execute one (or more) prompts that are generated by LangChain.
Requests routed to `/internal/elastic_assistant/actions/connector/${apiConfig?.connectorId}/_execute` will be processed by a new Kibana plugin located in:
```
x-pack/plugins/elastic_assistant
```
- Requests are processed in the `postActionsConnectorExecuteRoute` handler in `x-pack/plugins/elastic_assistant/server/routes/post_actions_connector_execute.ts`.
The `postActionsConnectorExecuteRoute` route handler:
1. Extracts the chat messages sent by the assistant
2. Converts the extracted messages to the format expected by LangChain
3. Passes the converted messages to `executeCustomLlmChain`
- The `executeCustomLlmChain` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts`:
1. Splits the messages into `pastMessages` and `latestMessage`, where the latter contains only the last message sent by the user
2. Wraps the conversation history in the `BufferMemory` LangChain abstraction
3. Executes the chain, kicking it off with `latestMessage`
```typescript
const llm = new ActionsClientLlm({ actions, connectorId, request });
const pastMessages = langchainMessages.slice(0, -1); // all but the last message
const latestMessage = langchainMessages.slice(-1); // the last message
const memory = new BufferMemory({
chatHistory: new ChatMessageHistory(pastMessages),
});
const chain = new ConversationChain({ llm, memory });
await chain.call({ input: latestMessage[0].content }); // kick off the chain with the last message
};
```
- When LangChain executes the chain, it will invoke `ActionsClientLlm`'s `_call` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/llm/actions_client_llm.ts` one or more times.
The `_call` function's signature is defined by LangChain:
```
async _call(prompt: string): Promise<string>
```
- The contents of `prompt` are completely determined by LangChain.
- The string returned by the promise is the "answer" from the LLM
The `ActionsClientLlm` extends LangChain's LLM interface:
```typescript
export class ActionsClientLlm extends LLM
```
This let's us do additional "work" in the `_call` function:
1. Create a new assistant message using the contents of the `prompt` (`string`) argument to `_call`
2. Create a request body in the format expected by the connector
3. Create an actions client from the authenticated request context
4. Execute the actions client with the request body
5. Save the raw response from the connector, because that's what the assistant expects
6. Return the result as a plain string, as per the contact of `_call`
## Desk testing
This experimental LangChain integration may NOT be enabled via a feature flag (yet).
Set
```typescript
assistantLangChain={true}
```
in `x-pack/plugins/security_solution/public/app/app.tsx` to enable this experimental feature in development environments.
## Summary
Adds new Elastic AI Assistant logo and global header menu item to all
Security Solution pages.
Resolves https://github.com/elastic/security-team/issues/7407
New logo within the assistant itself (header and assistant avatar):
<p align="center">
<img width="500"
src="2a94c2ca-37d6-49f0-af59-2b15fd37d81e"
/>
</p>
New global header menu for both on-prem and serverless security
`complete` deployments:
<p align="center">
<img width="500"
src="67b030fe-fb36-4a68-9331-d636e15a68f4"
/>
</p>
<p align="center">
<img width="500"
src="74751e3a-a88a-4b39-bec0-73497dcd98b1"
/>
</p>
Note: If Security Assistant RBAC privileges are `NONE` (which includes
serverless deployments that are NOT security `complete`), the global
header button will be hidden. We can revisit the upsell messaging
opportunity here for serverless deployments.
### Checklist
Delete any items that are not applicable to this PR.
- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
* @elastic/security-docs, will need to update images and make note of
new global header item, will create issue...
https://github.com/elastic/security-docs/issues/3804
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
UX: https://github.com/elastic/security-team/issues/7310
## Summary
* It creates an Upselling package to share the service and components
between ESS and Serverless plugins
* It implements upselling for entity analytics on serverless by
replicating the ESS approach
ESS
<img width="1520" alt="Screenshot 2023-08-17 at 13 34 59"
src="95c2c94e-7ab3-4d9f-aa24-b3e9c00eb3ed">
Serverless
<img width="1523" alt="Screenshot 2023-08-17 at 13 39 25"
src="618ce9dc-ef4e-469d-884a-dfb09834d0b0">
We are not displaying the upgrade button because we still don't know how
to link to the cloud settings page.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Resolves https://github.com/elastic/kibana/issues/159374 by ensuring
that if a user doesn't have the appropriate `Connectors & Actions`
privileges, they will be shown the appropriate messaging and any UI
controls for adding Connectors will be disabled or unavailable.
#### Connectors and Actions `NONE` or Connectors and Actions `READ` if
*NO* existing connectors exist:
<p align="center">
<img width="500"
src="d9535ae9-a31e-499b-9b18-6004e3db64de"
/>
</p>
#### Connectors and Actions `READ` if existing connector count > 0:
`Add Connector...` option isn't available:
<p align="center">
<img width="500"
src="bd6a06a7-ffa2-4cfc-a2b7-844da99cb171"
/>
</p>
<p align="center">
<img width="500"
src="4681086e-1015-45b9-9afb-ff604c52cd38"
/>
</p>
Also addresses:
* Fixes disabled state of header connector selector for setup flows.
* Adds `AssistantAvailability` interface to `AssistantContext` for
exposing ui feature controls like `Connectors & Actions` privileges.
* Hides `Add new connector...` option if user doesn't have `ALL`
`Connectors & Actions` privileges.
* Hoists dependencies from `assistant/index.tsx` to `connector_setup` as
it was already fetching dependencies from `useAssistantContext`.
Note: `ConnectorButton` and `ConnectorMissingCallout` should probably be
combined into a single component and show appropriate messaging given
the user's `Connectors & Actions` privileges. I kept them separate for
now as to not modify the control flow around the two components (till we
can further refactor `assistant/index.tsx`), which means the missing
connector callout is sort of doing double duty at the moment.
### Checklist
Delete any items that are not applicable to this PR.
- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
A recent EUI change has caused a problem with the theme when using the
deprecated `toMountPoint` inside `overlays.openFlyout` to create
flyouts.
This causes the contents of the rendered flyout to not know the current
theme, this is obvious when running in dark mode.
The fix is to switch to the non-deprecated version of `toMountPoint`.
Flyouts:
Create anomaly detection job from Lens flyout in Dashboard.
Anomaly swim lane and anomaly chart job embeddables job selection flyout
in Dashboard.
Log pattern analysis flyout in Discover.
Modals:
Trained models start deployment modal.
Trained models force stop deployment modal.
Trained models stop deployment modal when there are multiple
deployments.
Misc:
Page not found banner.
Jobs list header, which contains the settings button.
DFA clone job warning toast when the original data view no longer
exists.
Components in ml's date picker package
Fixes https://github.com/elastic/kibana/issues/164379
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Dima Arnautov <arnautov.dima@gmail.com>
## Summary
There's a few EUI imports out there that folks are reaching into
`@elastic/eui/lib/` for (which doesn't contain any types - something I'm
looking into separately) that could instead be imported at the top
`@elastic/eui` level, which is properly typed.
### Checklist
N/A - types only
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
