## Summary
This ticket is the initial implementation for the UI side for the
AI-driven custom integration creation.
This PR only contains the implementation of the UI, due to the tight
timing it will not include tests, everything will be tested manually for
8.15 FF. We'll implement the tests later.
#### Enable Feature
The new integration assistant plugin is disabled by default, to enable
it:
```
xpack.integration_assistant.enabled: true
```
#### Complete tasks
- [x] New integration button on the /integrations page
- [x] New integration "landing" page with buttons to upload zip and
assistant
- [x] Upload zip page to install integration
- [x] Integration assistant:
- [x] Connector selection step
- [x] Integration details step
- [x] Data stream step
- [x] Review and install
#### Follow-ups (will be implemented in separate PRs)
- [ ] Add RBAC
- [ ] Add telemetry
- [ ] Documentation
- [ ] Add license/productType controls
- [ ] Add links to the create integration page
- [ ] Improve package name retrieval:
https://github.com/elastic/kibana/issues/185932
- [ ] Add time estimation on the generation stage
- [ ] Add support for multi-valuated "input type"
- [ ] Enable Langsmith tracing using AI assistant settings
#### Demo
b04c21c6-09cf-49bb-be8f-bf4b9d3feb8e
## Files by Code Owner
### elastic/docs
* packages/kbn-doc-links/src/get_doc_links.ts
* packages/kbn-doc-links/src/types.ts
### elastic/fleet
* x-pack/plugins/fleet/kibana.jsonc
*
x-pack/plugins/fleet/public/applications/integrations/hooks/use_breadcrumbs.tsx
*
x-pack/plugins/fleet/public/applications/integrations/layouts/default.tsx
*
x-pack/plugins/fleet/public/applications/integrations/sections/epm/index.tsx
*
x-pack/plugins/fleet/public/applications/integrations/sections/epm/screens/create/index.tsx
* x-pack/plugins/fleet/public/components/header.tsx
* x-pack/plugins/fleet/public/constants/page_paths.ts
* x-pack/plugins/fleet/public/plugin.ts
* x-pack/plugins/fleet/tsconfig.json
### elastic/kibana-core
* x-pack/plugins/fleet/kibana.jsonc
* x-pack/plugins/integration_assistant/kibana.jsonc
### elastic/kibana-operations
* packages/kbn-optimizer/limits.yml
### elastic/security-solution
* x-pack/plugins/integration_assistant/**/*
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This is a PR to add a new backend plugin (frontend will be done in
separate [PR](https://github.com/elastic/kibana/pull/184546)).
The purpose of the plugin is to provide a set of API routes that is used
to perform a variety of GenAI workflows to generate new integrations
based on provided inputs.
It reuses the existing GenAI connectors for its LLM communication, and
provides a set of API's to create ECS mapping, Categorization, Related
Fields and an API to generate the actual integration package zip, which
is forwarded to the UI component.
### Planned follow-up changes:
As the PR is getting way too large, some planned changes would be added
in much smaller follow-ups. This includes mostly more improved try/catch
for certain routes, adding debug/error log entries where relevant,
especially for the API endpoints themself, some more unit and end2end
tests.
- OpenAPI spec for the API will be handled in a separate PR
- All the missing unit tests will be added as a followup PR
### Testing
The `integration_assistant` plugin will be disabled by default while
it's being implemented so we can iterate and merge partial PRs without
interfering with the releases. This config will work as our feature
flag:
6aefd4ff7b/x-pack/plugins/integration_assistant/server/config.ts (L11-L13)
To test it add this to your _kibana.dev.yml_:
```
xpack.integration_assistant.enabled: true
```
### Checklist
Delete any items that are not applicable to this PR.
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
Co-authored-by: Sergi Massaneda <sergi.massaneda@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Bharat Pasupula <saibharatchandra.pasupula@elastic.co>
Co-authored-by: Bharat Pasupula <123897612+bhapas@users.noreply.github.com>
Use one react-hooks/exhaustive-deps across our Obs plugins, for
consistency reasons.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Carlos Crespo <crespocarlos@users.noreply.github.com>
## Summary
Upgrading intl packages from v2 to v6
### Packages upgrade:
- [x] Add @formatJS packages
- [x] `react-intl` Upgraded
- [x] `intl-messageformat` Upgraded
- [x] `intl-format-cache` removed
- [x] `intl-relativeformat` removed
- [x] `intl-messageformat-parser` removed
### Todo list:
- [x] Refactor HTML tags
- [x] Refactor Upgrade tags
- [x] Refactor `kbn-i18n`
- [x] Refactor `kbn-i18n-react`
- [x] Refactor `FormattedRelative` to `FormattedRelativeTime`
- [x] Refactor polyfills
- [x] Refactor IntlShape types
- [x] Rewrite Providers
- [x] Rewrite tests using i18n
- [x] Removed current pseudolocale implementation (tracker:
https://github.com/elastic/kibana/issues/180244)
- [x] Fix jest tests using rendered `Provider`
- [x] Remove no longer valid i18n packages documentation (tracker:
https://github.com/elastic/kibana/issues/180259)
Closes https://github.com/elastic/kibana/issues/178968
Closes https://github.com/elastic/kibana/issues/38642
## Notes to code reviewers
For team other than the core team, please review your plugins code
changes by filtering files by codeowners.
### Test Snapshot updates
Most of the changes are refactors of renamed functions and changed ICU
syntax.
The main updates are snapshot changes where `FormattedMessage` is now
memoized so snapshots capturing the html tree needed to be updated to
use `<Memo(MemoizedFormattedMessage)` instead of `<FormattedMessage`
### ICU now supports HTML tags:
before:
```
<FormattedMessage
defaultMessage="To buy a shoe, { link } and { cta }"
values={{
link: (
<a class="external_link" target="_blank" href="https://www.shoe.com/">
visit our website
</a>
),
cta: <strong class="important">eat a shoe</strong>,
}}
/>
```
after:
```
<FormattedMessage
defaultMessage="To buy a shoe, <a>visit our website</a> and <cta>eat a shoe</cta>"
values={{
a: msg => (
<a class="external_link" target="_blank" href="https://www.shoe.com/">
{msg}
</a>
),
cta: msg => <strong class="important">{msg}</strong>,
}}
/>
```
### Escape character to prevent ICU parsing changed from double slashes
to single quotes:
before: `\\{escaped\\}`
after: `'{escaped}'`
### No need for Intl Shape
the new packages under formatJS are written in typescript and come with
types support out of the box so no need to set types when using i18n.
Renamed `InjectedIntlProps` with `WrappedComponentProps`.
Removed `prop-types` and `intlShape` in favor of `IntlShape`.
### FormattedRelative has been renamed to FormattedRelativeTime and its
API has changed significantly. See
[FormattedRelativeTime](https://formatjs.io/docs/react-intl/upgrade-guide-3x#formattedrelativetime)
for more details.
### All tags specified must have corresponding values and will throw
error if it's missing
All tags are now parsed and expected to be formatted properly (all
opened tags must be closed).
To skip this check you can use the `ignoreTag: true` property
```
i18n.translate('xpack.apm.agentConfig.captureJmxMetrics.description', {
defaultMessage: 'This is not an HTML tag <JMX object name pattern>' +
ignoreTag: true,
}),
```
**When do I use ignore tags?**
If your message has HTML tags, it is preferred not to ignore the Tag to
have some string verification that the html tags you are adding are
properly formatted and closed.
If it the text between brackets is not an HTML tag and it is just a
fomat preference then using `ignoreTag` makes sense.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
## Summary
Close https://github.com/elastic/kibana/issues/182390
### To test
You can test all aspects of the script by first making some changes to
`packages/kbn-esql-validation-autocomplete/src/validation/validation.test.ts`
- delete several tests from an existing function describe block ("block
1"—except don't choose "date_diff" since it has a bunch of custom tests)
- delete another function describe block completely ("block 2")
- change the expected result of several of the tests in a third function
describe block ("block 3")
Then, run `yarn maketests` from within
`packages/kbn-esql-validation-autocomplete`
**Expected result**
- Block 1 should have the deleted tests restored
- Block 2 should be restored entirely (though it may be moved in the
tests file)
- Block 3 should be untouched
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Adds the same Eslint rules as the rest of security_solution to the
`cell-actions` package, adapting the code.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Enables telemetry warning eslint rule.
Fixes a bunch of missing telemetry strings.
### Checklist
Delete any items that are not applicable to this PR.
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Fixes https://github.com/elastic/kibana/issues/176420
## 🍒 Summary
This PR copies the SLO code that was inside the Observability app into
its own app under `observability-solution/slo` folder.
4f6b8dfb-9612-4d30-ad50-4ee5c55a9c32
## ✔️ Acceptance criteria
- URL of new app: `app/slos`
- Design and functionality are not changed.
- Git history has been retained for all files in
`x-pack/plugins/observability_solution/slo`.
- SLO should appear on server less
- SLO code inside `observability_solution/observability` code has been
removed. A new clean up round might be needed though for possible
leftovers.
- Burn rate rule is registered within the new slo app
- SLO embeddables are moved inside the new slo app
- overview
- alerts embeddable
- error budget burn down
- Alerts table configuration registration for slo details page and
alerts table embeddable is still done in the observability app. Response
Ops team is working on removing the need to register the alert table
anyway
- Slo app is wrapped into `ApplicationUsageTrackingProvider` which will
send slo `Application usage` information tracked by the `slo` appId
- Redirect old `app/observability/slos` route to `app/slos`
- Rename old `xpack.observability.slo` keys to `xpack.slo` in the
translation files
## 🌮 How to test
Design and functionality didn't change, so simply navigate to existing
slo pages and try to break it
- Slo list page
- group by
- unified search
- toggle buttons
- actions
- Slo creation
- try group by as well
- Slo detail page
- Actions on top
- navigate to overview and alerts tabs
- Create SLO flyout in Logs Explorer
- Create burn rate rules and verify they appear on rules page
- Verify SLO alerts appear on Alerts page and slo details page
- Embeddables
- Through the dashboard app
- Using the attach to dashboard action on the slo card item on slo list
page and the error budget burn down chart on the slo detail page
- SLOs only for platinum users
- Permissions
- Spaces
## TODO
- [x] Move slo stuff from observability folder to new slo plugin
- [x] Remove old slo stuff from observability folder
- [x] Update references
- [x] Fix typescript and eslint errors
- [x] Paths
- [x] Locators
- [x] Burn rate rule registration
- [x] Embeddable Alerts table configuration registration
- [x] Embeddables
- [x] Translations
- [x] Verify plugin.ts files contain all registration logic
- [x] public
- [x] server
- [x] Final cleanup for observability folder
- [x] Run tests
- [x] Application Usage (Telemetry)
- [x] Permissions
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: shahzad31 <shahzad31comp@gmail.com>
Co-authored-by: Coen Warmer <coen.warmer@gmail.com>
## Summary
issue https://github.com/elastic/security-team/issues/7382
### Data Stream Adapter
This PR introduces the `@kbn/data-stream-adapter` package, which is a
utility library to facilitate Data Stream creation and maintenance in
Kibana, it was inspired by the data stream implementation in the Alerts
plugin.
The library has two exports:
- `DataStreamSpacesAdapter`: to manage space data streams. It uses the
`name-of-the-data-stream-<spaceId>` naming pattern.
- `DataStreamAdapter`: to manage single (not space-aware) data streams.
Usage examples in the package
[README](450be0369d/packages/kbn-data-stream-adapter/README.md)
### Data Quality Dashboard
The `DataStreamSpacesAdapter` has been integrated into the data quality
dashboard to store all the quality checks users perform. The information
stored is the metadata (also used for telemetry) and the actual data
rendered in the tables.
FieldMap definition
[here](450be0369d/x-pack/plugins/ecs_data_quality_dashboard/server/lib/data_stream/results_field_map.ts)
### Demo
311a0bf5-004b-46d7-8140-52a233361c91
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Philippe Oberti <philippe.oberti@elastic.co>
Co-authored-by: Garrett Spong <spong@users.noreply.github.com>
Co-authored-by: Efe Gürkan YALAMAN <efeguerkan.yalaman@elastic.co>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Co-authored-by: Sander Philipse <94373878+sphilipse@users.noreply.github.com>
Co-authored-by: JD Kurma <JDKurma@gmail.com>
Co-authored-by: Jan Monschke <jan.monschke@elastic.co>
Co-authored-by: Patryk Kopyciński <contact@patrykkopycinski.com>
Co-authored-by: Khristinin Nikita <nikita.khristinin@elastic.co>
Co-authored-by: Marco Liberati <dej611@users.noreply.github.com>
Co-authored-by: Julia Rechkunova <julia.rechkunova@elastic.co>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: Davis McPhee <davis.mcphee@elastic.co>
Co-authored-by: Eyo O. Eyo <7893459+eokoneyo@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Marta Bondyra <4283304+mbondyra@users.noreply.github.com>
Co-authored-by: Søren Louv-Jansen <soren.louv@elastic.co>
Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
Co-authored-by: Candace Park <56409205+parkiino@users.noreply.github.com>
## Summary
Refactoring general ui service to a kbn package.
Resolves an [Appex QA](https://github.com/elastic/appex-qa-team) issue.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## [Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts
This PR implements _Retrieval Augmented Generation_ (RAG) for Alerts in the Security Solution. This feature enables users to ask the assistant questions about the latest and riskiest open alerts in their environment using natural language, for example:
- _How many alerts are currently open?_
- _Which alerts should I look at first?_
- _Did we have any alerts with suspicious activity on Windows machines?_
### More context
Previously, the assistant relied solely on the knowledge of the configured LLM and _singular_ alerts or events passed _by the client_ to the LLM as prompt context. This new feature:
- Enables _multiple_ alerts to be passed by the _server_ as context to the LLM, via [LangChain tools](https://github.com/elastic/kibana/pull/167097)
- Applies the user's [anonymization](https://github.com/elastic/kibana/pull/159857) settings to those alerts
- Only fields allowed by the user will be sent as context to the LLM
- Users may enable or disable anonymization for specific fields (via settings)
- Click the conversation's `Show anonymized` toggle to see the anonymized values sent to / received from the LLM:
### Settings
This feature is enabled and configured via the `Knowledge Base` > `Alerts` settings in the screenshot below:
- The `Alerts` toggle enables or disables the feature
- The slider has a range of `10` - `100` alerts (default: `20`)
When the setting above is enabled, up to `n` alerts (as determined by the slider) that meet the following criteria will be returned:
- the `kibana.alert.workflow_status` must be `open`
- the alert must have been generated in the last `24 hours`
- the alert must NOT be a `kibana.alert.building_block_type` alert
- the `n` alerts are ordered by `kibana.alert.risk_score`, to prioritize the riskiest alerts
### Feature flag
To use this feature:
1) Add the `assistantRagOnAlerts` feature flag to the `xpack.securitySolution.enableExperimental` setting in `config/kibana.yml` (or `config/kibana.dev.yml` in local development environments), per the example below:
```
xpack.securitySolution.enableExperimental: ['assistantRagOnAlerts']
```
2) Enable the `Alerts` toggle in the Assistant's `Knowledge Base` settings, per the screenshot below:
## How it works
- When the `Alerts` settings toggle is enabled, http `POST` requests to the `/internal/elastic_assistant/actions/connector/{id}/_execute` route include the following new (optional) parameters:
- `alertsIndexPattern`, the alerts index for the current Kibana Space, e.g. `.alerts-security.alerts-default`
- `allow`, the user's `Allowed` fields in the `Anonymization` settings, e.g. `["@timestamp", "cloud.availability_zone", "file.name", "user.name", ...]`
- `allowReplacement`, the user's `Anonymized` fields in the `Anonymization` settings, e.g. `["cloud.availability_zone", "host.name", "user.name", ...]`
- `replacements`, a `Record<string, string>` of replacements (generated on the server) that starts empty for a new conversation, and accumulates anonymized values until the conversation is cleared, e.g.
```json
"replacements": {
"e4f935c0-5a80-47b2-ac7f-816610790364": "Host-itk8qh4tjm",
"cf61f946-d643-4b15-899f-6ffe3fd36097": "rpwmjvuuia",
"7f80b092-fb1a-48a2-a634-3abc61b32157": "6astve9g6s",
"f979c0d5-db1b-4506-b425-500821d00813": "Host-odqbow6tmc",
// ...
},
```
- `size`, the numeric value set by the slider in the user's `Knowledge Base > Alerts` setting, e.g. `20`
- The `postActionsConnectorExecuteRoute` function in `x-pack/plugins/elastic_assistant/server/routes/post_actions_connector_execute.ts` was updated to accept the new optional parameters, and to return an updated `replacements` with every response. (Every new request that is processed on the server may add additional anonymized values to the `replacements` returned in the response.)
- The `callAgentExecutor` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts` previously used a hard-coded array of LangChain tools that had just one entry, for the `ESQLKnowledgeBaseTool` tool. That hard-coded array was replaced in this PR with a call to the (new) `getApplicableTools` function:
```typescript
const tools: Tool[] = getApplicableTools({
allow,
allowReplacement,
alertsIndexPattern,
assistantLangChain,
chain,
esClient,
modelExists,
onNewReplacements,
replacements,
request,
size,
});
```
- The `getApplicableTools` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/index.ts` examines the parameters in the `KibanaRequest` and only returns a filtered set of LangChain tools. If the request doesn't contain all the parameters required by a tool, it will NOT be returned by `getApplicableTools`. For example, if the required anonymization parameters are not included in the request, the `open-alerts` tool will not be returned.
- The new `alert-counts` LangChain tool returned by the `getAlertCountsTool` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/alert_counts/get_alert_counts_tool.ts` provides the LLM the results of an aggregation on the last `24` hours of alerts (in the current Kibana Space), grouped by `kibana.alert.severity`. See the `getAlertsCountQuery` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/alert_counts/get_alert_counts_query.ts` for details
- The new `open-alerts` LangChain tool returned by the `getOpenAlertsTool` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/open_alerts/get_open_alerts_tool.ts` provides the LLM up to `size` non-building-block alerts generated in the last `24` hours (in the current Kibana Space) with an `open` workflow status, ordered by `kibana.alert.risk_score` to prioritize the riskiest alerts. See the `getOpenAlertsQuery` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/open_alerts/get_open_alerts_query.ts` for details.
- On the client, a conversation continues to accumulate additional `replacements` (and send them in subsequent requests) until the conversation is cleared
- Anonymization functions that were only invoked by the browser were moved from the (browser) `kbn-elastic-assistant` package in `x-pack/packages/kbn-elastic-assistant/` to a new common package: `x-pack/packages/kbn-elastic-assistant-common`
- The new `kbn-elastic-assistant-common` package is also consumed by the `elastic_assistant` (server) plugin: `x-pack/plugins/elastic_assistant`
## Summary
This PR adds a bunch of plugins to help manage AI Assistant Management
settings.
It offers a 'selection' plugin inside Stack Management where a user can
select which AI Assistant she wants to manage.
The Security team can hook into this one, so settings for both AI
Assistants can be accessed from inside one place inside Stack
Management.
This PR also adds the plugin to manage settings for the AI Assistant for
Observability, including Knowledge Base management. This plugin is
available both in Stack Management (stateful) and Project Settings
(serverless).
## What it looks like
51392ec5-05c9-4947-9bf2-810d8d0b7525
## Detailed
1. **Adds a Stack Management plugin**
(`/src/plugins/ai_assistant_management/selection`). Its primary function
is to render a selection screen to help users navigate to the settings
plugin for the AI Assistant for a specific solution. This plugin is
displayed in Stack Management, which is only available in stateful
versions of Kibana.
2. **Adds a AI Assistant for Observability Settings plugin**
(`/src/plugins/ai_assistant_management/observability`). This plugin
allows management of specific Observability AI Assistant settings. It is
available in stateful versions of Kibana (via the aforementioned Stack
Management plugin) or in serverless versions via Project Management.
3. **Knowledge Base management for Observability AI Assistant**: The AI
Assistant for Observability Settings plugin has a Knowledge Base tab,
which allows users to add / read / update / delete and bulk import
entries into the Knowledge Base of the Observability AI Assistant.
4. **Moving of KB endpoints in Observability AI Assistant plugin**: KB
endpoints and functions were located in the same folder. As this PR adds
new endpoints for the KB for CRUD operations, it also moves the existing
ones from the function folder into a dedicated one so there's a clearer
distinction between kb and functions.
5. **Adding of GenAI Connector inside Chat Flyout**: If the user has
admin rights, it is possible to set up a GenAI connector from within the
Observability AI Assistant Chat Flyout. This provides a faster and more
seamless onboarding experience. If the user does not, she will be
redirected to the Settings page.
## Bug fixes
* Fixes chat item styling issues (padding, background color).
## How to test
* Check if the Stack Management plugin works on stateful
* Check if the AI Assistant Settings plugin works on stateful +
serverless
* Check if CRUD operations on KB work
* Check if searching on KB entries work
* Check if its possible to navigate to KB tab directly
(`app/management/kibana/aiAssistantManagementObservability?tab=knowledge_base`)
## Todo
- [x] Add sorting to getEntries
- [x] Add params for tab routing
- [x] Add unit tests
- [ ] Add API tests
- [ ] Add fallback for already indexed entries when searching
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes#169733
#### Reporting fix
Change point detection embeddable was incorrectly reporting render
completion. It was relying on the `onLoad` callback from the Lens
embeddable responsible for chart rendering, which only indicates that
data fetching is complete, but not the actual rendering. Current
implementation relies on the `renderComplete` event from each child
embeddable. Both PNG and PDF exports tested and work as expected.
#### Additional fixes
- Fixes the metric and split field controls states when editing existing
Change point embeddable from a dashboard
- Fixes `filter` query if partitions input is initialized as an empty
array.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
This makes the newly created i18n ESLint rule actually activate on
paths, and narrows the use of the Telemetry ESLint rule to only activate
on .tsx files. This is done because the rule only makes sense in the
context of JSX.
## More words
Noticed that this:
```
'x-pack/plugins/aiops/**/*.{tsx}',
'x-pack/plugins/apm/**/*.{tsx}',
'x-pack/plugins/exploratory_view/**/*.{tsx}',
```
does not work.
This:
```
'x-pack/plugins/aiops/**/*.tsx',
'x-pack/plugins/apm/**/*.tsx',
'x-pack/plugins/exploratory_view/**/*.tsx',
```
does.
# Summary
- Addresses https://github.com/elastic/kibana/issues/161537
## Description
- This PR follows the second option defined in this
[document](https://docs.google.com/document/d/1mqkpjDdFQRFvx_RPvNmjstVj8SXYMr2mrETMv3esda8/edit#heading=h.rpv1zyeb04ay)
the [Mocha tagging ](https://github.com/mochajs/mocha/wiki/Tagging)
- It introduces a new folder
`x-pack/test/security_solution_api_integration` which will serve as a
centralized location to meet all the requirements related to renaming
tests appropriately and grouping similar tests together. It will
facilitate the management of tests that must be run in Serverless and
ESS environments.
- Within this folder, there is a "config" subdirectory that stores base
configurations specific to both the
[Serverless](https://github.com/elastic/kibana/pull/166755/files#diff-afe1f42d5ac2006de8dc09069448b9e8734a6a950586376cd6e8eeb9110ab5f1R1)
and
[ESS](https://github.com/elastic/kibana/pull/166755/files#diff-4a60bd8c91da08a3f7ec14bf3bfef8449af155611374c32579b0318da03e292cR1)
environments, These configurations build upon the base configuration
provided by test_serverless and api_integrations, incorporating
additional settings such as environment variables and tagging options.
- It demonstrates scenarios involving `@ess`, `@serverless`, and
`@brokenInServerless`.
- The file`
x-pack/test/security_solution_api_integration/test_suites/detections_response/rule_creation/create_rules.ts`
is functional in both **Serverless** and **ESS**. However, some tests
related to roles are currently skipped for Serverless until they are
resolved, and these tests are tagged with `@brokenInServerless`
## CI
- It includes a new entry in the ftr_configs.yml to execute the newly
added tests in the pipeline.
- It involves the addition of mochaOptions in both
serverless/config.base.ts and ess/config.base.ts. In the case of
serverless, it includes **@serverless** while excluding
@brokenInServerless. Similarly, for **ess**, it includes @ess and
excludes **@brokenInEss**.
from `x-pack/test/security_solution_api_integration/config/serverless`
## Update in x-pack/test/detection_engine_api_integration
- The `create_rules.ts` and `create_rule_exceptions` files have been
relocated from
`x-pack/test/detection_engine_api_integration/security_and_spaces/group1`
to their respective domains within the
`x-pack/test/security_solution_api_integration` folder.
- The util files now are copied over from the old folder
`x-pack/test/detection_engine_api_integration` to the new folder and
will be removed once all tests are moved to the new folder to don't
break the existing tests
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Implements #153108.
This enables the
`@kbn/telemetry/event_generating_elements_should_be_instrumented` eslint
rule for the `aiops` plugin to enforce `data-test-subj` attributes on
actionable EUI components so they are auto-instrumented by telemetry.
The ids were first auto-created using `node scripts/eslint --fix
x-pack/plugins/aiops` and then adapted.
### What this PR changes
branched from elastic/kibana/pull/163759
- Introduces new AppFeatures package `@kbn/security-solution-features`
with the common logic and `AppFeatureService` to apply offering specific
configurations for Security Solution features independently for
Serverless and ESS. This logic is replacing the earlier `AppFeatures` in
order to introduce new Kibana feature privileges for serverless PLIs so
that new Kibana privileges introduced for serverless PLIs do not
affect/show up as new Kibana feature privileges in ESS.
- Gates endpoint exceptions on alerts/rules based on serverless PLI
configurations. On serverless `Endpoint exceptions` should be
accessible/seen only on endpoint essentials/complete.
New AppFeatures logic architecture diagram:
**Note:** Corresponding API changes related to endpoint exceptions will
be in a new PR, along with the last set of UX changes for hiding the
`Endpoint exceptions` tab from the Rules details page.
### How to review
- Setup for _Servlerless_
- Run `yarn es snapshot` on a terminal window to start ES.
- Copy `config/serverless.security.yml` to
`config/serverless.security.dev.yml`
- Run `yarn serverless-security --no-base-path` on another terminal
window to start kibana in serverless mode
- Run `node
x-pack/plugins/security_solution/scripts/endpoint/endpoint_agent_emulator.js
--asSuperuser` on a new window and then select `1` to `Load Endoints`
and then `1` to `Run` the loader script. This will load some fake
agents/alerts data to test with.
### Tests (Serverless)
- with
`{ product_line: 'security', product_tier: 'essentials' }` or `{
product_line: 'security', product_tier: 'complete' }`
and
`{ product_line: 'endpoint', product_tier: 'essentials' }` or `{
product_line: 'endpoint', product_tier: 'complete' }`
1. Navigate to Rules>Shared exception lists via
`http://localhost:5601/app/security/exceptions`
2. Test that you can see `Endpoint Security Exception List` card on the
shared exception lists page.
3. Navigate to `Alerts` page via `app/security/alerts`, you should see
endpoint alerts. If not, then click on `Manage Rules` and then
disable/enable `Endpoint Security` rules. That should trigger alerts to
show up on the Alerts table.
4. Click on `View Details` button under `Actions` column. Once the
flyout is visible, click on `Take Action` and verify that `Add Endpoint
exception` is visible/enabled/clickable on the menu.
5. Click on `More actions` button under `Actions` column and verify that
`Add Endpoint exception` is visible/enabled/clickable on the menu.
6. Click on `Investigate in timeline` button under `Actions` column;
when the timeline view is visible and the alert item is displayed, click
on buttons mentioned in 4. and 5. above and verify the same.
7. Navigate to `Rules`>`DetectionRules`>`Endpoint Security` rule under
the `Rules` table. Select the `Alerts` tab.
8. Click and verify `View details`,`More actions` and `Investigate in
timeline` buttons same as in 4., 5., 6. above.
9. You should be able to see the `Endpoint exceptions` tab as well.
Click and verify that you can see the tab's content.
- with
`{ product_line: 'security', product_tier: 'essentials' }` or `{
product_line: 'security', product_tier: 'complete' }`
1. Edit `config/serverless.security.dev.yml` so that `endpoint` product
line item is commented out.
2. Test that you can not see `Endpoint Security Exception List` card on
the shared exception lists page.
3. Items 4. 5. 6. as above but the menu items should be disabled. This
can be verified with fake data only as with a real endpoint, endpoint
alerts are actually not visible at all.
### Tests (ESS)
On the ESS side, endpoint exceptions are not affected by this change and
work as usual based on index privileges.
---------
Co-authored-by: semd <sergi.massaneda@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: YulNaumenko <jo.naumenko@gmail.com>
Co-authored-by: Pablo Neves Machado <pablo.nevesmachado@elastic.co>
Co-authored-by: Pablo Machado <machadoum@gmail.com>
## [Security Solution] [Elastic AI Assistant] LangChain integration (experimental)
This PR integrates [LangChain](https://www.langchain.com/) with the [Elastic AI Assistant](https://www.elastic.co/blog/introducing-elastic-ai-assistant) as an experimental, alternative execution path.
### How it works
- There are virtually no client side changes to the assistant, apart from a new branch in `x-pack/packages/kbn-elastic-assistant/impl/assistant/api.tsx` that chooses a path based on the value of the `assistantLangChain` flag:
```typescript
const path = assistantLangChain
? `/internal/elastic_assistant/actions/connector/${apiConfig?.connectorId}/_execute`
: `/api/actions/connector/${apiConfig?.connectorId}/_execute`;
```
Execution of the LangChain chain happens server-side. The new route still executes the request via the `connectorId` in the route, but the connector won't execute the request exactly as it was sent by the client. Instead, the connector will execute one (or more) prompts that are generated by LangChain.
Requests routed to `/internal/elastic_assistant/actions/connector/${apiConfig?.connectorId}/_execute` will be processed by a new Kibana plugin located in:
```
x-pack/plugins/elastic_assistant
```
- Requests are processed in the `postActionsConnectorExecuteRoute` handler in `x-pack/plugins/elastic_assistant/server/routes/post_actions_connector_execute.ts`.
The `postActionsConnectorExecuteRoute` route handler:
1. Extracts the chat messages sent by the assistant
2. Converts the extracted messages to the format expected by LangChain
3. Passes the converted messages to `executeCustomLlmChain`
- The `executeCustomLlmChain` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts`:
1. Splits the messages into `pastMessages` and `latestMessage`, where the latter contains only the last message sent by the user
2. Wraps the conversation history in the `BufferMemory` LangChain abstraction
3. Executes the chain, kicking it off with `latestMessage`
```typescript
const llm = new ActionsClientLlm({ actions, connectorId, request });
const pastMessages = langchainMessages.slice(0, -1); // all but the last message
const latestMessage = langchainMessages.slice(-1); // the last message
const memory = new BufferMemory({
chatHistory: new ChatMessageHistory(pastMessages),
});
const chain = new ConversationChain({ llm, memory });
await chain.call({ input: latestMessage[0].content }); // kick off the chain with the last message
};
```
- When LangChain executes the chain, it will invoke `ActionsClientLlm`'s `_call` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/llm/actions_client_llm.ts` one or more times.
The `_call` function's signature is defined by LangChain:
```
async _call(prompt: string): Promise<string>
```
- The contents of `prompt` are completely determined by LangChain.
- The string returned by the promise is the "answer" from the LLM
The `ActionsClientLlm` extends LangChain's LLM interface:
```typescript
export class ActionsClientLlm extends LLM
```
This let's us do additional "work" in the `_call` function:
1. Create a new assistant message using the contents of the `prompt` (`string`) argument to `_call`
2. Create a request body in the format expected by the connector
3. Create an actions client from the authenticated request context
4. Execute the actions client with the request body
5. Save the raw response from the connector, because that's what the assistant expects
6. Return the result as a plain string, as per the contact of `_call`
## Desk testing
This experimental LangChain integration may NOT be enabled via a feature flag (yet).
Set
```typescript
assistantLangChain={true}
```
in `x-pack/plugins/security_solution/public/app/app.tsx` to enable this experimental feature in development environments.
## Summary
Implementation of serverless-specific pages within the Unified IA
Navigation.
#### Links implemented:
- `Machine Learning`
- Landing page created on serverless only
- All links in the landing page go to `/ml` app
- `Dev Tools`
- Links directly to `/dev_tools` app
#### Links not implemented:
```// TODO: in a follow-up PR```
- Project Settings
- Change the _Settings_ name by _Project Settings_
- Modify the landing page items according to the design
## Changes
### Plugin contract changes
The Machine Learning landing page is the first page that is only available on serverless and should not exist in ess (there are more of this kind in the pipeline), so this PR implements the foundations to enable the _security_solution_serverless_ plugin to implement its own page components, configure the link definition and create new routes to render them in the Security Solution application.
These new APIs can be called from either `security_solution_serverless` or `security_solution_ess`, allowing those plugins to have their own offering-specific pages.
The new APIs exposed in the security_solution public contract are the following:
- `extraAppLinks$`: Observable to add extra app_links into the application links configuration, so they are stored and included in the SecuritySolution plugin `deepLinks` registry, to make them accessible from anywhere in the application using the `chrome.navLinks` API.
- `extraRoutes$`: Observable to add extra routes into the main Router, so it can render the new page components. These additional routes are appended after the "sub-plugin" (_alerts_, _timeline_, ...) routes, so it is not possible to override an existing route path.
### New `security-solution-navigation` package
Since now we need to use the same navigation components and hooks in different plugins, these functionalities have been extracted to the `@kbn/security-solution-navigation` package, which all Security plugins will depend on (generic, serverless, and ess).
The modules exposed by this package have been extracted from the main security_solution plugin and standardized. They include the Landing pages components (new [storybook](https://ci-artifacts.kibana.dev/storybooks/pr-161667/394abe76676c6a76b2982c1d3f5bb675739c3477/security_solution_packages/index.html?path=/story/landing-links-landing-links-icons-categories--landing-links-icons-categories) available), navigation hooks, and link utilities. Also, some types and constants have been moved to this package.
A new context provider has also been created, which needs to be in place in order to use this package. The `<NavigationProvider core={core}>` is required for the package functionalities to have access to the Kibana core navigation APIs: `navigateToUrl`, `navigateToApp`, and `getUrlForApp`.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: YulNaumenko <jo.naumenko@gmail.com>
## Summary
closes: https://github.com/elastic/kibana/issues/159685
- Renaming _x-pack/plugins_:
`serverless_security` -> `security_solution_serverless`
`ess_security` -> `security_solution_ess`
- All the related configurations and types have also been renamed.
- i18n translation prefixes updated
- relocation of internal `security_solution_serverless` directories to
be consistent with `security_solution_ess`
### Eslint
I also added the plugins in the `.eslintrc` configuration, defining the
same rules as the `security_solution` plugin.
All eslint errors have been addressed (mainly _type_ imports errors)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
As part of the actions for making Profiling production ready, this PR
adds basic API tests on the Profiling APIs checking if only users with
`access:profiling` are allowed to call our APIs, other users must be
forbidden.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Why?
To simplify the process of migration to react-router@6.
https://github.com/remix-run/react-router/discussions/8753
What problems exactly it solves?
- In my previous PR I added `CompatRouter`
https://github.com/elastic/kibana/pull/159173, which caused changes in
~50 files and pinged 15 Teams. And this is just meant to be a temporary
change, so when we're done with the migration I would have to revert
these changes and engage everyone to review the PR again. And it is just
a single step in the migration strategy. So to make our lives easier I
think it would be better to have a common place where we do import our
router components because it will allow us to surface some extra logic
in single place instead of going through the whole source code again.
- `react-router@6` doesn't support a custom `Route` component, so that
means our custom `Route` component that we're using almost everywhere
today, will need to be replaced by a different solution. I have decided
to add `Routes` component, which will be responsible for rendering the
proper component (`react-router@6` renamed `Switch` to `Routes`, so I
have named this component to align with the dictionary of the new
router) and also is going to add the logic that today is done in `Route`
(moving logic to `Routes` will be done in the follow-up PR, here I just
wanted to focus on using the common router components to make the review
process easier)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
