# Backport
This will backport the following commits from `main` to `8.14`:
- [[Search] Adds telemetry warning rules and fixes a bunch of telemetry
issues (#181350)](https://github.com/elastic/kibana/pull/181350)
<!--- Backport version: 9.4.3 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)
<!--BACKPORT [{"author":{"name":"Efe Gürkan
YALAMAN","email":"efeguerkan.yalaman@elastic.co"},"sourceCommit":{"committedDate":"2024-04-23T13:49:25Z","message":"[Search]
Adds telemetry warning rules and fixes a bunch of telemetry issues
(#181350)\n\n## Summary\r\n\r\nEnables telemetry warning eslint
rule.\r\n\r\nFixes a bunch of missing telemetry strings.\r\n\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [ ] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"0a4081e1d81eb980998585144c85ac8fbd078f8c","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:EnterpriseSearch","v8.14.0","v8.15.0"],"title":"[Search]
Adds telemetry warning rules and fixes a bunch of telemetry
issues","number":181350,"url":"https://github.com/elastic/kibana/pull/181350","mergeCommit":{"message":"[Search]
Adds telemetry warning rules and fixes a bunch of telemetry issues
(#181350)\n\n## Summary\r\n\r\nEnables telemetry warning eslint
rule.\r\n\r\nFixes a bunch of missing telemetry strings.\r\n\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [ ] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"0a4081e1d81eb980998585144c85ac8fbd078f8c"}},"sourceBranch":"main","suggestedTargetBranches":["8.14"],"targetPullRequestStates":[{"branch":"8.14","label":"v8.14.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.15.0","branchLabelMappingKey":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/181350","number":181350,"mergeCommit":{"message":"[Search]
Adds telemetry warning rules and fixes a bunch of telemetry issues
(#181350)\n\n## Summary\r\n\r\nEnables telemetry warning eslint
rule.\r\n\r\nFixes a bunch of missing telemetry strings.\r\n\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [ ] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"0a4081e1d81eb980998585144c85ac8fbd078f8c"}}]}]
BACKPORT-->
Co-authored-by: Efe Gürkan YALAMAN <efeguerkan.yalaman@elastic.co>
Fixes https://github.com/elastic/kibana/issues/176420
## 🍒 Summary
This PR copies the SLO code that was inside the Observability app into
its own app under `observability-solution/slo` folder.
4f6b8dfb-9612-4d30-ad50-4ee5c55a9c32
## ✔️ Acceptance criteria
- URL of new app: `app/slos`
- Design and functionality are not changed.
- Git history has been retained for all files in
`x-pack/plugins/observability_solution/slo`.
- SLO should appear on server less
- SLO code inside `observability_solution/observability` code has been
removed. A new clean up round might be needed though for possible
leftovers.
- Burn rate rule is registered within the new slo app
- SLO embeddables are moved inside the new slo app
- overview
- alerts embeddable
- error budget burn down
- Alerts table configuration registration for slo details page and
alerts table embeddable is still done in the observability app. Response
Ops team is working on removing the need to register the alert table
anyway
- Slo app is wrapped into `ApplicationUsageTrackingProvider` which will
send slo `Application usage` information tracked by the `slo` appId
- Redirect old `app/observability/slos` route to `app/slos`
- Rename old `xpack.observability.slo` keys to `xpack.slo` in the
translation files
## 🌮 How to test
Design and functionality didn't change, so simply navigate to existing
slo pages and try to break it
- Slo list page
- group by
- unified search
- toggle buttons
- actions
- Slo creation
- try group by as well
- Slo detail page
- Actions on top
- navigate to overview and alerts tabs
- Create SLO flyout in Logs Explorer
- Create burn rate rules and verify they appear on rules page
- Verify SLO alerts appear on Alerts page and slo details page
- Embeddables
- Through the dashboard app
- Using the attach to dashboard action on the slo card item on slo list
page and the error budget burn down chart on the slo detail page
- SLOs only for platinum users
- Permissions
- Spaces
## TODO
- [x] Move slo stuff from observability folder to new slo plugin
- [x] Remove old slo stuff from observability folder
- [x] Update references
- [x] Fix typescript and eslint errors
- [x] Paths
- [x] Locators
- [x] Burn rate rule registration
- [x] Embeddable Alerts table configuration registration
- [x] Embeddables
- [x] Translations
- [x] Verify plugin.ts files contain all registration logic
- [x] public
- [x] server
- [x] Final cleanup for observability folder
- [x] Run tests
- [x] Application Usage (Telemetry)
- [x] Permissions
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: shahzad31 <shahzad31comp@gmail.com>
Co-authored-by: Coen Warmer <coen.warmer@gmail.com>
## Summary
issue https://github.com/elastic/security-team/issues/7382
### Data Stream Adapter
This PR introduces the `@kbn/data-stream-adapter` package, which is a
utility library to facilitate Data Stream creation and maintenance in
Kibana, it was inspired by the data stream implementation in the Alerts
plugin.
The library has two exports:
- `DataStreamSpacesAdapter`: to manage space data streams. It uses the
`name-of-the-data-stream-<spaceId>` naming pattern.
- `DataStreamAdapter`: to manage single (not space-aware) data streams.
Usage examples in the package
[README](450be0369d/packages/kbn-data-stream-adapter/README.md)
### Data Quality Dashboard
The `DataStreamSpacesAdapter` has been integrated into the data quality
dashboard to store all the quality checks users perform. The information
stored is the metadata (also used for telemetry) and the actual data
rendered in the tables.
FieldMap definition
[here](450be0369d/x-pack/plugins/ecs_data_quality_dashboard/server/lib/data_stream/results_field_map.ts)
### Demo
311a0bf5-004b-46d7-8140-52a233361c91
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Philippe Oberti <philippe.oberti@elastic.co>
Co-authored-by: Garrett Spong <spong@users.noreply.github.com>
Co-authored-by: Efe Gürkan YALAMAN <efeguerkan.yalaman@elastic.co>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Co-authored-by: Sander Philipse <94373878+sphilipse@users.noreply.github.com>
Co-authored-by: JD Kurma <JDKurma@gmail.com>
Co-authored-by: Jan Monschke <jan.monschke@elastic.co>
Co-authored-by: Patryk Kopyciński <contact@patrykkopycinski.com>
Co-authored-by: Khristinin Nikita <nikita.khristinin@elastic.co>
Co-authored-by: Marco Liberati <dej611@users.noreply.github.com>
Co-authored-by: Julia Rechkunova <julia.rechkunova@elastic.co>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: Davis McPhee <davis.mcphee@elastic.co>
Co-authored-by: Eyo O. Eyo <7893459+eokoneyo@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Marta Bondyra <4283304+mbondyra@users.noreply.github.com>
Co-authored-by: Søren Louv-Jansen <soren.louv@elastic.co>
Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
Co-authored-by: Candace Park <56409205+parkiino@users.noreply.github.com>
## Summary
Refactoring general ui service to a kbn package.
Resolves an [Appex QA](https://github.com/elastic/appex-qa-team) issue.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## [Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts
This PR implements _Retrieval Augmented Generation_ (RAG) for Alerts in the Security Solution. This feature enables users to ask the assistant questions about the latest and riskiest open alerts in their environment using natural language, for example:
- _How many alerts are currently open?_
- _Which alerts should I look at first?_
- _Did we have any alerts with suspicious activity on Windows machines?_
### More context
Previously, the assistant relied solely on the knowledge of the configured LLM and _singular_ alerts or events passed _by the client_ to the LLM as prompt context. This new feature:
- Enables _multiple_ alerts to be passed by the _server_ as context to the LLM, via [LangChain tools](https://github.com/elastic/kibana/pull/167097)
- Applies the user's [anonymization](https://github.com/elastic/kibana/pull/159857) settings to those alerts
- Only fields allowed by the user will be sent as context to the LLM
- Users may enable or disable anonymization for specific fields (via settings)
- Click the conversation's `Show anonymized` toggle to see the anonymized values sent to / received from the LLM:
### Settings
This feature is enabled and configured via the `Knowledge Base` > `Alerts` settings in the screenshot below:
- The `Alerts` toggle enables or disables the feature
- The slider has a range of `10` - `100` alerts (default: `20`)
When the setting above is enabled, up to `n` alerts (as determined by the slider) that meet the following criteria will be returned:
- the `kibana.alert.workflow_status` must be `open`
- the alert must have been generated in the last `24 hours`
- the alert must NOT be a `kibana.alert.building_block_type` alert
- the `n` alerts are ordered by `kibana.alert.risk_score`, to prioritize the riskiest alerts
### Feature flag
To use this feature:
1) Add the `assistantRagOnAlerts` feature flag to the `xpack.securitySolution.enableExperimental` setting in `config/kibana.yml` (or `config/kibana.dev.yml` in local development environments), per the example below:
```
xpack.securitySolution.enableExperimental: ['assistantRagOnAlerts']
```
2) Enable the `Alerts` toggle in the Assistant's `Knowledge Base` settings, per the screenshot below:
## How it works
- When the `Alerts` settings toggle is enabled, http `POST` requests to the `/internal/elastic_assistant/actions/connector/{id}/_execute` route include the following new (optional) parameters:
- `alertsIndexPattern`, the alerts index for the current Kibana Space, e.g. `.alerts-security.alerts-default`
- `allow`, the user's `Allowed` fields in the `Anonymization` settings, e.g. `["@timestamp", "cloud.availability_zone", "file.name", "user.name", ...]`
- `allowReplacement`, the user's `Anonymized` fields in the `Anonymization` settings, e.g. `["cloud.availability_zone", "host.name", "user.name", ...]`
- `replacements`, a `Record<string, string>` of replacements (generated on the server) that starts empty for a new conversation, and accumulates anonymized values until the conversation is cleared, e.g.
```json
"replacements": {
"e4f935c0-5a80-47b2-ac7f-816610790364": "Host-itk8qh4tjm",
"cf61f946-d643-4b15-899f-6ffe3fd36097": "rpwmjvuuia",
"7f80b092-fb1a-48a2-a634-3abc61b32157": "6astve9g6s",
"f979c0d5-db1b-4506-b425-500821d00813": "Host-odqbow6tmc",
// ...
},
```
- `size`, the numeric value set by the slider in the user's `Knowledge Base > Alerts` setting, e.g. `20`
- The `postActionsConnectorExecuteRoute` function in `x-pack/plugins/elastic_assistant/server/routes/post_actions_connector_execute.ts` was updated to accept the new optional parameters, and to return an updated `replacements` with every response. (Every new request that is processed on the server may add additional anonymized values to the `replacements` returned in the response.)
- The `callAgentExecutor` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts` previously used a hard-coded array of LangChain tools that had just one entry, for the `ESQLKnowledgeBaseTool` tool. That hard-coded array was replaced in this PR with a call to the (new) `getApplicableTools` function:
```typescript
const tools: Tool[] = getApplicableTools({
allow,
allowReplacement,
alertsIndexPattern,
assistantLangChain,
chain,
esClient,
modelExists,
onNewReplacements,
replacements,
request,
size,
});
```
- The `getApplicableTools` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/index.ts` examines the parameters in the `KibanaRequest` and only returns a filtered set of LangChain tools. If the request doesn't contain all the parameters required by a tool, it will NOT be returned by `getApplicableTools`. For example, if the required anonymization parameters are not included in the request, the `open-alerts` tool will not be returned.
- The new `alert-counts` LangChain tool returned by the `getAlertCountsTool` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/alert_counts/get_alert_counts_tool.ts` provides the LLM the results of an aggregation on the last `24` hours of alerts (in the current Kibana Space), grouped by `kibana.alert.severity`. See the `getAlertsCountQuery` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/alert_counts/get_alert_counts_query.ts` for details
- The new `open-alerts` LangChain tool returned by the `getOpenAlertsTool` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/open_alerts/get_open_alerts_tool.ts` provides the LLM up to `size` non-building-block alerts generated in the last `24` hours (in the current Kibana Space) with an `open` workflow status, ordered by `kibana.alert.risk_score` to prioritize the riskiest alerts. See the `getOpenAlertsQuery` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/open_alerts/get_open_alerts_query.ts` for details.
- On the client, a conversation continues to accumulate additional `replacements` (and send them in subsequent requests) until the conversation is cleared
- Anonymization functions that were only invoked by the browser were moved from the (browser) `kbn-elastic-assistant` package in `x-pack/packages/kbn-elastic-assistant/` to a new common package: `x-pack/packages/kbn-elastic-assistant-common`
- The new `kbn-elastic-assistant-common` package is also consumed by the `elastic_assistant` (server) plugin: `x-pack/plugins/elastic_assistant`
## Summary
This PR adds a bunch of plugins to help manage AI Assistant Management
settings.
It offers a 'selection' plugin inside Stack Management where a user can
select which AI Assistant she wants to manage.
The Security team can hook into this one, so settings for both AI
Assistants can be accessed from inside one place inside Stack
Management.
This PR also adds the plugin to manage settings for the AI Assistant for
Observability, including Knowledge Base management. This plugin is
available both in Stack Management (stateful) and Project Settings
(serverless).
## What it looks like
51392ec5-05c9-4947-9bf2-810d8d0b7525
## Detailed
1. **Adds a Stack Management plugin**
(`/src/plugins/ai_assistant_management/selection`). Its primary function
is to render a selection screen to help users navigate to the settings
plugin for the AI Assistant for a specific solution. This plugin is
displayed in Stack Management, which is only available in stateful
versions of Kibana.
2. **Adds a AI Assistant for Observability Settings plugin**
(`/src/plugins/ai_assistant_management/observability`). This plugin
allows management of specific Observability AI Assistant settings. It is
available in stateful versions of Kibana (via the aforementioned Stack
Management plugin) or in serverless versions via Project Management.
3. **Knowledge Base management for Observability AI Assistant**: The AI
Assistant for Observability Settings plugin has a Knowledge Base tab,
which allows users to add / read / update / delete and bulk import
entries into the Knowledge Base of the Observability AI Assistant.
4. **Moving of KB endpoints in Observability AI Assistant plugin**: KB
endpoints and functions were located in the same folder. As this PR adds
new endpoints for the KB for CRUD operations, it also moves the existing
ones from the function folder into a dedicated one so there's a clearer
distinction between kb and functions.
5. **Adding of GenAI Connector inside Chat Flyout**: If the user has
admin rights, it is possible to set up a GenAI connector from within the
Observability AI Assistant Chat Flyout. This provides a faster and more
seamless onboarding experience. If the user does not, she will be
redirected to the Settings page.
## Bug fixes
* Fixes chat item styling issues (padding, background color).
## How to test
* Check if the Stack Management plugin works on stateful
* Check if the AI Assistant Settings plugin works on stateful +
serverless
* Check if CRUD operations on KB work
* Check if searching on KB entries work
* Check if its possible to navigate to KB tab directly
(`app/management/kibana/aiAssistantManagementObservability?tab=knowledge_base`)
## Todo
- [x] Add sorting to getEntries
- [x] Add params for tab routing
- [x] Add unit tests
- [ ] Add API tests
- [ ] Add fallback for already indexed entries when searching
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes#169733
#### Reporting fix
Change point detection embeddable was incorrectly reporting render
completion. It was relying on the `onLoad` callback from the Lens
embeddable responsible for chart rendering, which only indicates that
data fetching is complete, but not the actual rendering. Current
implementation relies on the `renderComplete` event from each child
embeddable. Both PNG and PDF exports tested and work as expected.
#### Additional fixes
- Fixes the metric and split field controls states when editing existing
Change point embeddable from a dashboard
- Fixes `filter` query if partitions input is initialized as an empty
array.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
This makes the newly created i18n ESLint rule actually activate on
paths, and narrows the use of the Telemetry ESLint rule to only activate
on .tsx files. This is done because the rule only makes sense in the
context of JSX.
## More words
Noticed that this:
```
'x-pack/plugins/aiops/**/*.{tsx}',
'x-pack/plugins/apm/**/*.{tsx}',
'x-pack/plugins/exploratory_view/**/*.{tsx}',
```
does not work.
This:
```
'x-pack/plugins/aiops/**/*.tsx',
'x-pack/plugins/apm/**/*.tsx',
'x-pack/plugins/exploratory_view/**/*.tsx',
```
does.
# Summary
- Addresses https://github.com/elastic/kibana/issues/161537
## Description
- This PR follows the second option defined in this
[document](https://docs.google.com/document/d/1mqkpjDdFQRFvx_RPvNmjstVj8SXYMr2mrETMv3esda8/edit#heading=h.rpv1zyeb04ay)
the [Mocha tagging ](https://github.com/mochajs/mocha/wiki/Tagging)
- It introduces a new folder
`x-pack/test/security_solution_api_integration` which will serve as a
centralized location to meet all the requirements related to renaming
tests appropriately and grouping similar tests together. It will
facilitate the management of tests that must be run in Serverless and
ESS environments.
- Within this folder, there is a "config" subdirectory that stores base
configurations specific to both the
[Serverless](https://github.com/elastic/kibana/pull/166755/files#diff-afe1f42d5ac2006de8dc09069448b9e8734a6a950586376cd6e8eeb9110ab5f1R1)
and
[ESS](https://github.com/elastic/kibana/pull/166755/files#diff-4a60bd8c91da08a3f7ec14bf3bfef8449af155611374c32579b0318da03e292cR1)
environments, These configurations build upon the base configuration
provided by test_serverless and api_integrations, incorporating
additional settings such as environment variables and tagging options.
- It demonstrates scenarios involving `@ess`, `@serverless`, and
`@brokenInServerless`.
- The file`
x-pack/test/security_solution_api_integration/test_suites/detections_response/rule_creation/create_rules.ts`
is functional in both **Serverless** and **ESS**. However, some tests
related to roles are currently skipped for Serverless until they are
resolved, and these tests are tagged with `@brokenInServerless`
## CI
- It includes a new entry in the ftr_configs.yml to execute the newly
added tests in the pipeline.
- It involves the addition of mochaOptions in both
serverless/config.base.ts and ess/config.base.ts. In the case of
serverless, it includes **@serverless** while excluding
@brokenInServerless. Similarly, for **ess**, it includes @ess and
excludes **@brokenInEss**.
from `x-pack/test/security_solution_api_integration/config/serverless`
## Update in x-pack/test/detection_engine_api_integration
- The `create_rules.ts` and `create_rule_exceptions` files have been
relocated from
`x-pack/test/detection_engine_api_integration/security_and_spaces/group1`
to their respective domains within the
`x-pack/test/security_solution_api_integration` folder.
- The util files now are copied over from the old folder
`x-pack/test/detection_engine_api_integration` to the new folder and
will be removed once all tests are moved to the new folder to don't
break the existing tests
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Implements #153108.
This enables the
`@kbn/telemetry/event_generating_elements_should_be_instrumented` eslint
rule for the `aiops` plugin to enforce `data-test-subj` attributes on
actionable EUI components so they are auto-instrumented by telemetry.
The ids were first auto-created using `node scripts/eslint --fix
x-pack/plugins/aiops` and then adapted.
### What this PR changes
branched from elastic/kibana/pull/163759
- Introduces new AppFeatures package `@kbn/security-solution-features`
with the common logic and `AppFeatureService` to apply offering specific
configurations for Security Solution features independently for
Serverless and ESS. This logic is replacing the earlier `AppFeatures` in
order to introduce new Kibana feature privileges for serverless PLIs so
that new Kibana privileges introduced for serverless PLIs do not
affect/show up as new Kibana feature privileges in ESS.
- Gates endpoint exceptions on alerts/rules based on serverless PLI
configurations. On serverless `Endpoint exceptions` should be
accessible/seen only on endpoint essentials/complete.
New AppFeatures logic architecture diagram:
**Note:** Corresponding API changes related to endpoint exceptions will
be in a new PR, along with the last set of UX changes for hiding the
`Endpoint exceptions` tab from the Rules details page.
### How to review
- Setup for _Servlerless_
- Run `yarn es snapshot` on a terminal window to start ES.
- Copy `config/serverless.security.yml` to
`config/serverless.security.dev.yml`
- Run `yarn serverless-security --no-base-path` on another terminal
window to start kibana in serverless mode
- Run `node
x-pack/plugins/security_solution/scripts/endpoint/endpoint_agent_emulator.js
--asSuperuser` on a new window and then select `1` to `Load Endoints`
and then `1` to `Run` the loader script. This will load some fake
agents/alerts data to test with.
### Tests (Serverless)
- with
`{ product_line: 'security', product_tier: 'essentials' }` or `{
product_line: 'security', product_tier: 'complete' }`
and
`{ product_line: 'endpoint', product_tier: 'essentials' }` or `{
product_line: 'endpoint', product_tier: 'complete' }`
1. Navigate to Rules>Shared exception lists via
`http://localhost:5601/app/security/exceptions`
2. Test that you can see `Endpoint Security Exception List` card on the
shared exception lists page.
3. Navigate to `Alerts` page via `app/security/alerts`, you should see
endpoint alerts. If not, then click on `Manage Rules` and then
disable/enable `Endpoint Security` rules. That should trigger alerts to
show up on the Alerts table.
4. Click on `View Details` button under `Actions` column. Once the
flyout is visible, click on `Take Action` and verify that `Add Endpoint
exception` is visible/enabled/clickable on the menu.
5. Click on `More actions` button under `Actions` column and verify that
`Add Endpoint exception` is visible/enabled/clickable on the menu.
6. Click on `Investigate in timeline` button under `Actions` column;
when the timeline view is visible and the alert item is displayed, click
on buttons mentioned in 4. and 5. above and verify the same.
7. Navigate to `Rules`>`DetectionRules`>`Endpoint Security` rule under
the `Rules` table. Select the `Alerts` tab.
8. Click and verify `View details`,`More actions` and `Investigate in
timeline` buttons same as in 4., 5., 6. above.
9. You should be able to see the `Endpoint exceptions` tab as well.
Click and verify that you can see the tab's content.
- with
`{ product_line: 'security', product_tier: 'essentials' }` or `{
product_line: 'security', product_tier: 'complete' }`
1. Edit `config/serverless.security.dev.yml` so that `endpoint` product
line item is commented out.
2. Test that you can not see `Endpoint Security Exception List` card on
the shared exception lists page.
3. Items 4. 5. 6. as above but the menu items should be disabled. This
can be verified with fake data only as with a real endpoint, endpoint
alerts are actually not visible at all.
### Tests (ESS)
On the ESS side, endpoint exceptions are not affected by this change and
work as usual based on index privileges.
---------
Co-authored-by: semd <sergi.massaneda@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: YulNaumenko <jo.naumenko@gmail.com>
Co-authored-by: Pablo Neves Machado <pablo.nevesmachado@elastic.co>
Co-authored-by: Pablo Machado <machadoum@gmail.com>
## [Security Solution] [Elastic AI Assistant] LangChain integration (experimental)
This PR integrates [LangChain](https://www.langchain.com/) with the [Elastic AI Assistant](https://www.elastic.co/blog/introducing-elastic-ai-assistant) as an experimental, alternative execution path.
### How it works
- There are virtually no client side changes to the assistant, apart from a new branch in `x-pack/packages/kbn-elastic-assistant/impl/assistant/api.tsx` that chooses a path based on the value of the `assistantLangChain` flag:
```typescript
const path = assistantLangChain
? `/internal/elastic_assistant/actions/connector/${apiConfig?.connectorId}/_execute`
: `/api/actions/connector/${apiConfig?.connectorId}/_execute`;
```
Execution of the LangChain chain happens server-side. The new route still executes the request via the `connectorId` in the route, but the connector won't execute the request exactly as it was sent by the client. Instead, the connector will execute one (or more) prompts that are generated by LangChain.
Requests routed to `/internal/elastic_assistant/actions/connector/${apiConfig?.connectorId}/_execute` will be processed by a new Kibana plugin located in:
```
x-pack/plugins/elastic_assistant
```
- Requests are processed in the `postActionsConnectorExecuteRoute` handler in `x-pack/plugins/elastic_assistant/server/routes/post_actions_connector_execute.ts`.
The `postActionsConnectorExecuteRoute` route handler:
1. Extracts the chat messages sent by the assistant
2. Converts the extracted messages to the format expected by LangChain
3. Passes the converted messages to `executeCustomLlmChain`
- The `executeCustomLlmChain` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts`:
1. Splits the messages into `pastMessages` and `latestMessage`, where the latter contains only the last message sent by the user
2. Wraps the conversation history in the `BufferMemory` LangChain abstraction
3. Executes the chain, kicking it off with `latestMessage`
```typescript
const llm = new ActionsClientLlm({ actions, connectorId, request });
const pastMessages = langchainMessages.slice(0, -1); // all but the last message
const latestMessage = langchainMessages.slice(-1); // the last message
const memory = new BufferMemory({
chatHistory: new ChatMessageHistory(pastMessages),
});
const chain = new ConversationChain({ llm, memory });
await chain.call({ input: latestMessage[0].content }); // kick off the chain with the last message
};
```
- When LangChain executes the chain, it will invoke `ActionsClientLlm`'s `_call` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/llm/actions_client_llm.ts` one or more times.
The `_call` function's signature is defined by LangChain:
```
async _call(prompt: string): Promise<string>
```
- The contents of `prompt` are completely determined by LangChain.
- The string returned by the promise is the "answer" from the LLM
The `ActionsClientLlm` extends LangChain's LLM interface:
```typescript
export class ActionsClientLlm extends LLM
```
This let's us do additional "work" in the `_call` function:
1. Create a new assistant message using the contents of the `prompt` (`string`) argument to `_call`
2. Create a request body in the format expected by the connector
3. Create an actions client from the authenticated request context
4. Execute the actions client with the request body
5. Save the raw response from the connector, because that's what the assistant expects
6. Return the result as a plain string, as per the contact of `_call`
## Desk testing
This experimental LangChain integration may NOT be enabled via a feature flag (yet).
Set
```typescript
assistantLangChain={true}
```
in `x-pack/plugins/security_solution/public/app/app.tsx` to enable this experimental feature in development environments.
## Summary
Implementation of serverless-specific pages within the Unified IA
Navigation.
#### Links implemented:
- `Machine Learning`
- Landing page created on serverless only
- All links in the landing page go to `/ml` app
- `Dev Tools`
- Links directly to `/dev_tools` app
#### Links not implemented:
```// TODO: in a follow-up PR```
- Project Settings
- Change the _Settings_ name by _Project Settings_
- Modify the landing page items according to the design
## Changes
### Plugin contract changes
The Machine Learning landing page is the first page that is only available on serverless and should not exist in ess (there are more of this kind in the pipeline), so this PR implements the foundations to enable the _security_solution_serverless_ plugin to implement its own page components, configure the link definition and create new routes to render them in the Security Solution application.
These new APIs can be called from either `security_solution_serverless` or `security_solution_ess`, allowing those plugins to have their own offering-specific pages.
The new APIs exposed in the security_solution public contract are the following:
- `extraAppLinks$`: Observable to add extra app_links into the application links configuration, so they are stored and included in the SecuritySolution plugin `deepLinks` registry, to make them accessible from anywhere in the application using the `chrome.navLinks` API.
- `extraRoutes$`: Observable to add extra routes into the main Router, so it can render the new page components. These additional routes are appended after the "sub-plugin" (_alerts_, _timeline_, ...) routes, so it is not possible to override an existing route path.
### New `security-solution-navigation` package
Since now we need to use the same navigation components and hooks in different plugins, these functionalities have been extracted to the `@kbn/security-solution-navigation` package, which all Security plugins will depend on (generic, serverless, and ess).
The modules exposed by this package have been extracted from the main security_solution plugin and standardized. They include the Landing pages components (new [storybook](https://ci-artifacts.kibana.dev/storybooks/pr-161667/394abe76676c6a76b2982c1d3f5bb675739c3477/security_solution_packages/index.html?path=/story/landing-links-landing-links-icons-categories--landing-links-icons-categories) available), navigation hooks, and link utilities. Also, some types and constants have been moved to this package.
A new context provider has also been created, which needs to be in place in order to use this package. The `<NavigationProvider core={core}>` is required for the package functionalities to have access to the Kibana core navigation APIs: `navigateToUrl`, `navigateToApp`, and `getUrlForApp`.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: YulNaumenko <jo.naumenko@gmail.com>
## Summary
closes: https://github.com/elastic/kibana/issues/159685
- Renaming _x-pack/plugins_:
`serverless_security` -> `security_solution_serverless`
`ess_security` -> `security_solution_ess`
- All the related configurations and types have also been renamed.
- i18n translation prefixes updated
- relocation of internal `security_solution_serverless` directories to
be consistent with `security_solution_ess`
### Eslint
I also added the plugins in the `.eslintrc` configuration, defining the
same rules as the `security_solution` plugin.
All eslint errors have been addressed (mainly _type_ imports errors)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
As part of the actions for making Profiling production ready, this PR
adds basic API tests on the Profiling APIs checking if only users with
`access:profiling` are allowed to call our APIs, other users must be
forbidden.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Why?
To simplify the process of migration to react-router@6.
https://github.com/remix-run/react-router/discussions/8753
What problems exactly it solves?
- In my previous PR I added `CompatRouter`
https://github.com/elastic/kibana/pull/159173, which caused changes in
~50 files and pinged 15 Teams. And this is just meant to be a temporary
change, so when we're done with the migration I would have to revert
these changes and engage everyone to review the PR again. And it is just
a single step in the migration strategy. So to make our lives easier I
think it would be better to have a common place where we do import our
router components because it will allow us to surface some extra logic
in single place instead of going through the whole source code again.
- `react-router@6` doesn't support a custom `Route` component, so that
means our custom `Route` component that we're using almost everywhere
today, will need to be replaced by a different solution. I have decided
to add `Routes` component, which will be responsible for rendering the
proper component (`react-router@6` renamed `Switch` to `Routes`, so I
have named this component to align with the dictionary of the new
router) and also is going to add the logic that today is done in `Route`
(moving logic to `Routes` will be done in the follow-up PR, here I just
wanted to focus on using the common router components to make the review
process easier)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## [Security Solution] Elastic Security Assistant
The _Elastic Security Assistant_ has entered the chat, integrating generative AI and large language models (LLMs) into the workflows of Elastic Security users.
Bring your alerts, events, rules, and data quality checks into the conversation.
<31d65c78-5692-4817-b726-820c5df0801c>
This PR merges a feature branch developed by @spong and @andrew-goldstein , seeded by @jamesspi 's prototype of the assistant. Connectivity to LLMs is provided the [Generative AI Connector](<https://github.com/elastic/kibana/pull/157228>) , developed by @stephmilovic . This PR includes:
- A new reusable Kibana package containing the assistant: `x-pack/packages/kbn-elastic-assistant`
- See the `How to embed the Assistant in other parts of Kibana` for details
- Assistant integration into Elastic Security Solution workflows (e.g. alerts, cases, Timeline, rules, data quality)
### An assistant trained on the Elastic stack and Elastic Security
The [Generative AI Connector](<https://github.com/elastic/kibana/pull/157228>) connects the assistant to OpenAI and Azure OpenAI models trained with knowledge of the Elastic stack and the Elastic Security solution, including:
- The Elastic open [Detection Rules](https://github.com/elastic/detection-rules)
- The [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current/index.html)
- Elastic query languages, including [KQL](https://www.elastic.co/guide/en/kibana/current/kuery-query.html), [EQL](https://www.elastic.co/guide/en/elasticsearch/reference/current/eql-syntax.html), and the [Elastic Query DSL](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html)
- [Elasticsearch API documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/rest-apis.html)
This training enables the assistant to offer fully interactive chat experiences that include:
- alert summarization
- interactive query generation
- workflow suggestions
- generating ingestion configurations that conform to the Elastic Common Schema
- your imagination
using context from Elastic Security.
### Take action from your conversations
The Actions (from assistant response):
- Send KQL to Timeline
- Send EQL to Timeline
- Send Elasticsearch DSL to Timeline
- Send Note to timeline
- Create new case
- Add to existing case
- Copy to clipboard
### Components architecture diagram
### How to embed the Assistant in other parts of Kibana
Follow the general instructions in `x-pack/packages/kbn-elastic-assistant/index.ts` to integrate the assistant into a Kibana app.
#### Step 1 - Wrap your Kibana app in the `AssistantProvider` component
```ts
// Step 1: Wrap your Kibana app in the `AssistantProvider` component. This typically
// happens in the root of your app. Optionally provide a custom title for the assistant:
/** provides context (from the app) to the assistant, and injects Kibana services, like `http` */
export { AssistantProvider } from './impl/assistant_context';
```
#### Step 2: Add the `AssistantOverlay` component to your app
```ts
// Step 2: Add the `AssistantOverlay` component to your app. This component displays the assistant
// overlay in a modal, bound to a shortcut key:
/** modal overlay for Elastic Assistant conversations */
export { AssistantOverlay } from './impl/assistant/assistant_overlay';
// In addition to the `AssistantOverlay`, or as an alternative, you may use the `Assistant` component
// to display the assistant without the modal overlay:
/** this component renders the Assistant without the modal overlay to, for example, render it in a Timeline tab */
export { Assistant } from './impl/assistant';
```
#### Step 3: Wherever you want to bring context into the assistant, use the any combination of the following
```ts
// Step 3: Wherever you want to bring context into the assistant, use the any combination of the following
// components and hooks:
// - `NewChat` component
// - `NewChatById` component
// - `useAssistantOverlay` hook
/**
* `NewChat` displays a _New chat_ icon button, providing all the context
* necessary to start a new chat. You may optionally style the button icon,
* or override the default _New chat_ text with custom content, like `🪄✨`
*
* USE THIS WHEN: All the data necessary to start a new chat is available
* in the same part of the React tree as the _New chat_ button.
*/
export { NewChat } from './impl/new_chat';
/**
* `NewChatByID` displays a _New chat_ icon button by providing only the `promptContextId`
* of a context that was (already) registered by the `useAssistantOverlay` hook. You may
* optionally style the button icon, or override the default _New chat_ text with custom
* content, like {'🪄✨'}
*
* USE THIS WHEN: all the data necessary to start a new chat is NOT available
* in the same part of the React tree as the _New chat_ button. When paired
* with the `useAssistantOverlay` hook, this option enables context to be be
* registered where the data is available, and then the _New chat_ button can be displayed
* in another part of the tree.
*/
export { NewChatById } from './impl/new_chat_by_id';
/**
* `useAssistantOverlay` is a hook that registers context with the assistant overlay, and
* returns an optional `showAssistantOverlay` function to display the assistant overlay.
* As an alterative to using the `showAssistantOverlay` returned from this hook, you may
* use the `NewChatById` component and pass it the `promptContextId` returned by this hook.
*
* USE THIS WHEN: You want to register context in one part of the tree, and then show
* a _New chat_ button in another part of the tree without passing around the data, or when
* you want to build a custom `New chat` button with features not not provided by the
* `NewChat` component.
*/
export { useAssistantOverlay } from './impl/assistant/use_assistant_overlay';
```
Co-authored-by: Garrett Spong <garrett.spong@elastic.co>
Co-authored-by: Andrew Macri <andrew.macri@elastic.co>
## Summary
This PR adds boilerplate code and a few initial end-to-end tests to
serverless plugins.
Note that the tests defined in this PR are not part of any CI run yet,
this will be done in a follow-up after this PR is merged.
### Details
The serverless test structure corresponds to what we have in
`x-pack/test` with API tests in `api_integration` and UI tests in
`functional`, each with their set of helper methods and sub-directories
for
- `common` functionality shared across serverless projects (core, shared
UX, ...)
- `observability` project specific functionality
- `search` project specific functionality
- `security` project specific functionality
The `shared` directory contains fixtures, services, ... that are shared
across `api_integration` abd `functional` tests.
```
x-pack/test_serverless/
├─ api_integration
│ ├─ services
│ ├─ test_suites
│ │ ├─ common
│ │ ├─ observability
│ │ ├─ search
│ │ ├─ security
├─ functional
│ ├─ page_objects
│ ├─ services
│ ├─ test_suites
│ │ ├─ common
│ │ ├─ observability
│ │ ├─ search
│ │ ├─ security
├─ shared
│ ├─ services
│ ├─ types
```
See also `x-pack/test_serverless/README.md`
### Run tests
Similar to how functional tests are run in `x-pack/test`, you can point
the functional tests server and test runner to config files in this
`x-pack/test_serverless` directory, e.g. from the `x-pack` directory
run:
```
node scripts/functional_tests_server.js --config test_serverless/api_integration/test_suites/common/config.ts
```
and
```
node scripts/functional_test_runner.js --config test_serverless/api_integration/test_suites/common/config.ts
```
### Additional changes
- The stateful `common_page` page object used the existence of the
global nav to determine `isChromeVisible` and `isChromeHidden`, which is
not working when the global nav is disabled. To solve this, a
`data-test-subj` that indicates the chrome visible state is added to the
Kibana app wrapper and is used for the checks.
- Add a few `data-test-subj` entries to the Observability overview page.
- Add optional `dataTestSubj` to the `Navigation` component and use that
for the serverless search nav.
- Add optional `titleDataTestSubj` to the `SolutionNav` component and
use it for the serverless security nav.
- Add a data-test-subj entry to the Search overview page.
## Summary
closes: https://github.com/elastic/kibana/issues/155301
Moves _packages/security-solution/_ to
_x-pack/packages/security-solution/_
Moves _x-pack/packages/kbn-securitysolution-*_ into the new
_x-pack/packages/security-solution/_
It contains 3 packages now:
- data_view/
- ecs_data_quality_dashboard/
- side_nav/
Package names and ids have not changed.
## Other
- eslint configured for all the packages in the directory
- i18n prefix `securitySolutionPackages` configured for all packages in
the directory
- generic storybook configuration, run with: `yarn storybook
security_solution_packages`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Resolves https://github.com/elastic/kibana/issues/144887
## Summary
This PR adds an ESLint Plugin which checks specific `Eui` elements for
the existence of a `data-test-subj` prop. This rule will make having one
for these elements required.
This rule is currently only enabled for Observability apps (APM, Infra,
Observability, Synthetics, Uptime).
The plugin is also able to generate a suggestion based on the context in
which the element is used. In the IDE this suggestion can be applied by
using the autofix capability (see video below).
When opening a PR, the CI will automatically apply the suggestion to
qualifying Eui elements in the branch.
https://user-images.githubusercontent.com/535564/225449622-bbfccb40-fdd2-4f69-9d5a-7d5a97bf62e6.mov
## Why do this?
There is an increased push to move towards data driven feature
development. In order to facilitate this, we need to have an increased
focus on instrumenting user event generating elements in the Kibana
codebase. This linting rule is an attempt to nudge Kibana engineers to
not forget to add this property when writing frontend code. It also
saves a bit of work for engineers by suggesting a value for the
`data-test-subj` based on the location of the file in the codebase and
any potential default values that might be present in the JSX node tree.
Finally, because the suggestion is always of the same form, it can
increase the consistency in the values given to these elements.
## Shape of the suggestion
The suggestion for the value of data-test-subj is of the form:
`[app][componentName][intent][euiElementName]`.
For example, when working in a component in the location:
`x-pack/plugins/observability/public/pages/overview/containers/overview_page/header_actions.tsx`,
and having the code:
```
function HeaderActions() {
return (
<EuiButton>{i18n.translate('id', { defaultMessage: 'Submit Form' })}</EuiButton>
)
}
```
the suggestion becomes:
`data-test-subj=o11yHeaderActionsSubmitFormButton`.
For elements that don't take a `defaultMessage` prop / translation, the
suggestion takes the form: `[app][componentName][euiElementName]`
## Which elements are checked by the ESLint rule?
In its current iteration the rule checks these Eui elements:
* `EuiButton`
* `EuiButtonEmpty`
* `EuiLink`
* `EuiFieldText`
* `EuiFieldSearch`
* `EuiFieldNumber`
* `EuiSelect`
* `EuiRadioGroup`
* 'EuiTextArea`
## What types of prop setting does this rule support?
* `<EuiButton data-test-subj="foo">` (direct prop)
* `<EuiButton {...foo}>` (via spreaded object; rule checks for
`data-test-subj` key in object)
## What types of function declarations does this rule support?
* `function Foo(){}` (Named function)
* `const Foo = () => {}` (Arrow function assigned to variable)
* `const Foo = memo(() => {})` (Arrow function assigned to variable
wrapped in function)
* `const Foo = hoc(uponHoc(uponHoc(() => {})))` (Arrow function assigned
to variable wrapped in infinite levels of functions)
## Things to note
* If an element already has a value for `data-test-subj` the rule will
not kick in as any existing instrumentation might depend on the value.
* the auto suggestion is just a suggestion: the engineer can always
adjust the value for a `data-test-subj` before or after committing. Once
a value is present (autofixed or manually set) the rule will not kick
in.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Dario Gieselaar <d.gieselaar@gmail.com>
Co-authored-by: Katerina Patticha <kate@kpatticha.com>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Fixes#110422
TL;DR: The `lodash.set` function is unsafe and shouldn't be called.
Cause of error: If you specify multiple `no-restricted-imports` paths
for the same module, only the last path is used. Instead you need to
combine them into a single path as I've done in this PR.
This regression was introduced in #100277
