Closes#161002Closes#170073
## Summary
This PR implements a createModelVersion API in the Encrypted Saved
Objects plugin to support upward migrations for model version encrypted
saved objects.
Much like how the `createMigration` API provided a way to wrap migration
functions to support migration of encrypted saved objects prior to the
model version paradigm, the new `createModelVersion` API provides a way
to wrap a model version definition for the same purpose.
`createModelVersion` manipulates the changes defined for a model version
('unsafe_transform', 'data_backfill', 'data_removal'), merging them into
a single transform function in which the saved object document is
decrypted, transformed, and then encrypted again. The document is
decrypted with the `encrypted saved object type registration` provided
by the required `inputType` parameter. Similarly, the document is by
encrypted with the `encrypted saved object type registration` provided
by the required `outputType` parameter.
An example plugin (`examples/eso_model_version_example`) provides a
demonstration of how the createModelVersion API should be used. The UI
of the example plugin gives an idea of what the encrypted saved objects
look like before and after the model version changes are applied.
## Testing
### Manual Testing
- Modify the example plugin implementation in
`examples/eso_model_version_example` to include different changes or
additional model versions.
### Unit Tests
-
`x-pack/plugins/encrypted_saved_objects/server/create_model_version.test.ts`
### Functional Tests
-
`x-pack/test/encrypted_saved_objects_api_integration/tests/encrypted_saved_objects_api.ts`
-
`x-pack/test/encrypted_saved_objects_api_integration/tests/encrypted_saved_objects_decryption.ts`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Part of https://github.com/elastic/kibana/issues/171570
PR breaks EsError out of data plugin and into @kbn/search-errors
package. This will allow for future efforts to move [lens error
handing](https://github.com/elastic/kibana/blob/main/x-pack/plugins/lens/public/editor_frame_service/error_helper.tsx)
into this package and more closely align error handling across kibana.
An unplanned part of this PR was moving `BfetchRequestError` into its
own package `@kbn/bfetch`. This is required because `@kbn/search-errors`
package can not import code from a plugin. `BfetchRequestError` could
not be part of `@kbn/search-errors` because that created a circular
dependency graph.
Another unexpected problem found while working on this PR is that the
package had to be included in `ui-shared-deps`. The reason for this is
that `renderSearchError` uses `instanceof EsError` check to determine if
an error is an EsError. This check failed when the package was not
included in `ui-shared-deps` because each plugin got its own copy of
`kbn/search-error` plugin. This meant that `data` plugin had its own
copy of `EsError` class when the error is generated, then `discover`
plugin had its own copy of `EsError` class when the instanceof check
occurs. `instanceof` check unexpectedly returned false in this case.
Moving plugin to `ui-shared-deps` results in a single instance of the
plugin. Thanks @mistic for the help with moving package into
`ui-shared-deps`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Co-authored-by: Davis McPhee <davismcphee@hotmail.com>
## [Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts
This PR implements _Retrieval Augmented Generation_ (RAG) for Alerts in the Security Solution. This feature enables users to ask the assistant questions about the latest and riskiest open alerts in their environment using natural language, for example:
- _How many alerts are currently open?_
- _Which alerts should I look at first?_
- _Did we have any alerts with suspicious activity on Windows machines?_
### More context
Previously, the assistant relied solely on the knowledge of the configured LLM and _singular_ alerts or events passed _by the client_ to the LLM as prompt context. This new feature:
- Enables _multiple_ alerts to be passed by the _server_ as context to the LLM, via [LangChain tools](https://github.com/elastic/kibana/pull/167097)
- Applies the user's [anonymization](https://github.com/elastic/kibana/pull/159857) settings to those alerts
- Only fields allowed by the user will be sent as context to the LLM
- Users may enable or disable anonymization for specific fields (via settings)
- Click the conversation's `Show anonymized` toggle to see the anonymized values sent to / received from the LLM:
### Settings
This feature is enabled and configured via the `Knowledge Base` > `Alerts` settings in the screenshot below:
- The `Alerts` toggle enables or disables the feature
- The slider has a range of `10` - `100` alerts (default: `20`)
When the setting above is enabled, up to `n` alerts (as determined by the slider) that meet the following criteria will be returned:
- the `kibana.alert.workflow_status` must be `open`
- the alert must have been generated in the last `24 hours`
- the alert must NOT be a `kibana.alert.building_block_type` alert
- the `n` alerts are ordered by `kibana.alert.risk_score`, to prioritize the riskiest alerts
### Feature flag
To use this feature:
1) Add the `assistantRagOnAlerts` feature flag to the `xpack.securitySolution.enableExperimental` setting in `config/kibana.yml` (or `config/kibana.dev.yml` in local development environments), per the example below:
```
xpack.securitySolution.enableExperimental: ['assistantRagOnAlerts']
```
2) Enable the `Alerts` toggle in the Assistant's `Knowledge Base` settings, per the screenshot below:
## How it works
- When the `Alerts` settings toggle is enabled, http `POST` requests to the `/internal/elastic_assistant/actions/connector/{id}/_execute` route include the following new (optional) parameters:
- `alertsIndexPattern`, the alerts index for the current Kibana Space, e.g. `.alerts-security.alerts-default`
- `allow`, the user's `Allowed` fields in the `Anonymization` settings, e.g. `["@timestamp", "cloud.availability_zone", "file.name", "user.name", ...]`
- `allowReplacement`, the user's `Anonymized` fields in the `Anonymization` settings, e.g. `["cloud.availability_zone", "host.name", "user.name", ...]`
- `replacements`, a `Record<string, string>` of replacements (generated on the server) that starts empty for a new conversation, and accumulates anonymized values until the conversation is cleared, e.g.
```json
"replacements": {
"e4f935c0-5a80-47b2-ac7f-816610790364": "Host-itk8qh4tjm",
"cf61f946-d643-4b15-899f-6ffe3fd36097": "rpwmjvuuia",
"7f80b092-fb1a-48a2-a634-3abc61b32157": "6astve9g6s",
"f979c0d5-db1b-4506-b425-500821d00813": "Host-odqbow6tmc",
// ...
},
```
- `size`, the numeric value set by the slider in the user's `Knowledge Base > Alerts` setting, e.g. `20`
- The `postActionsConnectorExecuteRoute` function in `x-pack/plugins/elastic_assistant/server/routes/post_actions_connector_execute.ts` was updated to accept the new optional parameters, and to return an updated `replacements` with every response. (Every new request that is processed on the server may add additional anonymized values to the `replacements` returned in the response.)
- The `callAgentExecutor` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts` previously used a hard-coded array of LangChain tools that had just one entry, for the `ESQLKnowledgeBaseTool` tool. That hard-coded array was replaced in this PR with a call to the (new) `getApplicableTools` function:
```typescript
const tools: Tool[] = getApplicableTools({
allow,
allowReplacement,
alertsIndexPattern,
assistantLangChain,
chain,
esClient,
modelExists,
onNewReplacements,
replacements,
request,
size,
});
```
- The `getApplicableTools` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/index.ts` examines the parameters in the `KibanaRequest` and only returns a filtered set of LangChain tools. If the request doesn't contain all the parameters required by a tool, it will NOT be returned by `getApplicableTools`. For example, if the required anonymization parameters are not included in the request, the `open-alerts` tool will not be returned.
- The new `alert-counts` LangChain tool returned by the `getAlertCountsTool` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/alert_counts/get_alert_counts_tool.ts` provides the LLM the results of an aggregation on the last `24` hours of alerts (in the current Kibana Space), grouped by `kibana.alert.severity`. See the `getAlertsCountQuery` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/alert_counts/get_alert_counts_query.ts` for details
- The new `open-alerts` LangChain tool returned by the `getOpenAlertsTool` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/open_alerts/get_open_alerts_tool.ts` provides the LLM up to `size` non-building-block alerts generated in the last `24` hours (in the current Kibana Space) with an `open` workflow status, ordered by `kibana.alert.risk_score` to prioritize the riskiest alerts. See the `getOpenAlertsQuery` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/open_alerts/get_open_alerts_query.ts` for details.
- On the client, a conversation continues to accumulate additional `replacements` (and send them in subsequent requests) until the conversation is cleared
- Anonymization functions that were only invoked by the browser were moved from the (browser) `kbn-elastic-assistant` package in `x-pack/packages/kbn-elastic-assistant/` to a new common package: `x-pack/packages/kbn-elastic-assistant-common`
- The new `kbn-elastic-assistant-common` package is also consumed by the `elastic_assistant` (server) plugin: `x-pack/plugins/elastic_assistant`
## Summary
This PR adds a bunch of plugins to help manage AI Assistant Management
settings.
It offers a 'selection' plugin inside Stack Management where a user can
select which AI Assistant she wants to manage.
The Security team can hook into this one, so settings for both AI
Assistants can be accessed from inside one place inside Stack
Management.
This PR also adds the plugin to manage settings for the AI Assistant for
Observability, including Knowledge Base management. This plugin is
available both in Stack Management (stateful) and Project Settings
(serverless).
## What it looks like
51392ec5-05c9-4947-9bf2-810d8d0b7525
## Detailed
1. **Adds a Stack Management plugin**
(`/src/plugins/ai_assistant_management/selection`). Its primary function
is to render a selection screen to help users navigate to the settings
plugin for the AI Assistant for a specific solution. This plugin is
displayed in Stack Management, which is only available in stateful
versions of Kibana.
2. **Adds a AI Assistant for Observability Settings plugin**
(`/src/plugins/ai_assistant_management/observability`). This plugin
allows management of specific Observability AI Assistant settings. It is
available in stateful versions of Kibana (via the aforementioned Stack
Management plugin) or in serverless versions via Project Management.
3. **Knowledge Base management for Observability AI Assistant**: The AI
Assistant for Observability Settings plugin has a Knowledge Base tab,
which allows users to add / read / update / delete and bulk import
entries into the Knowledge Base of the Observability AI Assistant.
4. **Moving of KB endpoints in Observability AI Assistant plugin**: KB
endpoints and functions were located in the same folder. As this PR adds
new endpoints for the KB for CRUD operations, it also moves the existing
ones from the function folder into a dedicated one so there's a clearer
distinction between kb and functions.
5. **Adding of GenAI Connector inside Chat Flyout**: If the user has
admin rights, it is possible to set up a GenAI connector from within the
Observability AI Assistant Chat Flyout. This provides a faster and more
seamless onboarding experience. If the user does not, she will be
redirected to the Settings page.
## Bug fixes
* Fixes chat item styling issues (padding, background color).
## How to test
* Check if the Stack Management plugin works on stateful
* Check if the AI Assistant Settings plugin works on stateful +
serverless
* Check if CRUD operations on KB work
* Check if searching on KB entries work
* Check if its possible to navigate to KB tab directly
(`app/management/kibana/aiAssistantManagementObservability?tab=knowledge_base`)
## Todo
- [x] Add sorting to getEntries
- [x] Add params for tab routing
- [x] Add unit tests
- [ ] Add API tests
- [ ] Add fallback for already indexed entries when searching
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR:
* extracts `enterprise_search` index documents component to common
package.
* Uses EUI pagination, converting Elasticsearch`from` & `size` to EUI
pagination standard for documents list
### Screen Recording
f585d9cc-f92c-44f4-aead-23c75c107a0f
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Add possibility to Isolate/Release SentinelOne host from Alert details
flyout.
Add support for displaying S1 Agent status in UI.
Add an experimental flag to S1 Connector.
Rename S1 connector actions from `Agent` to `Host`
Add a feature flag to security_solution to control enrollment of this
feature.
Update parallel script to support all FTR config options
Add `cypress-data-session` plugin to allow better caching of test data
(mostly for Dev experience)
Testing instruction:
1. Ensure you have
2. From root Kibana folder run
https://p.elstc.co/paste/URVrCEcR#aG1X9p3BMCRUDY+IzfIg5mGomcTGxwkYO6RGxSIAyWz
3. In Cypress run
```x-pack/plugins/security_solution/public/management/cypress/e2e/sentinelone/isolate.cy.ts```
4. 💚
<img width="2375" alt="Zrzut ekranu 2023-11-15 o 12 38 27"
src="c7ddc20e-9944-452c-b739-fa2d9fbf072b">
<img width="2374" alt="Zrzut ekranu 2023-11-15 o 12 38 32"
src="ab3ced14-0a5c-4f40-a92e-844feb849bb4">
<img width="2370" alt="Zrzut ekranu 2023-11-15 o 12 38 38"
src="96ccd237-56a6-449e-979d-f4fe8ffbe048">
<img width="2373" alt="Zrzut ekranu 2023-11-15 o 12 38 46"
src="924013aa-79ef-405b-ae73-139cf0644ebf">
<img width="2374" alt="Zrzut ekranu 2023-11-15 o 12 39 17"
src="e1ff5b05-8b80-40a9-84b1-dd21bf9e059c">
<img width="2374" alt="Zrzut ekranu 2023-11-15 o 12 39 58"
src="15fc5d36-970f-47cb-ae2f-f8a19628e6f4">
<img width="2374" alt="Zrzut ekranu 2023-11-15 o 12 40 03"
src="5860a0c9-a6e5-43b9-b37d-aa68e4e71f26">
<img width="2373" alt="Zrzut ekranu 2023-11-15 o 12 40 09"
src="5e2c5d41-c96a-4c32-8d51-a8408efea8e3">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Partially resolves: https://github.com/elastic/kibana/issues/164255,
this is 2/3 of the scoped query changes.
Maintenance window scoped query frontend changes. Adds the ability to
add and edit scoped query for maintenance windows. Due to limitations
with the alerts search bar and each solution fetches AAD fields, we only
allow users to associate scoped query with 1 category (manangement,
o11y, or security solution). The intended usage in this case is for the
user to create multiple maintenance windows if they wish to apply scoped
queries to multiple solutions.
### To test:
go to
`x-pack/plugins/alerting/public/pages/maintenance_windows/constants.ts`
and set `IS_SCOPED_QUERY_ENABLED` to `true`
### Scoped query off, multiple category allowed:
### Scoped query on, multiple category disallowed:
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Closes#171613
## Summary
This PR adds the viewInApp URL to the custom threshold rule type. This
URL will send the user to the log explorer with the selected data view
and the rule's query filter. If there is only one document aggregation,
then the filter related to this aggregation will be added as shown
below:
|Rule|Discover with pre-fill data|
|---|---|
||
For the ad-hoc data view, you should be able to see the selected index
pattern in discover similar to this:
<img
src="046493ae-ba59-46b7-a40f-68d1836d43f1"
width=400 />
### 🧪 How to test
- Check the viewInApp URL both in action variables and the alert table
for the following scenarios:
- A rule with a persisted data view
- A rule with an ad-hoc data view
- A rule with count aggregation and filter
- A rule with an optional query filter
- A rule with non-count aggregation
In all the above scenarios, the starting time in the Discover should be
before the alert's start time.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
[](https://renovatebot.com)
This PR contains the following updates:
| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [elastic-apm-node](https://togithub.com/elastic/apm-agent-nodejs) |
[`^4.1.0` ->
`^4.2.0`](https://renovatebot.com/diffs/npm/elastic-apm-node/4.1.0/4.2.0)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
---
### Release Notes
<details>
<summary>elastic/apm-agent-nodejs (elastic-apm-node)</summary>
###
[`v4.2.0`](https://togithub.com/elastic/apm-agent-nodejs/releases/tag/v4.2.0)
[Compare
Source](https://togithub.com/elastic/apm-agent-nodejs/compare/v4.1.0...v4.2.0)
For more information, please see the
[changelog](https://www.elastic.co/guide/en/apm/agent/nodejs/current/release-notes-4.x.html#release-notes-4.2.0).
##### Elastic APM Node.js agent layer ARNs
|Region|ARN|
|------|---|
|af-south-1|arn:aws:lambda:af-south-1:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|ap-east-1|arn:aws:lambda:ap-east-1:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|ap-northeast-1|arn:aws:lambda:ap-northeast-1:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|ap-northeast-2|arn:aws:lambda:ap-northeast-2:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|ap-northeast-3|arn:aws:lambda:ap-northeast-3:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|ap-south-1|arn:aws:lambda:ap-south-1:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|ap-southeast-1|arn:aws:lambda:ap-southeast-1:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|ap-southeast-2|arn:aws:lambda:ap-southeast-2:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|ap-southeast-3|arn:aws:lambda:ap-southeast-3:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|ca-central-1|arn:aws:lambda:ca-central-1:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|eu-central-1|arn:aws:lambda:eu-central-1:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|eu-north-1|arn:aws:lambda:eu-north-1:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|eu-south-1|arn:aws:lambda:eu-south-1:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|eu-west-1|arn:aws:lambda:eu-west-1:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|eu-west-2|arn:aws:lambda:eu-west-2:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|eu-west-3|arn:aws:lambda:eu-west-3:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|me-south-1|arn:aws:lambda:me-south-1:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|sa-east-1|arn:aws:lambda:sa-east-1:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|us-east-1|arn:aws:lambda:us-east-1:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|us-east-2|arn:aws:lambda:us-east-2:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|us-west-1|arn:aws:lambda:us-west-1:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
|us-west-2|arn:aws:lambda:us-west-2:267093732750:layer:elastic-apm-node-ver-4-2-0:1|
</details>
---
### Configuration
📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/elastic/kibana).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy41OS44IiwidXBkYXRlZEluVmVyIjoiMzcuNTkuOCIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==-->
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: pgayvallet <pierre.gayvallet@elastic.co>
This PR upgrades re2 into v1.20.9.
- [x] Manual builds for `linux-arm64`.
- [x] Check if https://github.com/elastic/kibana-custom-nodejs-builds
needs to be run or updated
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Consolidates UI elements and backend code to create/delete data views
and destination indices related to transforms and data frame analytics.
We ended up with two different approaches for creating data views in the
wizards for transforms and data frame analytics, the original reason was
we were not aware of the `allowNoIndex: true` setting and worked around
that in different ways.
This PR aligns UI workflows and moves related code to a new package
`@kbn/ml-data-view-utils` for data views and
`@kbn/ml-creation-wizard-utils` for the destination index form. The
latter might be used for other shared components across wizard..
In Data Frame Analytics, the checkbox to create a data view was removed
from the last "Create" step, instead the option to create a data view
was moved to the "Details" step.
In Transforms, the UI component to create the destination index was
brought over from DFA where there is a switch option to automatically
use the job ID as the name for the destination index by default.
**Addresses:** https://github.com/elastic/security-team/issues/7981
## Summary
This PR adds an OpenAPI spec bundler to simplify integration with the
Docs Engineering team. The bundler produces a single bundled file by
dereferencing and inlining some of external references and bundling them
and paths into a single file.
## Details
Currently we maintain a number of schema files inside
`x-pack/plugins/security_solution/common/api/**.schema.yaml` and it
might be hard for external teams to keep track of all the changes in our
schemas. By creating a singular schema file, we provide a clear
integration point for others.
The bundler addresses the following issues
- hide endpoints that we don't want to expose (Endpoints related to
features hidden under a feature flag and all internal endpoints should
be excluded from the file)
- hide not finished data structures related to features hidden under a
feature flag or data structures that are not designed to be public (For
example `RuleActionAlertsFilter` or `RuleActionParams` are exposed
directly from the Alerting framework and might be considered
implementation details, we don't want to document interfaces that are
not designed to be public so hiding them is a good option)
- modify spec based on presence of `x-modify` property (Instead of
exposing `x-modify: partial` we need to make the exported data structure
partial and instead of exposing `x-modify: required` we need to make the
exported data structure required)
- remove any internal attributes used for code generation like
`x-codegen-enabled` and `x-modify`
- inline some of the reused data structures (We have a lot of low-level
reusable data structures `in common_attributes.schema.yaml` which might
make the final documentation hardly usable from the UX perspective, so
we can inline them)
and lives in a new `@kbn/openapi-bundler` package under
`packages/kbn-openapi-bundler` folder.
### Related changes
- Implicit version type `version: 2023-10-31` has been changed to
explicit string type `version: '2023-10-31'` for all specs under
`security_solution/common/api` folder. Implicit type causes `js-yaml`
parsing it as a `Data` JS object leading to serializing it like
`2023-10-31T00:00:00.000Z`.
- `ListRequestQuery` schema in
`security_solution/common/api/endpoint/actions/list.schema.yaml ` has
been renamed to `EndpointActionListRequestQuery` to avoid conflicts with
`ListRequestQuery` in
`security_solution/common/api/endpoint/metadata/list_metadata.schema.yaml`.
While it's not an issue to have completely different schemas sharing the
same name in different files it may be an indication of pitfalls in the
API design. I'd say it's an open question if such cases need to be
always resolved automatically or reviewed manually. At this moment the
bundler can't resolve such conflicts.
## How to test?
There is a a new JS script added to Security Solution plugin located at
`x-pack/plugins/security_solution/scripts/openapi/bundle.js` with a
corresponding entry in `package.json` named `openapi:bundle`.
To test the PR change directory to Security Solution plugin's root
folder and run the bundler like below
```sh
cd x-pack/plugins/security_solution
yarn openapi:bundle
```
It should produce a bundled OpenAPI spec at
`x-pack/plugins/security_solution/target/openapi/security_solution.bundled.schema.yaml`.
## Open issues
- [x] Circular references (implemented in
dfdf0a51ea)
- [x] Mix of OpenAPI `3.0` and `3.1` specs (Maybe convert automatically
to `3.1`?). Folder like OpenAPI bundling format implemented
[here](0ae7ad5abb)
allows to mix OpenAPI `3.0` and `3.1` specs.
## Improvements
- [ ] Flexible configuration
- [ ] CLI support?
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fix https://github.com/elastic/kibana/issues/170428
The bug this is intended to resolve requires some in-depth steps to
reproduce. Follow the instructions in the issue above. Then, merge in
this branch and compare.
### Checklist
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes partially two remaining tasks from
https://github.com/elastic/kibana/issues/168753
Fixes partially dataview issue from
https://github.com/elastic/kibana/issues/170398
It stretches to maximum approximate 60 characters if any of the labels
in the list is of this length. If the content doesn't need the container
to stretch, it doesn't do it.
<details>
<summary> Field picker in Lens</summary>
minimum width:
<img width="445" alt="Screenshot 2023-11-21 at 15 56 03"
src="2f0f8482-bd00-4ec2-bbde-cbc4f3198eed">
auto-expanded width:
<img width="575" alt="Screenshot 2023-11-21 at 15 58 22"
src="df7bab4d-0a08-4d49-8a91-9386eba15d93">
</details>
<details>
<summary>Layer data view picker in Lens</summary>
<img width="376" alt="Screenshot 2023-11-21 at 16 01 17"
src="b8a98d83-dabc-49bd-a3cc-fc3856de6d3e">
<img width="455" alt="Screenshot 2023-11-21 at 15 58 09"
src="f2c5bde8-7a4a-485f-bf97-fc2179171e50">
<img width="615" alt="Screenshot 2023-11-21 at 15 56 27"
src="0574fc6c-69a3-44e9-9d48-8d427c1c5dba">
</details>
<details> <summary>Data view picker in Unified Search</summary>
<img width="341" alt="Screenshot 2023-11-21 at 16 00 29"
src="1c838ded-0dc5-4632-94e4-1d94586f667c">
<img width="441" alt="Screenshot 2023-11-21 at 15 58 04"
src="87e4f1c0-7922-4b94-a114-f23ece544395">
<img width="561" alt="Screenshot 2023-11-21 at 15 56 20"
src="3ea0f222-5241-4c5b-b00b-4311972754cc">
</details>
<details>
<summary> Data view picker in dashboard Create control flyout</summary>
<img width="677" alt="Screenshot 2023-11-21 at 16 14 00"
src="0455b6ed-555d-4cff-9e34-0de377be6e04">
<img width="682" alt="Screenshot 2023-11-21 at 15 54 56"
src="2a67685c-379d-4c0b-bf56-dbf7c35b3bd4">
</details>
<details>
<summary> Unified search data view select component (tested in
maps)</summary>
<img width="570" alt="Screenshot 2023-11-22 at 14 38 25"
src="bb52ab22-626d-4556-b40c-c9bcc925f426">
</details>
<details>
<summary>Unified search field and value picker</summary>
Adds `panelMinWidth`, removes the custom flex width change behavior
<img width="1142" alt="Screenshot 2023-11-22 at 14 40 26"
src="2450957f-38b7-4a73-b531-7acb29cb56bc">
f4f33624-9287-403e-8472-81f705440f97
</details>
<details>
<summary> Discover breakdown field</summary>
Removes the focus stretching and instead uses the panelMinWidth prop
<img width="419" alt="Screenshot 2023-11-21 at 16 46 50"
src="e35125ad-8823-4bff-954b-8119a352829c">
<img width="619" alt="Screenshot 2023-11-21 at 16 48 20"
src="89e63daf-a59e-43e1-a6ec-91d1b15b0fcd">
</details>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR instruments the Elastic AI Assistant with the Kibana APM Agent
enabling the tracing of retrievers, llms, chains, and tools which can
then be viewed within the Observability app. This PR also improves the
Assistant Model Evaluation tooling by enabling support for pulling and
running test datasets from LangSmith.
If the `assistantModelEvaluation` experimental feature flag is enabled,
and an APM server is configured, messages that have a corresponding
trace will have an additional `View APM trace` action:
<p align="center">
<img width="500"
src="e0b372ee-139a-4eed-8b09-f01dd88c72b0"
/>
</p>
Viewing the trace you can see a breakdown of the time spent in each
retriever, llm, chain, and tool:
<p align="center">
<img width="500"
src="f7cbd4bc-207c-4c88-a032-70a8de4f9b9a"
/>
</p>
Additionally the Evaluation interface has been updated to support adding
additional metadata like `Project Name`, `Run Name`, and pulling test
datasets from LangSmith. Predictions can now also be run without having
to run an Evaluation, so datasets can quickly be run for manual
analysis.
<p align="center">
<img width="500"
src="acebf719-29fd-4fcc-aef1-99fd00ca800a"
/>
</p>
<p align="center">
<img width="500"
src="7081d993-cbe0-4465-a734-ff9be14d7d0d"
/>
</p>
## Testing
### Configuring APM
First, enable the `assistantModelEvaluation` experimental feature flag
by adding the following to your `kibana.dev.yml`:
```
xpack.securitySolution.enableExperimental: [ 'assistantModelEvaluation' ]
```
Next, you'll need an APM server to collect the traces. You can either
[follow the documentation for
installing](https://www.elastic.co/guide/en/apm/guide/current/installing.html)
the released artifact, or [run from
source](https://github.com/elastic/apm-server#apm-server-development)
and set up using the [quickstart guide
provided](https://www.elastic.co/guide/en/apm/guide/current/apm-quick-start.html)
(be sure to install the APM Server integration to ensure the necessary
indices are created!). Once your APM server is running, add your APM
server configuration to your `kibana.dev.yml` as well using the
following:
```
# APM
elastic.apm:
active: true
environment: 'SpongBox5002c™'
serverUrl: 'http://localhost:8200'
transactionSampleRate: 1.0
breakdownMetrics: true
spanStackTraceMinDuration: 10ms
# Disables Kibana RUM
servicesOverrides.kibana-frontend.active: false
```
> [!NOTE]
> If connecting to a cloud APM server (like our [ai-assistant apm
deployment](https://ai-assistant-apm-do-not-delete.kb.us-central1.gcp.cloud.es.io/)),
follow [these
steps](https://www.elastic.co/guide/en/apm/guide/current/api-key.html#create-an-api-key)
to create an API key, and then set it via `apiKey` and also set your
`serverUrl` as shown in the APM Integration details within fleet. Note
that the `View APM trace` button within the UI will link to your local
instance, not the cloud instance.
> [!NOTE]
> If you're an Elastic developer running Kibana from source, you can
just enable APM as above, and _not_ include a `serverUrl`, and your
traces will be sent to the https://kibana-cloud-apm.elastic.dev cluster.
Note that the `View APM trace` button within the UI will link to your
local instance, not the cloud instance.
### Configuring LangSmith
If wanting to push traces to LangSmith, or leverage any datasets that
you may have hosted in a project, all you need to do is configure a few
environment variables, and then start the kibana server. See the
[LangSmith Traces
documentation](https://docs.smith.langchain.com/tracing) for details, or
just add the below env variables to enable:
```
# LangChain LangSmith
export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_ENDPOINT="https://api.smith.langchain.com"
export LANGCHAIN_API_KEY=""
export LANGCHAIN_PROJECT="8.12 ESQL Query Generation"
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
In this PR, I'm relocating all Kibana Security types (along with a few
schemas necessary for some of these types, unfortunately) that are part
of public contracts to separate packages. This change will enable any
plugin to utilize Security APIs via "static" or
["runtime"](https://github.com/elastic/kibana/pull/167113) dependencies,
regardless of whether Kibana Security already relies on these plugins or
not.
__NOTE TO REVIEWERS:__ I tried to minimize changes as much as I could
via moving only necessary types. I also didn't move deprecated parts of
the Setup/Start contracts to these new packages.
__Triggered by:__ https://github.com/elastic/kibana/pull/168910
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
The package data isn't needed anymore, so we can remove that package. On
top, it seems like the package was causing some issues.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR adds support for outputs with secrets preconfigured in the
`kibana.yml` config file.
As Kibana needs to compare the value of the secret to manage updates, a
hash of the value is stored in the output's saved object. The
implementation follows [option 2 in Infosec's
recommendations](https://github.com/elastic/infosec/issues/14853#issuecomment-1788705606)
with the Argon2id algorithm.
See [here](https://www.npmjs.com/package/argon2) for information about
the `argon2` Node package and
[here](https://github.com/ranisalt/node-argon2/wiki/Options) for the
config options. Here, `argon2` was configured with the recommended
`m=19456 (19 MiB), t=2, p=1 ` (for some reason, `timeCost` cannot be set
to less than 2).
Closes#166360
### Testing
1. Ensure the [`outputSecretsStorage` experimental feature
](fd4fdb01bc/x-pack/plugins/fleet/common/experimental_features.ts (L26))is
enabled.
2. Add the following to your kibana config:
```
xpack.fleet.outputs:
- id: my-logstash-output-with-a-secret
name: preconfigured logstash output with a secret
type: logstash
hosts: ["localhost:9999"]
ssl:
certificate: xxxxxxxxxx
secrets:
ssl:
key: thisissecret
```
3. Verify the secret has been correctly created, e.g. by issuing a `GET
.fleet-secrets/_search` request in Dev Tools: the secret should be
listed there.
4. Change the preconfigured value and wait for kibana to restart: the
secret should be updated with the new value.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: jillguyonnet <jill.guyonnet@gmail.com>
Attempting to merge #170852 again now that the release artefact step has
been fixed as part of https://github.com/elastic/kibana/pull/171457
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
Adds the ability to quickly create a categorisation anomaly detection
job from the pattern analysis flyout.
Adds a new `created_by` ID `categorization-wizard-from-pattern-analysis`
which can be picked up by telemetry.
Creates a new package for sharing our AIOPs ui actions IDs. I think we
should move the pattern analysis ID to this package too, but that can be
done in a separate PR.
51349f93-f072-4983-85f0-98741902fb5a
6e618581-8916-4e63-930f-945c96c25e6c
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## 📓 Summary
Closes#170728
This work comes from the need to use agent and cloud provider icons in
the new Log Detail flyout.
Since those icons were already used across the `infra` and `apm`
plugins, this was a good opportunity to extract the shared logic into
packages.
The results of this refactoring are two new packages:
- **@kbn/elastic-agent-utils**: exports small utilities and type
definition used to parse the icon to render and exploits also across the
APM plugin.
- **@kbn/custom-icons**: exports custom icons built on top of EuiIcon,
encapsulating logic related to mapping from data to the relative icon.
Apart from creating the new plugins, this also applies their usage to
the `infra` and `apm` plugins, while the Log Explorer flyout will
benefit from these working on
https://github.com/elastic/kibana/issues/170721.
## 🧪 How to test
### Infra
- Navigate to `Infrastructure -> Hosts`
- Verify the hosts table correctly renders the cloud provider icon for
each table entry.
### APM
- Navigate to `APM -> Services`.
- Verify each table entry correctly displays the related agent icon.
- Navigate to `APM -> Services`.
- Click on a service where t a cloud provider icon is expected to appear
next to the service name.
- Verify the icon is correctly displayed.
- Navigate to `APM -> Services -> Service Map`.
- Create a new group.
- Verify the agent icon is correctly displayed for each entry in the
preview list.
- Navigate to `APM -> Traces`.
- Verify each table entry correctly displays the related agent icon.
- Navigate to `APM -> Settings -> Agent Explorer`.
- Verify each table entry correctly displays the related agent icon.
---------
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR refactors the export type classes into their own packages to be
then instantiated in the reporting plugin. This will reduce bloat in the
central reporting plugin.
**Main packages**
- `kbn/reporting-export-types-{png,pdf,csv}` are server packages with
export type declarations
- `kbn/reporting-export-types-{png,pdf,csv}-common` are shared common
packages with type declarations and constants
**Other changes**
- Remove `reporting.getScreenshots()`
- Remove duplicated `schema_utils.ts`
- Consolidate `JOB_STATUS` declaration as an enum
<img width="1063" alt="image"
src="bced8321-93c5-4ebd-b31e-1fd946166241">
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Timothy Sullivan <tsullivan@elastic.co>
Related to [#166340](https://github.com/elastic/kibana/issues/166340)
## Summary
Add mock identity provider and utils to test serverless user roles.
## Screenshot
### 1. Login selector
<img width="767" alt="Screenshot 2023-11-08 at 15 18 18"
src="82b4a29f-65b4-45d2-bed3-6d9f74043c48">
### 2. Single sign on screen
<img width="437" alt="Screenshot 2023-11-09 at 12 30 46"
src="3d5b6f26-5409-4169-a627-bcf6d09836d9">
### 3. User profile page
<img width="1041" alt="Screenshot 2023-11-08 at 17 36 22"
src="50bd4a5a-f9a8-4643-9384-9a352701b011">
## Testing
SAML is only supported by ES when running in SSL mode.
1. To test the mock identity provider run a serverless project in SSL
mode using:
```bash
yarn es serverless --ssl
yarn start --serverless=es --ssl
```
2. Then access Kibana and login in using "Continue as Test User".
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
Related to https://github.com/elastic/kibana/issues/169759.
## Summary
This PR creates the initial plugin for Dataset Quality.
The plugin has the basic boilerplate that allows us to continue working
on the UI and Server sides in parallel.
Further configs are to be added along with upcoming commits like:
1. Storybook
2. FTR configs
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Related to https://github.com/elastic/ems-client/pull/208
Update to a version of `ems-client` that supports Node 20.
I'm creating new PRs for:
* `@elastic/ems-client@8.4.1` in Kibana `8.11` branch
* `@elastic/ems-client@7.17.1` for Kibana `7.17` branch
Sorry for the small overhead on reviewing; this way, it seems faster to
get CI running in parallel for the three releases and unblock the
operations team on this issue.
Part of https://github.com/elastic/kibana-team/issues/646
This PR adds an example plugin in `examples/error_boundary` that shows
usage of KibanaErrorBoundary.
The example plugin is used in a functional test to ensure errors are
caught in the appropriate way, and error messages include a working
Refresh button.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Removes `testing-library/dom` from dependencies. As all the utilities
from`dom` are available already in `testing-library/react`, there's no
need to have both `dom` and `react` libraries available in our
package.json.
Following the [@testing-library/react
documentation:](https://testing-library.com/docs/react-testing-library/intro)
> [React Testing
Library](https://github.com/testing-library/react-testing-library)
builds on top of DOM Testing Library by adding APIs for working with
React components.
Let's just import everything from `testing-library/react`, this way we
won't need to worry about inconsistencies between `testing-library/dom`
we have in our `package.json` and the one that is
`testing-library/react` dependency.
## Summary
First step to update KeaJS to 3.x series. Currently we are using 2.4.2.
For a better jump ahead we should first update to 2.6.
### Checklist
Delete any items that are not applicable to this PR.
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
