**Epic:** https://github.com/elastic/kibana/issues/174168
**Partially addresses:**
https://github.com/elastic/kibana/issues/202068,
https://github.com/elastic/kibana/issues/202078,
https://github.com/elastic/kibana/issues/202079
**Follow-up to:** https://github.com/elastic/kibana/pull/211300
## Summary
We're cleaning up and refactoring our existing test plans for prebuilt
rule customization, upgrade, and export/import workflows.
Specifically, this PR:
- Creates an "entrypoint" file that should help navigate all the test
plans for prebuilt rules.
- Creates a file for keeping common information about prebuilt rules
that can be shared between the test plans.
- Extracts duplicated terminology to the file with common information.
- Extracts duplicated assumptions to the file with common information.
- Extracts duplicated non-functional requirements to the file with
common information.
- Adds user stories to each test plan.
- Updates links to tickets in every test plan.
No "functional" changes have been made to any test plans, such as
adding, removing, or updating any scenarios.
This refactoring prepares the test plans for being "functionally"
changed and improved in follow-up PRs. For example, we're going to cover
the logic described in https://github.com/elastic/kibana/issues/210358
and address any other gaps in coverage.
## Review tip
It might be easier to review this PR commit-by-commit as each of them
contains logically cohesive changes.
### [Security Solution] [Attack discovery] Fixes alerts filtering issues
This PR resolves the following Attack discovery alerts filtering issues:
- [[Security Solution] [Bug] A few filters show error 'Unexpected error from Elasticsearch' for the alerts flyout #208481](https://github.com/elastic/kibana/issues/208481)
- [[Security Solution] [Bug] Lucene not updated as the space holder when we apply Lucene as the filtering language #208170](https://github.com/elastic/kibana/issues/208170)
- Connector selection resets in non-default spaces
- Saving a filter edited via `Edit Query DSL` with an unknown `user.name` value results in a `filter value is invalid or incomplete` filter
- Local field Reset clears the preview dropdowns when they are in an error state
- Updates the formatting of `Up to _n_ alerts` for Borealis
#### Feature flag required for testing
The following feature flag is required to test this PR:
```yaml
xpack.securitySolution.enableExperimental:
- 'attackDiscoveryAlertFiltering'
```
The following sections provide details and desk testing steps for the alerts filtering issues fixed by this PR.
### [Security Solution] [Bug] A few filters show error 'Unexpected error from Elasticsearch' for the alerts flyout #208481
To resolve [[Security Solution] [Bug] A few filters show error 'Unexpected error from Elasticsearch' for the alerts flyout #208481](https://github.com/elastic/kibana/issues/208481):
- The `_ignored` metadata field was added to the [METADATA](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-metadata-fields.html) keyword in the `ES|QL` queries that power the `Alert summary` and `Alerts preview` tabs
- The `Alert summary` and `Alerts preview` tabs dropdowns are now filtered to only display fields in the alerts index
- Example: Previously, if the `dll.Ext.load_index` field was in a `logs-*` index, but not the alerts index, it would still appear in the `Alerts summary` dropdown, and result in an error if selected. After the fix in this PR, this (example) field would not appear in the dropdown.
#### Desk testing
To desk test this fix:
- See issue <https://github.com/elastic/kibana/issues/208481>
- In addition to the steps in [issue #208481](https://github.com/elastic/kibana/issues/208481), verify the `Alert summary` and `Alerts preview` dropdowns
- Do NOT display an error when the `_ignored` field is selected
- Identify a field that is present in a `logs-*` index, but not the alerts index, for example `dll.Ext.load_index`. Verify this field is NOT displayed in the dropdown
### [Security Solution] [Bug] Lucene not updated as the space holder when we apply Lucene as the filtering language #208170
To resolve [[Security Solution] [Bug] Lucene not updated as the space holder when we apply Lucene as the filtering language #208170](https://github.com/elastic/kibana/issues/208170), the custom placeholder was removed, as illustrated by the _before_ and _after_ screenshots below:
**Before**
**After**
#### Desk testing
To desk test this fix, see <https://github.com/elastic/kibana/issues/208170>
### Connector selection resets in non-default spaces
This PR fixes an issue where the last selected connector would reset in non-default spaces when all of the following were true:
- The user is in a non-default space
- Two or more generative AI connectors are configured
This issue occurred in non-default spaces because:
- `spaceId` loads asynchronously
- Kibana's `package.json` references an older version of `react-use`, with a known bug in the `useLocalStorage` hook, which is fixed by <https://github.com/streamich/react-use/pull/1944>
- I verified (locally) the fix from <https://github.com/streamich/react-use/pull/1944> would work if Kibana's version of `react-use` was updated in `package.json`, however that effort appears to be on hold: <https://github.com/elastic/kibana/pull/179268> . For now (to minimize changes), `spaceId` has been removed from all Attack discovery local storage keys.
#### Desk testing
1. Create a new space (if you only have the default space)
2. Configure two or more Gen AI connectors
3. Select the newly created space
4. Navigate to Security > Attack discovery
5. Select a connector, for example `Claude 3.5 Sonnet`
6. Now select a _different_ connector, for example `Gemini 1.5 Pro 002`
7. Navigate to a different page in the Security solution, for example Security > Alerts
9. Once again, navigate to Security > Attack discovery
**Expected result**
- The previously selected connector, e.g. `Gemini 1.5 Pro 002` is still selected
### Saving a filter edited via `Edit Query DSL` with an unknown `user.name` value results in a `filter value is invalid or incomplete` filter
This PR fixes an issue where editing a previously created non-Query DSL filter via `Edit Query DSL`, and then entering an unknown `user.name`, resulted in a filter with text that reads: `filter value is invalid or incomplete`, as illustrated by the screenshot below:
Generating attack discoveries with a filter like the one shown in the screenshot above would also result in errors.
This issue was resolved by adding a `FilterManager` to manage the local state of the filters in the settings panel.
#### Desk testing
1. Navigate to Security > Attack discovery
2. Click the settings gear
3. Click the `+` button to open the `Add filter` popover
4. In the popover, configure a `user.name` `is` `Administrator` filter
Note: replace `Administrator` with a real `user.name` value if your alerts index doesn't have the value ``Administrator``
5. Click `Add filter` to close the popover
**Expected result**
- The `user.name: Administrator` filter appears below the query bar
6. Click the `user.name: Administrator` filter, and choose `Edit filter` from the popover
7. Click the `Edit as Query DSL` button (in the upper right hand corner)
8. In the `Edit filter` Elasticsearch Query DSL editor, edit the Query DSL such that it has a value that does NOT exist in the index, like the following example:
```json
{
"match_phrase": {
"user.name": "Admasdfinistrator"
}
}
```
9. Click `Update filter`
**Expected results**
- The `user.name: Admasdfinistrator` filter, which references a value that does not exist in the alerts index, appears below the query bar
- The updated filter does NOT have the text `filter value is invalid or incomplete`, as illustrated by the the screenshot in the description of this issue above.
### Local field Reset clears the preview dropdowns when they are in an error state
This PR fixes an issue where the local (to the preview tab) reset button did not clear the preview dropdowns if they were in an error state.
The issue is fixed by calling `clearSearchValue()` to reset the stack by field when it's in an error state (i.e. because an invalid field was entered)
Note: The "local" (to the tab) `Reset` button shown in the screenshot below is fixed by this PR:
, however the `Reset` button at the bottom of the flyout will NOT clear the dropdown if it's in an error state. (For now, this is the expected behavior.) The workaround is to manually select a valid value in the dropdown, or click `Save` or `Cancel`. (The preview dropdown does not effect the Attack discovery query, is not saved, and automatically resets to the default every time the flyout is opened.)
#### Desk testing
1. Navigate to Security > Attack discovery
2. Click the settings gear
3. In the `Alert summary` tab, focus the dropdown and delete the text until it reads:
```
kibana.alert.rule.na
```
4. Blur the dropdown by clicking outside it
**Expected results**
- The dropdown is highlighted red
- The `Reset` button appears below the text `Select a field`
5. Click the `Reset` button below the text `Select a field`
**Expected results**
- The dropdown is NOT highlighted red (the error state is cleared)
- The dropdown text is reset to the (valid) default value: `kibana.alert.rule.name`
### Updates the formatting of `Up to _n_ alerts` for Borealis
This PR updates the formatting of the `Up to n alerts` text in Borealis, as illustrated by the before and after screenshots below:
**Before**
**After**
#### Desk testing
To desk test this fix:
1. Configure Kibana to use the `dark` theme
2. Navigate to Security > Attack discovery
3. In the connector selector, choose `+ Add new Connector...`
4. Click in the `Select a connector` dialog, click `OpenAI`
4. Enter a throwaway configuration for the connector (note: you won't actually use it), and click `Save`
**Expected results**
- The animated `Up to 100 alerts will be analyzed` message will appear
- The color of the animated numeric text, e.g. `100` matches the color of the `Up to` text that precedes it
- The extra whitespace trailing the `100`, shown in the _Before_ image (above) does NOT appear. The trailing whitespace after the `100` looks like the _After_ image (also above).
## Summary
This PR was created in response to
https://github.com/elastic/kibana/pull/208858, it migrates all existing
language definitions within the `@kbn/code-editor` package into the
`@kbn/monaco` package to provide a separation of concern for logic that
doesn't particularly relate to the configurations for the code editor
UI.
With this change, all supported languages are ingested from the
`@kbn/monaco` package where they will be domiciled from henceforth, and
in turn fix the issue that was discovered relating to the way the
language definitions within `@kbn/code-editor` get registered as a side
effect.
With this change, to add support for a new language, said language
should be defined within `@kbn/monaco`, and registered in
`languages/index.ts`.
<!--
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
-->
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes https://github.com/elastic/kibana/issues/208138
ES|QL response updated with `is_partial` key.
x-pack/test/api_integration/apis/maps/search.ts tests where skipped to
unblock ES-snapshot promotion. This PR unskips the tests and updates the
expects for the new response shape
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Changes in this PR are in associated with Endpoint Management support
for spaces:
- When creating an endpoint artifact (Trusted Apps, Event Filters,
Blocklists, Host Isolation Exceptions, Endpoint Exceptions), the API
will ensure a new `tag` is dded to each item created that identifies the
space ID from where it was crated
- This functionality is behind the following feature flag:
`endpointManagementSpaceAwarenessEnabled`
- The tag that will be automatically added has a format of:
`ownerSpaceId:<space_id_here>`
- Likewise, when updating an artifact, the API will ensure that at least
1 owner space id tag is present on the item, and if not, it will add one
to it.
The useAbortController and useAbortableAsync hooks are very generic
hooks that are useful beyond observability. This PR moves them into
`react-hooks` which already houses other similar hooks.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Adding `Tech Preview` for `rainbow sprinkles` model
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Resolve https://github.com/elastic/kibana/issues/209008
It seems with ES > 9 that runtime field is sometimes failing, while I am
not sure why, that PR make it more robust and should avoid test
flakyness.
---------
Co-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
part of https://github.com/elastic/kibana/issues/208908
Replaces scss to css-in-js. I've tested all the changes.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Kibana permission for read-only is removed. This is not a breaking
change while the feature is not yet released.
Cluster requirements make it obsolete.
Also fixed warnings on permission names.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
This PR updates the behavior of `Permissions` tab in `Space Management`
when `xpack.security.enabled` is set to `false` to show a callout with a
meaningful explanation.
Closes: #210241
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Adds back the additional alerts table toolbar controls to edit the
grouping configuration. Adds test cases to check the correctness of the
Observability alerts table configurations.
## To verify
1. Create one or more rules that fire alerts in Observability
2. Navigate to Observability > Alerts
3. Verify that the grouping toggle shows and works correctly in the
table toolbar (`Group by: ...`)
Closes https://github.com/elastic/kibana/issues/205146
## Summary
We run a session clean up task that opens a point in time query to try
and delete any older sessions in the session index. We've noticed that
this task fails quite often with the same error
`no_shard_available_action_exception`. On investigating, it's possible
that the point in time query is opened when there are no shards
available for that index. This PR fixes that by checking if the PIT
query fails with 503 bails if it throws the error - allowing the task to
be tried again in the next run of the task manager. We allow for up to
10 failures of the clean up task in succession before logging an error.
### Testing
Unfortunately, there's no reliable way to simulate missing shards
locally. I've added a new integration test config here:
```
node scripts/functional_tests_server.js --config x-pack/test/security_api_integration/session_shard_missing.config.ts
```
This overrides the ES function to return 503 when opening PIT query and
then attempts to assert the result from the task manager.
### Release note
Updates session cleanup mechanism to account for potential missing
shards in Session index.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
[kibana-flaky-test-suite-runner#7836](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7836)
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Connected with https://github.com/elastic/kibana/issues/195188
## Summary
- Moved params of es query rule type to
`@kbn/response-ops-rule-params/es_query` package
- Moved params of index threshold rule type to
`@kbn/response-ops-rule-params/index_threshold` package
The following constants for the es query rule type have been duplicated:
- MAX_SELECTABLE_SOURCE_FIELDS
- MAX_SELECTABLE_GROUP_BY_TERMS
- ES_QUERY_MAX_HITS_PER_EXECUTION
Closes https://github.com/elastic/kibana/issues/210204
This will automatically re-index the knowledge base if upon adding a KB
entry there is this error:
> The [sparse_vector] field type is not supported on indices created on
versions 8.0 to 8.10
That error means that semantic_text is not supported in the given index,
and it should therefore be re-indexed.
**How to test this PR:**
**8.10**
- `git checkout -B 8.10 origin/8.10`
- Start Kibana:
- `nvm use && yarn kbn bootstrap && yarn start`
- Start ES
- `nvm use && yarn es snapshot --license trial --E
path.data="/Users/sorenlouv/elastic/kbn_es_data/upgrade_testing"`
**8.19**
- `git checkout -B 8.19 origin/8.x`
- Start Kibana:
- `nvm use && yarn kbn bootstrap && yarn start`
- Start ES
- `nvm use && yarn es snapshot --license trial --E
path.data="/Users/sorenlouv/elastic/kbn_es_data/upgrade_testing"`
- Install Knowledge base
- Try adding an item to KB (it should fail ❌️)
**9.1.0**
- `gh pr checkout 210386`
- Start Kibana:
- `nvm use && yarn kbn bootstrap && yarn start`
- Start ES
- `nvm use && yarn es snapshot --license trial --E
path.data="/Users/sorenlouv/elastic/kbn_es_data/upgrade_testing"`
- Try adding an item to KB (it should succeed ✅️)
**TODO:**
- Add an upgrade test that covers this flow
---------
Co-authored-by: Viduni Wickramarachchi <viduni.ushanka@gmail.com>
## Summary
Hide `elastic_conenctors` package by default. This excludes the package
from:
- integration page
- search results
#### Verification
Without the change package shows up as the integration, and search
results. With the change it's correctly excluded.
## Summary
This PR deletes all usages of ServiceEntityStoreEnabled feature flag.
The feature flag itself has been left in `experimental_features.ts`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR add a method to wait for map to be loaded to replace generic
`renderable.waitForRender()`.
While investigating the recent test failure on CI I found out that for
maps case we can simplify the logic with few facts:
- before start waiting for render to complete, we need to wait for main
container `#maps-plugin` to be in DOM. It takes 2-3 seconds.
- there is always a single div block with `data-render-complete`
attribute, and there is a comment in source code stating `See if the
"data-render-complete" attribute is "true". If so we're done!` which
means we can simply wait for
`div[data-dom-id][data-render-complete="true"]`
6de2ef0e6d/x-pack/platform/plugins/shared/maps/public/connected_components/map_container/map_container.tsx (L103-L116)
`renderable.waitForRender()` is a good waiter, but probably for
dashboard with multiple panels.
Adds avg ingestion per day, total doc count and ingestion rate graph to
the lifecycle view.
We use the dataset quality plugin to compute these values. I've added a
query string to optionally retrieve the creation date of a data stream
in the `data_streams/stats` endpoint.
-----
@elastic/obs-ux-logs-team the change in dataset quality involves the
optional retrieval of the data streams creation date in the `/stats`
endpoint. There are other ways in dataset quality to get these
informations but they rely on queries to compute the data. In our case
these queries will always be unbounded and using the `/stats` would be
more efficient as it relies on cluster state.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
[Internal link](https://github.com/elastic/security-team/issues/10820)
to the feature details
Part of https://github.com/elastic/security-team/issues/11232
This PR covers SIEM Migrations Get prebuilt rules API (route: `GET
/internal/siem_migrations/rules/{migration_id}/prebuilt_rules`)
integration test:
* get all prebuilt rules matched by migration rules
* return empty response when migration rules did not match prebuilt
rules
## Summary
This PR aims at relocating some of the Kibana modules (plugins and
packages) into a new folder structure, according to the _Sustainable
Kibana Architecture_ initiative.
> [!IMPORTANT]
> * We kindly ask you to:
> * Manually fix the errors in the error section below (if there are
any).
> * Search for the `packages[\/\\]` and `plugins[\/\\]` patterns in the
source code (Babel and Eslint config files), and update them
appropriately.
> * Manually review
`.buildkite/scripts/pipelines/pull_request/pipeline.ts` to ensure that
any CI pipeline customizations continue to be correctly applied after
the changed path names
> * Review all of the updated files, specially the `.ts` and `.js` files
listed in the sections below, as some of them contain relative paths
that have been updated.
> * Think of potential impact of the move, including tooling and
configuration files that can be pointing to the relocated modules. E.g.:
> * customised eslint rules
> * docs pointing to source code
> [!NOTE]
> * This PR has been auto-generated.
> * Any manual contributions will be lost if the 'relocate' script is
re-run.
> * Try to obtain the missing reviews / approvals before applying manual
fixes, and/or keep your changes in a .patch / git stash.
> * Please use
[#sustainable_kibana_architecture](https://elastic.slack.com/archives/C07TCKTA22E)
Slack channel for feedback.
Are you trying to rebase this PR to solve merge conflicts? Please follow
the steps describe
[here](https://elastic.slack.com/archives/C07TCKTA22E/p1734019532879269?thread_ts=1734019339.935419&cid=C07TCKTA22E).
#### 4 packages(s) are going to be relocated:
| Id | Target folder |
| -- | ------------- |
| `@kbn/alerting-rule-utils` |
`x-pack/platform/packages/shared/alerting-rule-utils` |
| `@kbn/observability-alert-details` |
`x-pack/solutions/observability/packages/alert-details` |
| `@kbn/observability-alerting-test-data` |
`x-pack/solutions/observability/packages/alerting-test-data` |
| `@kbn/observability-get-padded-alert-time-range-util` |
`x-pack/solutions/observability/packages/get-padded-alert-time-range-util`
|
<details >
<summary>Updated references</summary>
```
./.i18nrc.json
./package.json
./packages/kbn-ts-projects/config-paths.json
./src/platform/packages/private/kbn-repo-packages/package-map.json
./tsconfig.base.json
./x-pack/platform/packages/shared/alerting-rule-utils/jest.config.js
./x-pack/solutions/observability/packages/alert-details/jest.config.js
./x-pack/solutions/observability/packages/alerting-test-data/jest.config.js
./x-pack/solutions/observability/packages/get-padded-alert-time-range-util/jest.config.js
./yarn.lock
.github/CODEOWNERS
```
</details><details >
<summary>Updated relative paths</summary>
```
x-pack/platform/packages/shared/alerting-rule-utils/jest.config.js:10
x-pack/platform/packages/shared/alerting-rule-utils/tsconfig.json:2
x-pack/solutions/observability/packages/alert-details/jest.config.js:10
x-pack/solutions/observability/packages/alert-details/tsconfig.json:2
x-pack/solutions/observability/packages/alerting-test-data/jest.config.js:10
x-pack/solutions/observability/packages/alerting-test-data/tsconfig.json:2
x-pack/solutions/observability/packages/get-padded-alert-time-range-util/jest.config.js:10
x-pack/solutions/observability/packages/get-padded-alert-time-range-util/tsconfig.json:2
```
</details>
## Summary
[Internal link](https://github.com/elastic/security-team/issues/10820)
to the feature details
Part of https://github.com/elastic/security-team/issues/11232
This PR covers SIEM Migrations Install API (route: `POST
/internal/siem_migrations/rules/{migration_id}/install`) integration
test:
* install all installable custom migration rules
* install all installable migration rules matched with prebuilt rules
* install and enable all installable migration rules
* install migration rules by ids
* install rules of non-existing migration - nothing should be installed
* Error handling: an error if body payload is not passed
Fix: System Message Missing in Inference Plugin
Closes#209548
## Summary
A regression was introduced in 8.18
([#199286](https://github.com/elastic/kibana/pull/199286)), where the
system message is no longer passed to the inference plugin and,
consequently, the LLM.
Currently, only user messages are being sent, which impacts conversation
guidance and guardrails. The system message is crucial for steering
responses and maintaining contextual integrity.
The filtering of the system message happens here:
771a080ffa/x-pack/platform/plugins/shared/observability_ai_assistant/server/service/client/index.ts (L510-L512)
Fix Approach
- Ensure the `system` message is included as a parameter in
`inferenceClient.chatComplete.`
```typescript
const options = {
connectorId,
system,
messages: convertMessagesForInference(messages),
toolChoice,
tools,
functionCalling: (simulateFunctionCalling ? 'simulated' : 'native') as FunctionCallingMode,
};
if (stream) {
return defer(() =>
this.dependencies.inferenceClient.chatComplete({
...options,
stream: true,
})
).pipe(
convertInferenceEventsToStreamingEvents(),
instrumentAndCountTokens(name),
failOnNonExistingFunctionCall({ functions }),
tap((event) => {
if (
event.type === StreamingChatResponseEventType.ChatCompletionChunk &&
this.dependencies.logger.isLevelEnabled('trace')
) {
this.dependencies.logger.trace(`Received chunk: ${JSON.stringify(event.message)}`);
}
}),
shareReplay()
) as TStream extends true
? Observable<ChatCompletionChunkEvent | TokenCountEvent | ChatCompletionMessageEvent>
: never;
} else {
return this.dependencies.inferenceClient.chatComplete({
...options,
stream: false,
}) as TStream extends true ? never : Promise<ChatCompleteResponse>;
}
}
```
- Add an API test to verify that the system message is correctly passed to the LLM.
This pull request introduces the new `unassigned` criticality level for
the asset criticality's `bulk_upload` .
### Key Changes:
#### Schema and Configuration Updates:
* Added `unassigned` to the list of criticality levels in multiple
schema files (`kibana.serverless.yaml`, `kibana.yaml`,
`common.schema.yaml`,
`ess/security_solution_entity_analytics_api_2023_10_31.bundled.schema.yaml`,
`serverless/security_solution_entity_analytics_api_2023_10_31.bundled.schema.yaml`).
#### TypeScript and Constants:
* Updated `AssetCriticalityLevel` enum in `common.gen.ts` to include
`unassigned`.
* Added `UNASSIGNED` to `CriticalityLevels` enum and
`CriticalityModifiers` in `constants.ts`.
#### Tests:
* Updated test cases to include `unassigned` as a valid criticality
level in `parse_asset_criticality_csv_row.test.ts`,
`validations.test.ts`, `asset_criticality_data_client.test.ts`, and
`asset_criticality_csv_upload.ts`.
#### Backend Logic:
* Modified `AssetCriticalityDataClient` to handle `unassigned`
criticality level appropriately.
<img width="1488" alt="Screenshot 2025-01-30 at 2 03 11 PM"
src="https://github.com/user-attachments/assets/938411c4-725a-451c-ab38-aca36a704e91"
/>
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Testing Steps
The main idea for this change is that the `unassigned` criticality level
is actually marked as `deleted` in the ES documents.
ES index name for default space :
`.asset-criticality.asset-criticality-default`
#### API
1. `POST /api/asset_criticality/upload_csv`
```
curl --location 'http://localhost:5601/api/asset_criticality/upload_csv?output=stream' \
--header 'kbn-xsrf: hello' \
--header 'Accept: multipart/form-data' \
--header 'Authorization: *******' \
--form 'file=@"<Full path of the CSV file for asset criticality>"
```
Error response :
```
{"errors":[{"message":"Invalid criticality level \"unassigned_impact\", expected one of extreme_impact, high_impact, medium_impact, low_impact, unassigned","index":4}],"stats":{"successful":3,"failed":1,"total":4}}%
```
Success response :
```
{"errors":[],"stats":{"successful":4,"failed":0,"total":4}}%
```
Query the ES using below query to see if the criticality level is
`deleted`
```
GET .asset-criticality.asset-criticality-default/_search
{
"query": {
"match": {
"asset.criticality": "deleted"
}
}
}
```
### UI
1. Navigate to Entity Store page
2. Upload a csv file with incorrect asset criticality level
3. Rectify and upload the same file with correct criticality levels.
4. Navigate to EA Dashboard and scroll down to the Entities section.
5. Select an entity and open the flyout.
6. Try changing the asset criticality of the entity. No blank/empty
value should be present or assigned to asset criticality.
7. Should be able to successfully modify the asset criticality for the
entity.
Confirm this by querying the ES with the query :
```
GET .asset-criticality.asset-criticality-default/_search
{
"query": {
"match": {
"asset.criticality": "deleted"
}
}
}
```
Ensure that the Elastic search document's `_source` contains three keys,
with `deleted` present as the ` `criticality_level` for the below three
keys.
`criticality_level`
`host.asset.criticality`
`asset.criticality`
Example :
### Bulk upload
```
curl --location 'http://localhost:5601/api/asset_criticality/bulk' \
--header 'kbn-xsrf: hello' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic ****' \
--data '{
"records": [
{
"id_value": "host-1",
"id_field": "host.name",
"criticality_level": "low_impact"
},
{
"id_value": "host-2",
"id_field": "host.name",
"criticality_level": "medium_impact"
},
{
"id_value": "host-6",
"id_field": "host.name",
"criticality_level": "medium_impact"
},
{
"id_value": "host-3",
"id_field": "host.name",
"criticality_level": "high_impact"
},
{
"id_value": "host-4",
"id_field": "host.name",
"criticality_level": "high_impact"
},
{
"id_value": "host-bulkupload",
"id_field": "host.name",
"criticality_level": "unassigned"
}
]
}
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
# Backport
This will backport the following commits from `8.x` to `main`:
- [[UA] Handle frozen indices deprecations
(#208156)](https://github.com/elastic/kibana/pull/208156)
<!--- Backport version: 9.6.4 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)
<!--BACKPORT [{"author":{"name":"Gerard
Soldevila","email":"gerard.soldevila@elastic.co"},"sourceCommit":{"committedDate":"2025-01-30T22:36:28Z","message":"[UA]
Handle frozen indices deprecations (#208156)\n\n## Summary\r\n\r\nCloses
https://github.com/elastic/kibana-team/issues/1392\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"cc358e3c46e5df2af834e341114f23aa1fd4f7d1","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","release_note:skip","backport:version","v8.18.0","v8.19.0"],"title":"[UA]
Handle frozen indices
deprecations","number":208156,"url":"https://github.com/elastic/kibana/pull/208156","mergeCommit":{"message":"[UA]
Handle frozen indices deprecations (#208156)\n\n## Summary\r\n\r\nCloses
https://github.com/elastic/kibana-team/issues/1392\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"cc358e3c46e5df2af834e341114f23aa1fd4f7d1"}},"sourceBranch":"8.x","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/211165","number":211165,"state":"MERGED","mergeCommit":{"sha":"22a748198915940c4bbcf59eab167caae462284e","message":"[8.18]
[UA] Handle frozen indices deprecations (#208156) (#211165)\n\n#
Backport\n\nThis will backport the following commits from `8.x` to
`8.18`:\n- [[UA] Handle frozen indices
deprecations\n(#208156)](https://github.com/elastic/kibana/pull/208156)\n\n<!---
Backport version: 8.9.8 -->\n\n### Questions ?\nPlease refer to the
[Backport
tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT
[{\"author\":{\"name\":\"Gerard\nSoldevila\",\"email\":\"gerard.soldevila@elastic.co\"},\"sourceCommit\":{\"committedDate\":\"2025-01-30T22:36:28Z\",\"message\":\"[UA]\nHandle
frozen indices deprecations (#208156)\\n\\n##
Summary\\r\\n\\r\\nCloses\nhttps://github.com/elastic/kibana-team/issues/1392\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by:\nkibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\",\"sha\":\"cc358e3c46e5df2af834e341114f23aa1fd4f7d1\",\"branchLabelMapping\":{\"^v8.16.0$\":\"main\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"Team:Core\",\"release_note:skip\",\"backport:version\",\"v8.18.0\",\"v8.19.0\"],\"number\":208156,\"url\":\"https://github.com/elastic/kibana/pull/208156\",\"mergeCommit\":{\"message\":\"[UA]\nHandle
frozen indices deprecations (#208156)\\n\\n##
Summary\\r\\n\\r\\nCloses\nhttps://github.com/elastic/kibana-team/issues/1392\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by:\nkibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\",\"sha\":\"cc358e3c46e5df2af834e341114f23aa1fd4f7d1\"}},\"sourceBranch\":\"8.x\",\"suggestedTargetBranches\":[\"8.18\"],\"targetPullRequestStates\":[{\"branch\":\"8.18\",\"label\":\"v8.18.0\",\"labelRegex\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.19\",\"label\":\"v8.19.0\",\"labelRegex\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"}]}]\nBACKPORT-->\n\nCo-authored-by:
Gerard Soldevila
<gerard.soldevila@elastic.co>"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
