**Partially addresses:** #211808,
https://github.com/elastic/security-docs/issues/5981 (internal)
**Resolves: #208329**
## Summary
This is the second part of the migration effort, containing changes for:
- BULK CRUD (removing, for v.9.0)
The PR also contains changes for ticket #208329 - as changes for
removing of dead code for handling Bulk CRUD endpoints had to be
combined together with removing the schema files for Bulk CRUD
endpoints.
This PR will be backported only to versions for Kibana v9
# Testing
1. cd x-pack/solutions/security/plugins/security_solution
2. yarn openapi:bundle:detections
3. Take the bundled file
(docs/openapi/ess/security_solution_detections_api_2023_10_31.bundled.schema.yaml)
and load it into bump.sh console to see the changes.
4. Compare the changes with the [Legacy
documentation](https://www.elastic.co/guide/en/security/current/rule-api-overview.html)
You can also use this [link](https://bump.sh/jkelas2/doc/kibana_wip2/)
where I deployed the generated bundled doc.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Resolves first part of https://github.com/elastic/kibana/issues/198783
(snooze API)
Creates a public API for adding snooze schedule to rules.
For this purpose we created new schedule schema which will be used as
standard schedule schema across rules and alerting framework.
**Note**
The code to be reviewed for public API is under `common/routes/schedule`
and inside `external` folders.
Rest of the code is just moving existing internal API route and its code
to `/internal` folders.
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### How to test
- Create a rule in kibana
- Snooze that rule via new public API
```typescript
Path: https://localhost:5601/api/alerting/rule/<rule_id>/snooze_schedule
Method: POST
Body:
{
"schedule": {
"custom": {
"start": "2025-02-25T18:00:00.000Z",
"duration": "15m",
"recurring": {
"every": "1w",
"onWeekDay": ["TU", "FR"],
"occurrences": 10
}
}
}
}
```
- Verify various snooze schedule scenarios are generated correctly
#### Flaky test runner:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8009
### Release note
Allow users to create a snooze schedule for rule via API
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: lcawl <lcawley@elastic.co>
Add streams API to documentation as an experimental feature
<img width="2555" alt="Screenshot 2025-03-07 at 11 44 54"
src="https://github.com/user-attachments/assets/f54e5e6e-0c20-4bad-9cff-27747d0f76e2"
/>
There are a couple of changes in here:
* Split streams API in internal and public and mark the public parts as
experimental
* Add the public parts to the Kibana documentation
* Add description and summary
* Adjust the server repository wrapper to pass through summary and
description
# To test
* Generate OAS bundle: `node scripts/capture_oas_snapshot --include-path
/api/streams --update`
* Apply overlays `cd oas_docs && make api-docs`
* Make sure bump.sh is installed (`npm install -g bump-cli`)
* Run for preview: `cd oas_docs && bump preview output/kibana.yaml`
# Open questions
* Does the split into public and internal make sense?
* Is it a problem if this is visible in the user-facing documentation
page before we actually release streams? Or would it be OK if the API is
marked as experimental? (mostly a question for @LucaWintergerst )
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
[META
issue](https://github.com/elastic/security-docs-internal/issues/57)
This PR improves the API documentation for timeline/notes/pinned events
as per the definition in the meta issue.
### Notes
#### Timeline API
- `eventType`:
- is always set to `all`
- looks like it's been superseded by dataView?
- I marked it as `deperecated`
- `eventIdToNoteIds`:
- there's a type mismatch between what the frontend expects and what is
returned
- also it does not seem to be used anymore?
- it needs to be investigated further
#### Notes API
- `GlobalNote`:
- looks like it's not used anymore
- I removed it, nothing broke
- `eventIngested`, `eventTimestamp`, `eventDataView` and `overrideOwner`
are all not used on the patch note endpoint, I removed them. The
`event*` ones I have never seen before to be honest.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This PR contains the following updates:
| Package | Type | Update | Change | Pending |
|---|---|---|---|---|
| [@redocly/cli](https://redirect.github.com/Redocly/redocly-cli) |
devDependencies | patch | [`^1.32.2` ->
`^1.33.0`](https://renovatebot.com/diffs/npm/@redocly%2fcli/1.33.0/1.33.0)
| `1.33.1` |
| [@redocly/cli](https://redirect.github.com/Redocly/redocly-cli) |
dependencies | patch | [`^1.32.2` ->
`^1.33.0`](https://renovatebot.com/diffs/npm/@redocly%2fcli/1.33.0/1.33.0)
| `1.33.1` |
---
### Configuration
📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about these
updates again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Renovate
Bot](https://redirect.github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xMDcuMCIsInVwZGF0ZWRJblZlciI6IjM5LjEwNy4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJUZWFtOkNvcmUiLCJiYWNrcG9ydDpwcmV2LW1pbm9yIiwicmVsZWFzZV9ub3RlOnNraXAiXX0=-->
Co-authored-by: elastic-renovate-prod[bot] <174716857+elastic-renovate-prod[bot]@users.noreply.github.com>
This PR contains the following updates:
| Package | Type | Update | Change | Pending |
|---|---|---|---|---|
| [@redocly/cli](https://redirect.github.com/Redocly/redocly-cli) |
devDependencies | patch | [`^1.32.0` ->
`^1.32.1`](https://renovatebot.com/diffs/npm/@redocly%2fcli/1.33.0/1.32.1)
| `1.33.0` (+1) |
| [@redocly/cli](https://redirect.github.com/Redocly/redocly-cli) |
dependencies | patch | [`^1.32.0` ->
`^1.32.1`](https://renovatebot.com/diffs/npm/@redocly%2fcli/1.33.0/1.32.1)
| `1.33.0` (+1) |
---
### Configuration
📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about these
updates again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Renovate
Bot](https://redirect.github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xMDcuMCIsInVwZGF0ZWRJblZlciI6IjM5LjEwNy4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJUZWFtOkNvcmUiLCJiYWNrcG9ydDpwcmV2LW1pbm9yIiwicmVsZWFzZV9ub3RlOnNraXAiXX0=-->
Co-authored-by: elastic-renovate-prod[bot] <174716857+elastic-renovate-prod[bot]@users.noreply.github.com>
## Summary
Resolves https://github.com/elastic/kibana/issues/206924.
This PR adds the following query parameters to the agent list API (`GET
/api/fleet/agents`) in order to enable fetching beyond the first 10,000
hits:
```
searchAfter?: string;
openPit?: boolean;
pitId?: string;
pitKeepAlive?: string;
```
The list agent API response can now include the following properties
```
// the PIT ID used
pit?: string;
// stringified version of the last agent's `sort` field,
// can be passed as `searchAfter` in the next request
nextSearchAfter? string;
```
* `searchAfter` can be used with or without a `pitId`. If using
`searchAfter`, `page` parameter is not accepted.
* `searchAfter` expects a stringified array. (Reviewers: I couldn't get
the Kibana request schema to accept a multi-part query param and convert
it to an array... I think this would be better, please let me know if
you know how to get that to work 🙏)
* `pitKeepAlive` duration (i.e. `30s`, `1m`, etc) must be present when
opening a PIT or retrieving results using a PIT ID.
* These can be used with the existing `sortField` and `sortOrder`
params. They default to `enrolled_at` and `desc` respectively.
### Example using only `searchAfter`:
```
# Retrieve the first 10k hits
curl -X GET 'http://<user>:<pass>@<kibana url>/api/fleet/agents?perPage=10000'
# Grab the `nextSearchAfter` param from the response
# Pass it to the new request to retrieve the next page of 10k hits
curl -X GET 'http://<user>:<pass>@<kibana url>/api/fleet/agents?perPage=10000&searchAfter=<nextSearchAfter>'
```
### Example using `searchAfter` with point-in-time parameters:
```
# Retrieve the first 10k hits and open a PIT
curl -X GET 'http://<user>:<pass>@<kibana url>/api/fleet/agents?perPage=10000&openPit=true&pitKeepAlive=5m'
# Grab the `pit` ID from the response
# Grab the `nextSearchAfter` param from the response
# Pass both to the new request to retrieve the next page of 10k hits
curl -X GET 'http://<user>:<pass>@<kibana url>/api/fleet/agents?perPage=10000&searchAfter=<nextSearchAfter>&pitId=<pit id>&pitKeepAlive=5m'
```
## Testing
I recommend using `scripts/create_agents` to generate bulk agents and
testing the above requests. You can generate new agents between PIT
requests to test that using a PIT ID retains the original state. (An API
functional test was added for this)
Note: you may need to add `&showInactive=true` to all requests if your
fake agents become inactive.
TBD
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Relates https://github.com/elastic/ingest-dev/issues/4720
This PR adds retry logic to the task that handles automatic agent
upgrades originally implemented in
https://github.com/elastic/kibana/pull/211019.
Complementary fleet-server change which sets the agent's
`upgrade_attempts` to `null` once the upgrade is complete.:
https://github.com/elastic/fleet-server/pull/4528
### Approach
- A new `upgrade_attempts` property is added to agents and stored in the
agent doc (ES mapping update in
https://github.com/elastic/elasticsearch/pull/123256).
- When a bulk upgrade action is sent from the automatic upgrade task, it
pushes the timestamp of the upgrade to the affected agents'
`upgrade_attempts`.
- The default retry delays are `['30m', '1h', '2h', '4h', '8h', '16h',
'24h']` and can be overridden with the new
`xpack.fleet.autoUpgrades.retryDelays` setting.
- On every run, the automatic upgrade task will first process retries
and then query more agents if necessary (cf.
https://github.com/elastic/ingest-dev/issues/4720#issuecomment-2671660795).
- Once an agent has completed and failed the max retries defined by the
retry delays array, it is no longer retried.
### Testing
The ES query for fetching agents with existing `upgrade_attempts` needs
the updated mappings, so it might be necessary to pull the latest `main`
in the `elasticsearch` repo and run `yarn es source` instead of `yarn es
snapshot` (requires an up-to-date Java environment, currently 23).
In order to test that `upgrade_attempts` is set to `null` when the
upgrade is complete, fleet-server should be run in dev using the change
in https://github.com/elastic/fleet-server/pull/4528.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Low probability risk of incorrectly triggering agent upgrades. This
feature is currently behind the `enableAutomaticAgentUpgrades` feature
flag.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
BUG: https://github.com/elastic/security-team/issues/11783
This PR fixes the behaviour of the
`/api/security_ai_assistant/chat/complete` route where the
`conversationId` can be passed as an empty string. This may lead to
unexpected results described in
https://github.com/elastic/security-team/issues/11783#issuecomment-2696529040.
### Expected behaviour
We should throw a bad request (400) http error when empty
`conversationId` has been passed.
### Testing
* Use this `curl` command to test the endpoint.
```
curl --location 'http://localhost:5601/api/security_ai_assistant/chat/complete' \
--header 'kbn-xsrf: true' \
--header 'Content-Type: application/json' \
--data '{
"connectorId": "{{my-gpt4o-ai}}",
"conversationId": "",
"isStream": false,
"messages": [
{
"content": "Follow up",
"role": "user"
}
],
"persist": true
}'
```
You should see next error as a response:
```
{
"statusCode": 400,
"error": "Bad Request",
"message": "[request body]: conversationId: String must contain at least 1 character(s), conversationId: No empty strings allowed"
}
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes logic on fresh cluster where the ELSER was not started yet, in
this case API reports `status` as `uninstalled`, but it doesn't mean
that the Product documentation was actually uninstall, but rather it's a
default state.
Added internal `product_documentation_status` to KB status API to make
sure we keep track of the status internally and present the banner only
if the docs were intentionally uninstalled
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## For reviewers:
Only `*.schema.yml` files were edited (excluding
`*.bundled.schema.yml`). Rest of the changes comes from auto generation
and can be ignored.
## Description
Part of DW team effort - elastic/security-team#11804
This PR aligns the property/schema descriptions and examples in
AsciiDocs with OpenAPI schemas. The primary goal of this PR was not to
extend or enhance the documentation but to migrate from one system to
another.
Ascii docs -
https://www.elastic.co/guide/en/security/8.17/management-api-overview.html
OpenApi generated docs -
https://www.elastic.co/docs/api/doc/kibana/operation/operation-endpointgetactionslist
Changes:
Copied missing property descriptions from AsciiDoc to OpenApi properties
Copied existing AsciiDoc examples for both requests and responses
Fixed falsy query object in some GET requests - in OpenApi it was
defined as an object, not as path query params.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Paul Tavares <56442535+paul-tavares@users.noreply.github.com>
Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com>
Part of DW team effort -
https://github.com/elastic/security-team/issues/11804
This PR aligns the property/schema descriptions and examples in
AsciiDocs with OpenAPI schemas. The primary goal of this PR was not to
extend or enhance the documentation but to migrate from one system to
another.
Ascii docs -
https://www.elastic.co/guide/en/kibana/8.17/osquery-manager-api.html
OpenApi generated docs -
https://www.elastic.co/docs/api/doc/kibana/operation/operation-osqueryfindlivequeries
Changes:
1. Copied missing property descriptions from AsciiDoc to OpenApi
properties
2. Copied existing AsciiDoc examples for both requests and responses
3. Fixed falsy query object in some GET requests - in OpenApi it was
defined as an object, not as path query params.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com>
Part of DW team effort -
https://github.com/elastic/security-team/issues/11804
This PR aligns the property/schema descriptions and examples in
AsciiDocs with OpenAPI schemas. The primary goal of this PR was not to
extend or enhance the documentation but to migrate from one system to
another.
Ascii docs -
https://www.elastic.co/guide/en/kibana/8.17/osquery-manager-api.html
OpenApi generated docs -
https://www.elastic.co/docs/api/doc/kibana/operation/operation-endpointgetactionslist
Changes:
Copied missing property descriptions from AsciiDoc to OpenApi properties
Copied existing AsciiDoc examples for both requests and responses
Fixed falsy query object in some GET requests - in OpenApi it was
defined as an object, not as path query params.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com>
## Summary
The `/packages` folder at the root of the Kibana repository used to
contain a lot of packages.
In the context of SKA, they have been gradually moved to various
locations:
* `src/platform/packages`
* `x-pack/platform/packages`
* `src/core/packages`
Currently, only `devOnly: true` packages are left in this folder. This
comprises libraries for CLI scripts as well as testing utilities.
With this PR, we are moving ~half of these packages under
`src/platform/packages/(private|shared)/`.
In particular, we are moving those packages that are being used from
platform and/or solutions.
Since they are `"devOnly": true`, this means they are ONLY used from
tests, cypress tests, storybook configs, ./scripts/ folders inside some
modules, or other non-prod-time logic. Nonetheless, they are effectively
referenced from platform and/or solutions code, hence I decided they
should be placed under `platform` folders.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
It closes#210712
This PR adds the following features:
- Introduces a new `status` API for the asset inventory, which returns
the status of the entity store, including statuses like
`insufficient_privileges`, `disabled`, `initializing`, `ready`, and
`empty`.
- Implements a mechanism to check the current status of the host entity
store and processes the transform component metadata.
It also introduces the following changes:
- Implemented the use of the host entity store to support the new status
API while awaiting the readiness of the 'generic' entity store.
- Added a `metadata` field to the entity store stats object and used it
to add additional metadata from the transform components.
- Replaced the retrieval of the entity store ID with the inclusion of
`stats.id` from the transform stats.
- Unit tests were added to ensure that the new API and status behavior
are functioning correctly, covering all possible states and edge cases.
### Key Changes:
- **API Changes**: A new `status` endpoint was added to query the asset
inventory's status, leveraging the host entity store for now.
- **Metadata Changes**: The `metadata` field was added to the entity
store stats to include transform-related metadata, such as
`documents_processed` and `trigger_count`.
- **Testing**: New unit tests were added to validate the behaviour of
the status API, including scenarios like insufficient privileges, entity
store installation, and document processing status.
### How to test it locally
To test it locally, execute the following in the Kibana dev tools:
```
GET kbn:/api/asset_inventory/status
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/211666
Allow any type in `PackageInfoSchema` and `KibanaAssetReferenceSchema`
to allow new type of epm packages without change in kibana.
Covered with unit test.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This pull request introduces the new `unassigned` criticality level for
the asset criticality's `bulk_upload` .
### Key Changes:
#### Schema and Configuration Updates:
* Added `unassigned` to the list of criticality levels in multiple
schema files (`kibana.serverless.yaml`, `kibana.yaml`,
`common.schema.yaml`,
`ess/security_solution_entity_analytics_api_2023_10_31.bundled.schema.yaml`,
`serverless/security_solution_entity_analytics_api_2023_10_31.bundled.schema.yaml`).
#### TypeScript and Constants:
* Updated `AssetCriticalityLevel` enum in `common.gen.ts` to include
`unassigned`.
* Added `UNASSIGNED` to `CriticalityLevels` enum and
`CriticalityModifiers` in `constants.ts`.
#### Tests:
* Updated test cases to include `unassigned` as a valid criticality
level in `parse_asset_criticality_csv_row.test.ts`,
`validations.test.ts`, `asset_criticality_data_client.test.ts`, and
`asset_criticality_csv_upload.ts`.
#### Backend Logic:
* Modified `AssetCriticalityDataClient` to handle `unassigned`
criticality level appropriately.
<img width="1488" alt="Screenshot 2025-01-30 at 2 03 11 PM"
src="https://github.com/user-attachments/assets/938411c4-725a-451c-ab38-aca36a704e91"
/>
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Testing Steps
The main idea for this change is that the `unassigned` criticality level
is actually marked as `deleted` in the ES documents.
ES index name for default space :
`.asset-criticality.asset-criticality-default`
#### API
1. `POST /api/asset_criticality/upload_csv`
```
curl --location 'http://localhost:5601/api/asset_criticality/upload_csv?output=stream' \
--header 'kbn-xsrf: hello' \
--header 'Accept: multipart/form-data' \
--header 'Authorization: *******' \
--form 'file=@"<Full path of the CSV file for asset criticality>"
```
Error response :
```
{"errors":[{"message":"Invalid criticality level \"unassigned_impact\", expected one of extreme_impact, high_impact, medium_impact, low_impact, unassigned","index":4}],"stats":{"successful":3,"failed":1,"total":4}}%
```
Success response :
```
{"errors":[],"stats":{"successful":4,"failed":0,"total":4}}%
```
Query the ES using below query to see if the criticality level is
`deleted`
```
GET .asset-criticality.asset-criticality-default/_search
{
"query": {
"match": {
"asset.criticality": "deleted"
}
}
}
```
### UI
1. Navigate to Entity Store page
2. Upload a csv file with incorrect asset criticality level
3. Rectify and upload the same file with correct criticality levels.
4. Navigate to EA Dashboard and scroll down to the Entities section.
5. Select an entity and open the flyout.
6. Try changing the asset criticality of the entity. No blank/empty
value should be present or assigned to asset criticality.
7. Should be able to successfully modify the asset criticality for the
entity.
Confirm this by querying the ES with the query :
```
GET .asset-criticality.asset-criticality-default/_search
{
"query": {
"match": {
"asset.criticality": "deleted"
}
}
}
```
Ensure that the Elastic search document's `_source` contains three keys,
with `deleted` present as the ` `criticality_level` for the below three
keys.
`criticality_level`
`host.asset.criticality`
`asset.criticality`
Example :
### Bulk upload
```
curl --location 'http://localhost:5601/api/asset_criticality/bulk' \
--header 'kbn-xsrf: hello' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic ****' \
--data '{
"records": [
{
"id_value": "host-1",
"id_field": "host.name",
"criticality_level": "low_impact"
},
{
"id_value": "host-2",
"id_field": "host.name",
"criticality_level": "medium_impact"
},
{
"id_value": "host-6",
"id_field": "host.name",
"criticality_level": "medium_impact"
},
{
"id_value": "host-3",
"id_field": "host.name",
"criticality_level": "high_impact"
},
{
"id_value": "host-4",
"id_field": "host.name",
"criticality_level": "high_impact"
},
{
"id_value": "host-bulkupload",
"id_field": "host.name",
"criticality_level": "unassigned"
}
]
}
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes a small UI bug in the citations feature. Previously, after a
message with citations finished streaming, the citations would disappear
for a fraction of a second and then reappear again. This PR makes
improves the UI by making the citations not flash off and on after the
stream finishes.
### Changes:
- Fix flashing citations
- Refactor code related to parsing content references (to make it more
maintainable).
- Update the citations prompt slightly.
### Before:
https://github.com/user-attachments/assets/1021dd53-018a-43ba-b1f4-24aab44faca9
<img width="1782" alt="image"
src="https://github.com/user-attachments/assets/723cd29a-48a2-48e7-b031-0893484746b9"
/>
### After:
https://github.com/user-attachments/assets/21f340bc-9015-42b6-a574-0439d2f8f192
### How to test
- Enable the feature flag
```yaml
# kibana.dev.yml
xpack.securitySolution.enableExperimental: ['contentReferencesEnabled']
```
- Open the security assistant
- Ask it a question about your alerts of a document in your KB. The
response should contain citations.
- Observe the response stream carefully. Ensure the citations e.g. `[1]`
do not flash off and on when the response stream finishes. The expected
behavior is that while the message is streaming, the citations are
disabled and once the stream finishes the citations get enabled (while
always being visible).
#### Edge case to test
It is possible that citations completely disappear after streaming
finishes. This happens when the LLM produces an invalid citation.
Invalid citations are hidden client side when a message finishes
streaming. You can verify this behavior by asking GPT4o this question:
```
Prepend each line with this placeholder citation "{reference(1234)}" and append the actual citation at the end of the line. How many alerts do I have? Use the open and acknowledged alerts count tool to answer and repeat the answer 50 times on new lines.
```
While the response is getting streamed it should look like this:
<img width="200" alt="image"
src="https://github.com/user-attachments/assets/03d160bf-2404-4a4e-8701-e3183c604cc4"
/>
And when the stream finishes it should look like this:
<img width="200" alt="image"
src="https://github.com/user-attachments/assets/06367379-17da-438f-a93a-9d539067ab90"
/>
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [X]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [X] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [X] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [X] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [X] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
# Summary
As part of the effort to add missing content for Security APIs, this PR
introduces a few missing request, response, and parameter examples for
Detection Engine Alert and migration APIs.
## Summary
We anticipate generating a single document per API version for the
foreseeable future. This PR updates our OAS document merging logic to
avoid adding the `Elastic-Api-Version` header to our spec.
This will also remove this header from current documentation.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR introduces support for configuring the `timestamp` field for
entity store enablement.
By default, the `timestamp` field is set to `@timestamp`, but users can
opt to use `event.ingested` or another preferred value based on their
requirements.
### Entity Store API changes
#### Entity Store enable
```
POST kbn:/api/entity_store/enable
{
"timestampField": "event.ingested"
}
```
#### Result
```
{
"engines": [
{
"status": "started",
"type": "user",
"indexPattern": "",
"filter": "",
"fieldHistoryLength": 10,
"timestampField": "event.ingested"
},
{
"status": "started",
"type": "host",
"indexPattern": "",
"filter": "",
"fieldHistoryLength": 10,
"timestampField": "event.ingested"
},
{
"status": "started",
"type": "universal",
"indexPattern": "",
"filter": "",
"fieldHistoryLength": 10,
"timestampField": "event.ingested"
}
],
"status": "running"
}
```
```
POST kbn:/api/entity_store/enable
{
}
```
#### Result
```
{
"engines": [
{
"status": "started",
"type": "host",
"indexPattern": "",
"filter": "",
"fieldHistoryLength": 10,
"timestampField": "@timestamp"
},
{
"status": "started",
"type": "user",
"indexPattern": "",
"filter": "",
"fieldHistoryLength": 10,
"timestampField": "@timestamp"
},
{
"status": "started",
"type": "universal",
"indexPattern": "",
"filter": "",
"fieldHistoryLength": 10,
"timestampField": "@timestamp"
}
],
"status": "running"
}
```
Different entity types can have distinct `timestampField` values, as
described below:
For instance, the `host` entity uses `event.ingested` as its
`timestampField`, while other entities default to the `@timestamp`
field.
```
POST kbn:/api/entity_store/engines/host/init
{
"timestampField": "event.ingested"
}
POST kbn:/api/entity_store/engines/user/init
{
}
```
#### Result
```
{
"engines": [
{
"status": "started",
"type": "universal",
"indexPattern": "",
"filter": "",
"fieldHistoryLength": 10,
"timestampField": "@timestamp"
},
{
"status": "started",
"type": "host",
"indexPattern": "",
"filter": "",
"fieldHistoryLength": 10,
"timestampField": "event.ingested"
},
{
"status": "started",
"type": "user",
"indexPattern": "",
"filter": "",
"fieldHistoryLength": 10,
"timestampField": "@timestamp"
}
],
"status": "running"
}
```
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Testing steps
- Check out this PR branch.
- Start Kibana.
- Avoid enabling the Entity Store from the UI initially. Instead, enable
it via the API by following the steps outlined below.
```
DELETE kbn:/api/entity_store/engines/user
DELETE kbn:/api/entity_store/engines/host
GET kbn:/api/entity_store/status
All engines with the same timestamp Field
POST kbn:/api/entity_store/enable
{
"timestampField": "event.ingested"
}
# All engines with the same timestamp Field
POST kbn:/api/entity_store/enable
{
}
# Different entity with different timestamp Field
POST kbn:/api/entity_store/engines/host/init
{
"timestampField": "event.ingested"
}
POST kbn:/api/entity_store/engines/user/init
{
}
```
4. Afterward, you can test the functionality through the UI. First,
clear the entity data from the UI, then start the Entity Store. Note
that the UI currently does not offer a way to configure this parameter.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Mark Hopkin <mark.hopkin@elastic.co>
## Summary
This PR adds the following parameters to the `INIT` engine API:
* `frequency`: the transform run frequency
* `timeout`: the timeout for the initial creation of the transform
* `docsPerSecond`: transform throttling option. See
[here](https://arc.net/l/quote/vxcmfnhh)
* `delay`: The transform delay duration. See
[here](https://arc.net/l/quote/mzvaexhv)
Coming soon
In addition, the PR adds these fields to the Saved Object with the
engine descriptor, as well as providing a migration with the appropriate
backfilling.
Finally, there are some utility function that were/are helpful in
working with objects.
## How to test
*NOTE*: Always make sure the security default data view exists. Easiest
way it to just navigate to some Security UI.
### Checking the new defaults
1. Initialize an engine via dev tools by calling: `POST
kbn:/api/entity_store/engines/<entity_type>/init {}`
2. Call `GET kbn:/api/entity_store/status`. This response should now
contain all the default optional values.
### Observing the parameters are being applied
1. Initialize an engine via the API. This time pass any of the `timeout,
frequency, delay and docsPerSecond` options in the request body.
2. Once the `status` changes to `started`, query the respective
transform: `GET
_transform/entities-v1-latest-security_<entity_type>_default`
3. Check that the parameters have been applied to the transform
### Checking Saved Object Migration
1. Check out `main`.
2. Initialize the store.
3. Query `GET kbn:/api/entity_store/status`. Note down the fields in the
engine object.
4. Check out this branch.
5. Restart kibana.
6. Query `GET kbn:/api/entity_store/status` again. Observe the new
fields have been added and backfilled
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
**Resolves:** https://github.com/elastic/security-team/issues/10410
## Summary
We want to make Rule Customization available at higher license tiers.
### **Intended Workflows/UX**
#### **Basic/Platinum/Security Essentials License Tiers**
- **Editing Prebuilt Rules:**
- Allow the 8.16 behavior: only actions, exceptions, snoozing, and
enable/disable options can be modified.
- On the rule editing page, all tabs except *Actions* are disabled.
Disabled tabs will display a hover explanation:
- "Upgrade to Enterprise to enable prebuilt rule customization" for ECH.
- "Upgrade to Security Complete to enable prebuilt rule customization"
for Serverless.
<img width="356" alt="image"
src="https://github.com/user-attachments/assets/72e60933-aaaf-45a0-9660-4cd066d3afec"
/>
- Rule editing via API is not restricted (tracked separately:
https://github.com/elastic/security-team/issues/11504.
- **Bulk Actions:**
- Modifications to rule content via bulk actions are not allowed.
Prebuilt rules are excluded from bulk actions if the license level is
insufficient. Users will see an explanation for the exclusion.
- Serverless
<img width="737" alt="image"
src="https://github.com/user-attachments/assets/99fef72f-dd38-4c73-a9e3-7b4c8018b4ed"
/>
- ECH
- On the API level (`_bulk_action`), an error is returned if a user
tries to modify a prebuilt rule without the required license. Response
in this case looks like this:
```json
{
"statusCode": 500,
"error": "Internal Server Error",
"message": "Bulk edit failed",
"attributes": {
"errors": [
{
"message": "Elastic rule can't be edited",
"status_code": 500,
"rules": []
}
]
}
}
```
- **Rule Updates:**
- Updates are restricted to Elastic’s incoming updates only.
- The rule upgrade flyout is in read-only mode.
<img width="949" alt="image"
src="https://github.com/user-attachments/assets/16a56430-63e6-4096-8ffd-b97f828abdd4"
/>
- For previously customized rules where customization is now disabled
due to insufficient licensing, a notification will appear on the upgrade
flyout, clarifying that only an upgrade to Elastic's version is
available.
- On the API level (`_perform`), only requests with `pick_version =
target` are permitted. Requests with `rule.fields` values are not
allowed.
API response when `pick_version` is not `target`:
```json
{
"message": "Only the 'TARGET' version can be selected for a rule update;
received: 'CURRENT'",
"status_code": 400
}
```
API response when the `fields` value is provided:
```json
{
"message": "Rule field customization is not allowed. Received fields:
name, description",
"status_code": 400
}
```
- **Customized Rules:**
- Existing customizations remain intact, and the “Modified” badge is
retained
- On the rule management, monitoring, and update tables:
- On the rule update flyout:
- On the rule details page:
- When we edit a rule with customizations (e.g., change rule's actions),
the rule should stay marked as customized
- **Import/Export Scenarios:**
- These are handled separately
(https://github.com/elastic/security-team/issues/11502)
#### **Enterprise/Security Complete License Tiers**
- All rules can be fully edited
- Upgraded prebuilt or customized rules will have an editable view,
enabling full customization
Closes https://github.com/elastic/kibana/issues/186266
## Release notes
Enhanced Role management to manage larger number of roles by adding
server side filtering, pagination and querying.
## Summary
- Replaced the usage of Get Roles API with Query Role API
- Added server side pagination and filtering with a maximum limit of
10000 keys (default for max results on index). Added new label to
indicate that we show only 10k results.
- Search box replicates client side implementation by only filtering on
Role names.
### Run locally
Start ES ~with the JVM option to enable this feature~:
```
yarn es snapshot --license=trial
```
Start Kibana normally
```
yarn start --no-base-path
```
Navigate to Stack Management > Roles and verify the same behavior as the
screen recording below
### Screen recording
https://github.com/user-attachments/assets/a447e7df-8aa1-4044-a6b2-0aafe56844a9
## Technical notes
- Client side EuiInMemory table has been replaced by EuiSearchBar,
EuiBasicTable and Filters
- One new Kibana endpoint added
- `roles/_query`
- Replicates existing get_role endpoint by being public and added to
Open API spec
- Extra logic to handle previously UI only filter to show/hide reserved
roles
- Parse the query to construct the correct DSL if the filter is present
- Update Get All Roles by Space internal API to use the Query Role and
filter by space id using query DSL.
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
> [!note]
> Planning to merge before the 9.0 feature freeze.
Documentation update issue:
https://github.com/elastic/security-docs/issues/6473
> [!tip]
> ### Tip for the reviewer
> As a starting point to review this PR I would suggest reading the
section "How does it work (on a high level)" and viewing the hyperlinked
code. The linked code covers the main concepts of this feature and the
majority of the remaining changes in the PR are related to API schema
updates and tests.
## Summary
This PR adds citations to the security AI assistant. Citations are
produced when tools are used and they are displayed in the LLM response
as numbered superscript elements. A label appears when the user hovers
over the numbered elements and clicking on the label opens a new tab
that displays the cited data.
## How to test:
1. Enables the feature flags:
- Set value to `true`
[here](https://github.com/elastic/kibana/pull/206683/files#diff-f55be7c50853801c3933b48064ab9cbf0356e2941cd97c4c365be7a6ded9bffdR125)
2. Populate the security knowledge base with some information (e.g. a
document, an index, product documentation, the global threat report,
etc...).
3. Open the security assistant
4. Ask it the following questions about data in the knowledge base:
- What is an elastic search cold tier? - Make sure product docs are
installed
- What topics are covered in the security lab content?
- Ask it a question about one of your knowledge base documents.
- Which platform did my most recent alert happen on? - Make sure you
have a recent alert
## How does it work (on a high level)?
Citations are stored inside the
[ContentReferencesStore](https://github.com/elastic/kibana/pull/206683/files#diff-baf03ce192db4f13999748a38b4d920428358a4ffc62527a1d6ac0d9b234f306R17)
object. When tools are called, the tools [add citations to the
ContentReferencesStore](https://github.com/elastic/kibana/pull/206683/files#diff-5a333fdd9bf864dced06500263577e495c95c9b32c7dae9074090775df542d22R97-R99)
and pass the Id of the [ContentReferences back to the
LLM](https://github.com/elastic/kibana/pull/206683/files#diff-5a333fdd9bf864dced06500263577e495c95c9b32c7dae9074090775df542d22R102)
along side the result of the tool. The LLM can then use those
contentReference IDs in its response by forming a response like:
```
The sky is blue {reference(12345)}
```
The web client [parses out the
contentReference](https://github.com/elastic/kibana/pull/206683/files#diff-3a5c8305ac899a9e78903b0b60141dd997ba61e87342de2b9ec377165d99cfe6R23)
(`{reference(12345)}`) from the assistant message and[ replaces it with
the citation react
component](https://github.com/elastic/kibana/pull/206683/files#diff-db928fb87a862e3ebf7247baefc418de539f9c0f3fc5134a2ef56f921a52bdcbR125-R129).
### Tools that are cited:
Include citations for the following tools:
alert_counts_tool -> cites to alerts page
knowledge_base_retrieval_tool -> cites knowledge base management page
with specific entry pre-filtered
open_and_acknowledged_alerts_tool -> cites to specific alert
security_labs_tool -> cites knowledge base management page with specific
entry pre-filtered
knowledge_base indices -> opens ESQL view selecting the particular
document used
product_documentation -> cites documentation
### Endpoints impacted
- POST
/internal/elastic_assistant/actions/connector/{connectorId}/_execute
- POST /api/security_ai_assistant/chat/complete
- GET /api/security_ai_assistant/current_user/conversations/_find
- GET /api/security_ai_assistant/current_user/conversations/:id
- PUT /api/security_ai_assistant/current_user/conversations/{id}
### Considerations:
- One of the main objectives of this feature was to produce in-text
citations to create a great user experience. Multiple approaches were
tested to do this reliably. Attempts were made to make the LLM return
structured JSON containing the citations however this was unreliable
with smaller models. Generation post-processing (issuing an additional
LLM call to annotate the response with citations) was also explored
however this also had limitations as the second LLM call would not
contain enough contextual information to create the citations reliably.
Eventually, the approach described in the section above was used
alongside few shot promoting.
- Instead of using the ContentReferencesStore to store citations, the
langGraph state could be used to save the citations. I looked at doing
this but currently, there are a few blockers in the langgraph API the
prevent this.
- Lang graph must be updated to @langchain/langgraph>=0.2.31 to get
access to the Command type so that tools can update the graph state.
- It seems that DynamicStructuredTools do not support the Command type
yet. This is something that we can clarify with the langchain team.
Once these blockers have been addressed, ContentReferencesStore could
easily be refactored to the graph state.
- The feature has been put behind a feature flag so we can test during
the feature freeze and sync the release of the documentation update. The
only thing that is not behind a feature flag is the new anonymization
button in the settings menu (don't think it is necessary and it means a
lot more code changes are required).
On few occasions, you can nudge the LLM a bit more to include citations
by appending "Include citations" to your message.
Furthermore, the settings menu has been updated to include anonymized
values and citation toggles:
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [X] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [X] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
## Release note
Adds in-text citations to security solution AI assistant responses.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
