Closes#212957
Adds editing of the automatically-generated readMe file on custom
integrations.
- Allows the user to make edits and then save the readMe changes via a
new endpoint
- Reloads the UI after making changes to reflect the updated readMe
- Automatically updates associated policies
To test the new endpoint, you will need a custom integration installed,
then use it as such
```
PUT kbn:/api/fleet/epm/custom_integrations/{pkgName}
{
"readMeData": "New README content here"
}
```
https://github.com/user-attachments/assets/9a6f2197-aa7f-4610-9476-c1f8b4865c62
### Acceptance criteria
- [ ] An "edit" button with an icon appears in the README section of the
integration overview page for custom integrations generated by automatic
import
- [ ] Clicking the "edit" button should open a modal (TBD) containing a
markdown editor that allows the user to edit the content
- [ ] Users can save their changes, and the updated README content is
persisted
- [ ] Saving updates the version of the package and reloads the content
- [ ] Saving also starts updating policies automatically
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
N/A
# Release Note
Adds edit functionality to custom integrations, allowing a user to edit
the README file of a custom integration and save it to be persisted.
Additionally, saving will automatically increment the version of the
integration and update all associated policies.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Resolves#210025
Introduces starter API for bulk purging of SLI data.
**docs coming pending review**
Users can submit a list of SLO ids for which they would like to purge
rollup data, as well as a purge policy:
```
"purgePolicy": {
"purgeType": "fixed_age",
"age": "1w"
}
```
```
"purgePolicy": {
"purgeType": "fixed_time",
"age": "2025-04-01T00:00:00Z"
}
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## 🗒️ Summary
We want to add a new `root-level` key in the rule called `artifacts`.
The new key will be added to the `create`, `update` and `find` Rule apis
and will contain `dashboards` and `investigation_guide`. This PR handles
only the addition of dashboards.
```
artifacts: {
dasboards: Array<{ id: string }]>;
}
```
## Architecture diagram
<img width="1265" alt="Screenshot 2025-04-02 at 12 35 00"
src="https://github.com/user-attachments/assets/51783fd5-f63e-4a2e-80b0-7a724675c11c"
/>
## 🧪 How to test
This PR includes only backend changes to the Rule APIs. In order to test
you can use Kibana > Dev Tools.
### Create a rule
```
POST kbn:/api/alerting/rule
{
"name": "TEST RULE",
"params": {
"criteria": [
{
"comparator": ">",
"metrics": [
{
"name": "A",
"aggType": "count"
}
],
"threshold": [
3
],
"timeSize": 1,
"timeUnit": "m"
}
],
"alertOnNoData": false,
"alertOnGroupDisappear": false,
"searchConfiguration": {
"query": {
"query": "",
"language": "kuery"
},
"index": "e7744dbe-a7a4-457b-83aa-539e9c88764c"
}
},
"rule_type_id": "observability.rules.custom_threshold",
"consumer": "logs",
"schedule": {
"interval": "1m"
},
"artifacts": {
"dashboards": [
{ "id":"1" }
]
}
}
```
The artifacts shouldn't be returned in the rule response. The new
dashboard artifacts should be saved in the Rules SO. You can check it
directly under `Kibana > Stack Management > Saved Objects`. You should
verify a new `artifacts` entry is added in the rules SO:
```
"artifacts": {
"dashboards": [
{
"refId": "dashboard_0"
}
]
},
```
Also the new dashboard should be added into the references array:
```
"references": [
..
{
"id": "1",
"name": "dashboard_0",
"type": "dashboard"
}
],
```
### Update a rule
```
PUT kbn:/api/alerting/rule/{RULE_ID}
{
"name": "TEST RULE",
"schedule": {
"interval": "1m"
},
"params": {
"criteria": [
{
"comparator": ">",
"metrics": [
{
"name": "A",
"aggType": "count"
}
],
"threshold": [
3
],
"timeSize": 1,
"timeUnit": "m"
}
],
"alertOnNoData": false,
"alertOnGroupDisappear": false,
"searchConfiguration": {
"query": {
"query": "",
"language": "kuery"
},
"index": "e7744dbe-a7a4-457b-83aa-539e9c88764c"
}
},
"artifacts": {
"dashboards": [{"id": "2"}, {"id": "1"}]
}
}
```
The artifacts shouldn't be returned in the rule response. The updated
artifacts should be saved in the Rules SO. You can check it directly
under `Kibana > Stack Management > Saved Objects`. You should verify the
updated dashboards list is correct.
### Find a rule
```
GET kbn:/api/alerting/rules/_find?search="TEST RULE"
```
The artifacts shouldn't be returned in the rule response. Only the
internal Find API returns the artifacts.
```
POST kbn:/internal/alerting/rules/_find
{
"search": "TEST RULE"
}
```
## ✔️ Acceptance criteria
- The dashboard IDs should be saved as `references in the Rule SO
- The Rule APIs will not return the artifacts
- Only the `internal Find API` used by the UI will return the artifacts
- Integrations and unit tests should be added
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Justin Kambic <jk@elastic.co>
Co-authored-by: Christos Nasikas <xristosnasikas@gmail.com>
## Summary
Allows one to export and import content packs in archive format. The
format follows the integration content package's format so it becomes
possible to import existing integration packages.
Content packs only support dashboard assets at the moment.
A pattern replacement logic has been implemented for dashboards and
referenced data views:
- at export time, any pattern matching the source stream will be
replaced with a placeholder. Other patterns will remain as-is unless
user explicitly ask to replace them
- at import time, the placeholders are replaced with the target stream
pattern
For example, if a dashboard is first exported from stream `logs.nodejs`
and reads data from patterns `logs.nodejs` and `logs.nodejs.prod`, the
patterns will be updated to `logs.ruby` and `logs.ruby.prod` when
imported into `logs.ruby` stream.
The relevant UI components are hidden behind a feature flag, set the
following in `kibana.dev.yml` to enable them:
`feature_flags.overrides.featureFlagsStreams.contentPackUIEnabled: true`
https://github.com/user-attachments/assets/9fb07daf-9fb9-4c62-9f5b-387e1833eaf0
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
## Summary
Introduce a new Entity Engine Definition called Generic. The larger
context on why we are introducing a new entity definition is described
on this [private github
issue](https://github.com/elastic/security-team/issues/11857).
The tldr; is that we would like to have an entity store with all the
entities described by the [entity ecs
field](https://github.com/elastic/security-team/issues/11857). The
decision to call `generic` entity definition comes from the fact that
any entity can be described with the `entity` field - user, host,
service, database, queue, subscription and so on. Therefore it makes
sense to have the concept called `generic` entity, meanwhile the
existent entity definitions will be called concrete entities, because
they describe a very concrete type of entity (currently user, host,
service).
Other changes included on this PR:
- Don't override `entity.name` with `entity.id`, only set if no value is
found
- Migrate the usage of `entity.type` as the entity definition type to
`entity.EngineMetadata.Type`
- Changes touching Entity Analytics code around
`getRiskEngineEntityTypes` and `getAssetCriticalityEntityTypes`. There
was a somewhat unnecessary and duplicated logic in these functions which
essentially described the concrete entity definitions to be used by
entity analytics flows. A new function called
`getEntityAnalyticsEntityTypes` was introduced which unifies this logic
and returns the entity types that Entity Analytics care about.
Video of a scroll through the entities processed by the generic entity
store, source of the data is cloudbeat asset management integration.
https://github.com/user-attachments/assets/450afd05-dee0-4449-aaec-2cd69645d6ec
#### How to test:
- In Advanced Settings (`/app/management/kibana/settings`), enable
`securitySolution:enableAssetInventory`
<img width="883" alt="image"
src="https://github.com/user-attachments/assets/c342abb2-efb3-40a8-b945-d9558f085f34"
/>
- In Entity Store management (`/security/entity_analytics_entity_store`)
enable entity store
<img width="1251" alt="image"
src="https://github.com/user-attachments/assets/41f709e1-0aea-47dc-9c98-ffaebf18fdb1"
/>
- Verify Generic Engine Status
<img width="1203" alt="image"
src="https://github.com/user-attachments/assets/d26b764a-4695-436e-85f7-e3ed7df5a3be"
/>
- Ingest documents with `entity.id` and `entity.*` fields. Personally I
run `cloudbeat` asset discovery locally
- Verify ingested documents in
`.entities.v1.latest.security_generic_default`
<img width="1496" alt="image"
src="https://github.com/user-attachments/assets/88286cb9-38c1-4f9d-83a7-57ba33811c60"
/>
--
**OBS: Also test enabling the store without the uiSetting enabled, so
you can make sure that it doesn't enable**
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
- [x] I see risk on performance, given the amount of aggregations the
generated transform does
- tested, although we see a higher spike in CPU than before, it's behind
a feature flag and it's going to be used in controlled data sets (entity
centric logs that contain `entity.id` field) we decided it's good enough
to go.
- [ ] Enablement/disablement of entity store in a different uiSetting
configuration.
- [ ] Enable entity store with `securitySolution:enableAssetInventory`
disabled. Then enable `securitySolution:enableAssetInventory` ==> No
generic entity definition installed. You can manually install it in the
EntityStore status page
- [ ] Enable entity store with `securitySolution:enableAssetInventory`
enabled. Then disable `securitySolution:enableAssetInventory` definition
==> hanging assets of generic entity store that can be deleted manually
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR introduces the first building blocks for the [Entity Analytics
Privileged
Monitoring](https://github.com/elastic/security-team/issues/9971).
We follow the approach used in the Entity Store and add a new "Engine",
which consists of the following components:
* Public API
* INIT and HEALTH routes
* Kibana task
* Privilege Monitoring Data Client
* Engine Saved Object
* API key manager
* Related storage indices
* Feature Flag: `privilegeMonitoringEnabled` set to `false` by default.
* API integration test configuration
* only tests that the health endpoint is available
* Auditing and Telemetry
## Testing steps
1. Make sure to add `privilegeMonitoringEnabled` to your
`kibana.dev.yaml`
2. In devtools, ensure the API is working with `GET
kbn:/api/entity_analytics/monitoring/privileges/health`
3. Start the engine with: `POST
kbn:/api/entity_analytics/monitoring/engine/init`
4. Look for `DEBUG` logs mentioning the
`entity_analytics:monitoring:privileges:engine` task
---------
Co-authored-by: CAWilson94 <charlotte.wilson@elastic.co>
Co-authored-by: Charlotte Alexandra Wilson <CAWilson94@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/206556
This PR adds a setting to remote ES outputs for also uninstalling
integrations on remote clusters when integrations sync is enabled.
This new setting can be toggled in the UI with a new switch:
<img width="1728" alt="Screenshot 2025-04-09 at 11 53 43"
src="https://github.com/user-attachments/assets/34544aa9-28fd-4360-a32f-5031e3d4293f"
/>
### Testing
* Follow the steps in
https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/fleet/dev_docs/local_setup/remote_clusters_ccr.md
to set up two clusters with integrations syncing.
* Add some integrations in your main cluster and check that they are
also installed in the remote cluster.
* Disable uninstalling integrations on remote.
* Remove an integration in your main cluster and check that it is NOT
removed from the remote cluster.
* Enable uninstalling integrations on remote.
* Remove an integration in your main cluster and check that it is also
removed from the remote cluster.
* In your remote cluster, enroll an agent onto a policy that points to
at least 1 package policy of the installed integrations (cf. Docker
commands below if using dockerized fleet-server/agent).
* In your main cluster, uninstall the integration that is used by the
agent policy in the remote. This should cause the uninstall to fail into
the remote cluster.
* In your remote cluster, inspect the package SO of that integration
with `GET .kibana_ingest/_search?q=type:epm-packages`: the
`latest_uninstall_failed_attempts` field should be populated.
Docker command for running a fleet-server in your remote cluster:
```
docker run \
-e ELASTICSEARCH_HOST=http://host.docker.internal:9500 \
-e KIBANA_HOST=http://host.docker.internal:5701/<path> \
-e KIBANA_USERNAME=elastic \
-e KIBANA_PASSWORD=changeme \
-e KIBANA_FLEET_SETUP=1 \
-e FLEET_INSECURE=1 \
-e FLEET_SERVER_ENABLE=1 \
-e FLEET_SERVER_POLICY_ID=fleet-server-policy \
-p 8220:8220 \
--rm docker.elastic.co/beats/elastic-agent:9.0.0-SNAPSHOT
```
Docker command for enrolling an agent in your remote cluster:
```
docker run \
-e ELASTICSEARCH_HOST=http://host.docker.internal:9500 \
-e KIBANA_HOST=http://host.docker.internal:5701/<path> \
-e FLEET_URL=https://host.docker.internal:8220 \
-e FLEET_ENROLL=1 \
-e FLEET_ENROLLMENT_TOKEN=<token> \
-e FLEET_INSECURE=1 \
--rm docker.elastic.co/beats/elastic-agent:9.0.0-SNAPSHOT
```
### Checklist
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
This feature is currently in development and behind the
`enableSyncIntegrationsOnRemote` feature flag.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
**Resolves: https://github.com/elastic/security-docs/issues/3019**
## Summary
In this PR I am improving documentation for the bulk_actions endpoint.
The focus is on improving descriptions of parameters and providing
examples for all important situations, especially setting / adding
actions.
# Testing
1. cd x-pack/solutions/security/plugins/security_solution
2. yarn openapi:bundle:detections
3. Take the bundled file
(docs/openapi/ess/security_solution_detections_api_2023_10_31.bundled.schema.yaml)
and load it into bump.sh console to see the changes.
4. Compare the changes with the [Legacy
documentation](https://www.elastic.co/guide/en/security/current/rule-api-overview.html)
You can also use this [link](https://bump.sh/jkelas/doc/kibana_wip/)
where I deployed the generated bundled doc.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Closes https://github.com/elastic/kibana/issues/192363
## Summary
Add endpoint that compares integrations installed on remote cluster with
integrations in ccr index `fleet-synced-integrations-ccr-<outputId>`.
Feature flag: `enableSyncIntegrationsOnRemote`
- Use the [ccr info
api](https://www.elastic.co/guide/en/elasticsearch/reference/current/ccr-get-follow-info.html)
to check that the ccr index is active
- Compare the content of the two indices and report the sync status for
each integration:
```
GET kbn:/api/fleet/remote_synced_integrations/status
{
"integrations": [
{
"package_name": "akamai",
"package_version": "2.28.0",
"updated_at": "2025-03-27T10:29:52.485Z",
"sync_status": true
},
{
"package_name": "auth0",
"package_version": "1.21.0",
"updated_at": "2025-03-26T12:06:26.268Z",
"sync_status": false,
"error": "Installation status: not_installed"
},
]
```
- Check that the integrations have custom assets and report their sync
status (follow up of https://github.com/elastic/kibana/pull/215504)
### Testing
Setup local env with the guide added in dev_docs
([preview](b438a719ee/x-pack/platform/plugins/shared/fleet/dev_docs/local_setup/remote_clusters_ccr.md))
- Install some integrations on local cluster, wait that they are synced
on remote
- From remote cluster dev tools, run
```
GET kbn:/api/fleet/remote_synced_integrations/status
```
- To verify that custom assets are synced choose an integration, for
instance `system`
- From the package policy select a var, advanced options and add a
custom mapping and a custom pipeline. In my example I used `system`
<img width="1195" alt="Screenshot 2025-04-01 at 11 18 40"
src="https://github.com/user-attachments/assets/c339d6f0-4404-4e9b-b489-7b37977f201b"
/>
- Run the endpoint again and you should see the status of custom assets
too:
```
{
"integrations": [
{
"package_name": "akamai",
"package_version": "2.28.0",
"updated_at": "2025-03-27T10:29:52.485Z",
"sync_status": "completed"
},
{
"package_name": "elastic_agent",
"package_version": "2.2.0",
"updated_at": "2025-03-26T14:06:29.216Z",
"sync_status": "completed"
},
{
"package_name": "synthetics",
"package_version": "1.4.1",
"updated_at": "2025-03-26T14:06:31.909Z",
"sync_status": "completed"
},
{
"package_name": "system",
"package_version": "1.67.3",
"updated_at": "2025-03-28T10:08:00.602Z",
"sync_status": "completed"
}
],
"custom_assets": {
"component_template:logs-system.auth@custom": {
"name": "logs-system.auth@custom",
"type": "component_template",
"package_name": "system",
"package_version": "1.67.3",
"sync_status": "completed"
},
"ingest_pipeline:logs-system.auth@custom": {
"name": "logs-system.auth@custom",
"type": "ingest_pipeline",
"package_name": "system",
"package_version": "1.67.3",
"sync_status": "completed"
}
}
}
```
### Checklist
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes
[4718](https://github.com/elastic/ingest-dev/issues/4718?reload=1?reload=1?reload=1%3Freload%3D1)
## Summary
- After much discussion, updated to not show percentages but be very
similar to existing upgrade text, but with an icon/tooltip letting the
user know it was created from an automatic upgrade. Also applied to
completed actions.
- Made `policyId` persistent in newly created actions in order to
streamline accessing the upgrade modal from agent activity
- Added `is_automatic` field to `ActionStatus` type by retrieving from
the source doc in `getActions`
- Updated audit log to show if the action was created by the user or
from the auto-upgrade functionality
- Updated badging on table to show retry attempts
- Added `manage auto-upgrade` button to agent activity actions created
by automatic upgrades
- Updated check in `automatic_upgrade_task` to only consider active
agents in order to resolve an issue where uninstalled agents could
affect the upgrade of new ones.
- Reworked rounding functionality when percentages or counts of agents
to upgrade were over or under where they should be. Rounding is now done
in a way such that no agents get left behind, and we dont try to upgrade
more than exist.
- Added new test coverage for the rounding functionality, the new active
vs inactive agents check, as well as ensuring the manage auto-upgrades
button always renders.
Simplified UI with tooltip and button to quickly access auto-upgrade
settings for the policy the action belongs to:
Updated tooltip to let the user know that rounding is in place:
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
N/A
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
* Removes manually maintained OAS for Core's `_export` and `_import`
Saved Object APIs
* seeks to remove some code duplication and move our OAS descriptions
closer to the code definitions.
## Summary
Some users using the security AI assistant through the API may want to
disable content references (aka citations) programmatically. This PR
allows users to disable content references using the
`content_references_disabled` query arg in the following routes:
- POST
/internal/elastic_assistant/actions/connector/gpt4oAzure/_execute?content_references_disabled=true
- POST
/api/security_ai_assistant/chat/complete?content_references_disabled=true
## How to test:
Bellow are 2 example curl requests (with
content_references_disabled=true). When executed, you will notice that
the response stream does not contain the string `{reference(...)}`
anywhere. If you remove the query arg, the reference will be visible.
####
/internal/elastic_assistant/actions/connector/gpt4oAzure/_execute?content_references_disabled=true
```curl
curl --location 'http://localhost:5601/internal/elastic_assistant/actions/connector/gpt4oAzure/_execute?content_references_disabled=true' \
--header 'Accept: */*' \
--header 'Accept-Language: en-GB,en-US;q=0.9,en;q=0.8' \
--header 'Connection: keep-alive' \
--header 'Content-Type: application/json' \
--header 'Origin: http://localhost:5601' \
--header 'Referer: http://localhost:5601/app/security/dashboards?sourcerer=(default:(id:security-solution-default,selectedPatterns:!()))&timeline=(activeTab:query,graphEventId:%27%27,isOpen:!f)' \
--header 'Sec-Fetch-Dest: empty' \
--header 'Sec-Fetch-Mode: cors' \
--header 'Sec-Fetch-Site: same-origin' \
--header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36' \
--header 'elastic-api-version: 1' \
--header 'kbn-build-number: 9007199254740991' \
--header 'kbn-version: 9.1.0' \
--header 'sec-ch-ua: "Chromium";v="134", "Not:A-Brand";v="24", "Google Chrome";v="134"' \
--header 'sec-ch-ua-mobile: ?0' \
--header 'sec-ch-ua-platform: "macOS"' \
--header 'x-elastic-internal-origin: Kibana' \
--header 'x-kbn-context: %7B%22type%22%3A%22application%22%2C%22name%22%3A%22securitySolutionUI%22%2C%22url%22%3A%22%2Fapp%2Fsecurity%2Fdashboards%22%2C%22page%22%3A%22list%22%7D' \
--header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \
--data '{
"subAction": "invokeStream",
"actionTypeId": ".gen-ai",
"replacements": {},
"message": "What is elastic security? Include citations",
"screenContext": {
"timeZone": "Asia/Calcutta"
},
"alertsIndexPattern": ".alerts-security.alerts-default",
"size": 100
}'
```
####
/api/security_ai_assistant/chat/complete?content_references_disabled=true
```curl
curl --location 'http://localhost:5601/api/security_ai_assistant/chat/complete?content_references_disabled=true' \
--header 'Content-Type: application/json' \
--header 'Accept: application/octet-stream' \
--header 'kbn-xsrf: 123' \
--header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \
--data '{
"messages": [
{
"role": "user",
"content": "what is semantic text in esql. Include references to the product documentation"
}
],
"persist": false,
"connectorId": "gpt4oAzure",
"isStream": false
}'
```
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Currently, our `requiredPrivileges` structure supports `allRequired` and
`anyRequired` for defining authorization logic. However, there is [a
need to
support](https://github.com/elastic/kibana/pull/205335#issuecomment-2569275302)
more complex scenarios as `(privilege1 AND privilege2) OR (privilege3
AND privilege4)`
To achieve `anyRequired` has been extended to allow defining multiple
AND conditions evaluated with OR logic:
```ts
security: {
authz: {
requiredPrivileges: [{
anyRequired: [
{ allOf: ['privilege1', 'privilege2'] },
{ allOf: ['privilege3', 'privilege4'] }
]
}
]
}
}
```
`allRequired` now also supports scenarios `(privilege1 OR privilege2)
AND (privilege3 OR privilege4)`
```ts
security: {
authz: {
requiredPrivileges: [{
allRequired: [
{ anyOf: ['privilege1', 'privilege2'] },
{ anyOf: ['privilege3', 'privilege4'] }
]
}
]
}
}
```
> [!IMPORTANT]
> We expect to have unique privileges in `anyOf` or `allOf` conditions,
assuming that most complex conditions can be simplified by boolean
algebra laws (OR/AND distributive etc).
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
__Closes: https://github.com/elastic/kibana/issues/210977__
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/streams-program/issues/38.
- Adds configuration driven support for processors
- Adds support for the following processors:
- Geoip
- KV
- Rename
- Set
- URL Decode
- User agent
## Notes
- `if` is now conditional as not all processors support a condition
- With `set` you can either have a `value` OR a `copy_from`. To avoid
types, form, and validation complexity for now I've just allowed the use
of `value`.
- I didn't add `extract_device` to the user agent configuration as it's
Beta.
## Future improvements
- We could add configuration for the grok, dissect, and date processors
too. This would align things so that each processor has things "in one
place". We could extend the field types available to provide a `custom`
type, this would allow those processors to return a React component for
their specialised fields / UI (still driven by `react-hook-form` under
the hood). Overall the code footprint would decrease.
- A couple of types can possibly be refined but I wanted to try and get
this open before the Milestone 2 deadline.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
The error callouts were getting squished to the side, I have moved them
to be in their respective engine section.
**Before**
<img width="1728" alt="Screenshot 2025-03-27 at 11 05 38"
src="https://github.com/user-attachments/assets/c244acd1-a75e-4b43-a509-8bff9966b31e"
/>
**After**
<img width="1728" alt="Screenshot 2025-03-27 at 19 39 53"
src="https://github.com/user-attachments/assets/2b0dd9ad-568a-4e9c-aad7-6922b2085d69"
/>
Bonus refactor:
I have added type definitions for the error object keys on the engine
status object and moved the error callout to a common component as it's
now used in 3 places.
## Test Steps
the easies way to test this is to try and init the entity store in a
space that is lobger than 45 chars long.
- create a space with a very long name e.g
'this_is_a_very_long_space_name_indeed_hello_world_123456789'
- enable the entity store from the entity analytics dashboard (it should
error)
- navigate to the entity store management page to see the error
callouts.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This change reproduces the asciidoc change made in
https://github.com/elastic/kibana/pull/209403 into the new API docs.
It'll be picked up the next time the API docs output is generated and
refreshed.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
**Partially resolves: #211808**
## Summary
This is the first part of the migration effort, containing changes for:
- CRUD endpoints
- BULK Actions
- Export / Import Rule
- Find Rule
- List Tags
- Get Status
- Install Rule
I migrated the examples and the description of the fields. Some of the
fields contained description that was very similar to the legacy, then I
didn't change it. I only modified the descriptions where it was
valuable.
I also discovered some problems. For example the value for the 'query'
field, always shows 'EQL query to execute'. I reported this to the docs
team, in the 'next-api-reference' channel.
Another issue was with 'related_integrations field', which also didn't
show the long description. I also wrote about it to the docs team
[here](https://elastic.slack.com/archives/C05UL5YC06B/p1740137094701209).
In this PR I decided to try moving the description one level up, where
it renders properly.
# Testing
1. cd x-pack/solutions/security/plugins/security_solution
2. yarn openapi:bundle:detections
3. Take the bundled file
(docs/openapi/ess/security_solution_detections_api_2023_10_31.bundled.schema.yaml)
and load it into bump.sh console to see the changes.
4. Compare the changes with the [Legacy
documentation](https://www.elastic.co/guide/en/security/current/rule-api-overview.html)
You can also use this [link](https://bump.sh/jkelas/doc/kibana_wip/)
where I deployed the generated bundled doc.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
**Resolves: https://github.com/elastic/security-docs/issues/3545**
## Summary
I am fixing documentation for the version and revision fields.
I used this page as the base documentation:
https://docs.elastic.dev/security-solution/dev-docs/detections/rule-versions
The changes:
- by marking the RuleVersion as read-only I am making sure we are
properly documenting that this field is not to be used in the request.
It is only returned in the response.
- by introducing RuleRevision type and marking it as read-only, I am
doing the same for this field, saying that this field is not supposed to
be used in the request.
- I am not changing any code in the app, as the ticket says we shouldn't
do any breaking changes and the update of the version should not cause
400 error. Basically current behavior is kept: users can still update
the version to whatever value they want, including going backwards, and
the changes to revision field is completely ignored.
- I am adding a condensed description of these fields.
I wanted to introduce an internal link between these two fields, but I
couldn't make it work in Bump.sh (even though this should work, normal
Markdown links) so I abandoned this idea.
You can also use this [link](https://bump.sh/jkelas2/doc/kibana_wip2)
where I deployed the generated bundled doc.
Screenshots:
<img width="664" alt="image"
src="https://github.com/user-attachments/assets/34d82eb2-f7f0-4369-ad8e-2fd3c1f35447"
/>
<img width="660" alt="image"
src="https://github.com/user-attachments/assets/dc7772af-0185-4850-816e-60be003775d6"
/>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Fixes#210619
## Summary
Provides a tags array on the request and response bodies of dashboards.
This allows consumers of the Dashboards HTTP API and internal RPC API to
add an array of tag names to the attributes in the body of create and
update endpoints. The dashboard server will be responsible for
converting the tag names into references in the saved object.
If, during creation or update, a tag name does not have a matching tag
saved object, a new tag saved object will be created. If the user lacks
permissions to manage tags, then an error will be logged in the server
and the tag will not be added to the dashboard.
The server also injects the tag references as an array of tag names in
the attributes of the response body of get and search endpoints of the
HTTP and RPC APIs.
For backwards compatibility in create and update endpoints, tags can
alternatively be specified in the `references` array in the options
instead of (or in addition to) the `attributes.tags` in the request
body. Similarly, for backwards compatibility, tag references are
returned in the `references` of the response body of get and search
endpoints.
Client-side tag handling is out of scope for this PR. Dashboards listing
page and dashboard settings continue to use the tag references and do
not use the `tags` attribute in the response.
For example:
Here's how we currently create a dashboard with tag references.
```
## Creating a dashboard with tag references
POST kbn:/api/dashboards/dashboard
{
"attributes": {
"title": "my tagged dashboard"
},
"references": [
{
"type": "tag",
"id": "37aab5de-a34d-47cb-9aa5-9375d5db595f",
"name": "tag-ref-37aab5de-a34d-47cb-9aa5-9375d5db595f"
},
{
"type": "tag",
"id": "5ed29bba-c14d-4302-9a8c-9602e40dbc2a",
"name": "tag-ref-5ed29bba-c14d-4302-9a8c-9602e40dbc2a"
},
{
"type": "tag",
"id": "fc7890e8-c00f-44a1-88a2-250e4d27e61d",
"name": "tag-ref-fc7890e8-c00f-44a1-88a2-250e4d27e61d"
}
]
}
```
With this PR, creating a dashboard with tags is much simpler.
```
## Creating a dashboard with tag names
POST kbn:/api/dashboards/dashboard
{
"attributes": {
"title": "my tagged dashboard",
"tags": [
"boo",
"foo",
"bingo",
"bongo"
]
}
}
```
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
- [ ] If there are more than one tag saved objects with the same name,
only one of the tag references will be added to the saved object when
creating a dashboard. Creating tags with duplicate names are not
permitted via the UI. But there is no such restrictions when creating
tags from imported saved objects. Having multiple tags with the same
name is an edge case that Kibana guards against with reasonable
restrictions, so I think we should not be too concerned about it.
- [ ] ...
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## 📓 Summary
Part of https://github.com/elastic/streams-program/issues/38
This work adds the `date` processor along with the dissect and grok ones
in the enrichment section.
It scales well following the current folder structure, but we should
definitely polish it a bit more once more processors get added, such as
getting the right form component, improve form state derivation, etc.
https://github.com/user-attachments/assets/824d15c8-ce9d-455a-ae0b-97aeec8cf025
---------
Co-authored-by: Kerry Gallagher <kerry.gallagher@elastic.co>
Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Kerry Gallagher <471693+Kerry350@users.noreply.github.com>
# Purpose
This change introduces new validations that ensure no loss of data is
possible if a user accidentally sets the Security Entity Store enrich
policy execution interval to a value that “doesn’t play nicely” with the
lookback period value.
The specific logic (greater than or equal to half the value) was chosen
to not only ensure no loss of data, but also provide extra resiliency in
case of a failed enrich policy execution.
(Note that this is not considered a breaking change, as the parameters
are not yet available on any version of Elastic, including Serverless.)
# How to test
1. Load appropriate entity log data to your Kibana instance (for
example, using the
[security-documents-generator](https://github.com/elastic/security-documents-generator))
2. Navigate to the Developer console
3. Attempt to enable the Entity Store via the /enable or /init routes
(examples below), and pass in values that are expected to error. For
example, “lookbackPeriod”: “24h” and “enrichPolicyExecutionInterval”:
“24h” should fail, because of the validation logic
4. Expect results similar to those shown below, specifically a 400
error, or else a success message
<img width="1902" alt="Screenshot 2025-02-27 at 12 57 45 AM"
src="https://github.com/user-attachments/assets/a7f4b0fb-9899-4e00-a0ae-d172245bd506"
/>
<img width="1909" alt="Screenshot 2025-02-27 at 12 58 06 AM"
src="https://github.com/user-attachments/assets/372acde2-9d7b-4c75-8596-af8374088f79"
/>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Creates basic routes to download and upload content packs associated to
a stream.
Only dashboard assets will be exported and linked to a stream.
The endpoints are currently a proxy to the savedObjects
importer/exporter interfaces:
- download exports the dashboard linked to a stream
- upload imports a content pack file and link the dashboards to the
targeted stream. Dashboards are imported as-is with no index pattern
replacement performed, this will be implemented separately
### Testing
- download `curl -XPOST -H "x-elastic-internal-origin: 'kibana'" -H
"kbn-xsrf: true"
http://elastic:changeme@localhost:5601/pat/api/streams/logs/content/export
--output content.json`
- upload `curl -XPOST -H "kbn-xsrf: true"
http://elastic:changeme@localhost:5601/pat/api/streams/logs.foo/content/import
-F 'content=@content.json'`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
**Partially addresses:** #211808,
https://github.com/elastic/security-docs/issues/5981 (internal)
**Resolves: #208329**
## Summary
This is the second part of the migration effort, containing changes for:
- BULK CRUD (removing, for v.9.0)
The PR also contains changes for ticket #208329 - as changes for
removing of dead code for handling Bulk CRUD endpoints had to be
combined together with removing the schema files for Bulk CRUD
endpoints.
This PR will be backported only to versions for Kibana v9
# Testing
1. cd x-pack/solutions/security/plugins/security_solution
2. yarn openapi:bundle:detections
3. Take the bundled file
(docs/openapi/ess/security_solution_detections_api_2023_10_31.bundled.schema.yaml)
and load it into bump.sh console to see the changes.
4. Compare the changes with the [Legacy
documentation](https://www.elastic.co/guide/en/security/current/rule-api-overview.html)
You can also use this [link](https://bump.sh/jkelas2/doc/kibana_wip2/)
where I deployed the generated bundled doc.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Resolves first part of https://github.com/elastic/kibana/issues/198783
(snooze API)
Creates a public API for adding snooze schedule to rules.
For this purpose we created new schedule schema which will be used as
standard schedule schema across rules and alerting framework.
**Note**
The code to be reviewed for public API is under `common/routes/schedule`
and inside `external` folders.
Rest of the code is just moving existing internal API route and its code
to `/internal` folders.
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### How to test
- Create a rule in kibana
- Snooze that rule via new public API
```typescript
Path: https://localhost:5601/api/alerting/rule/<rule_id>/snooze_schedule
Method: POST
Body:
{
"schedule": {
"custom": {
"start": "2025-02-25T18:00:00.000Z",
"duration": "15m",
"recurring": {
"every": "1w",
"onWeekDay": ["TU", "FR"],
"occurrences": 10
}
}
}
}
```
- Verify various snooze schedule scenarios are generated correctly
#### Flaky test runner:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8009
### Release note
Allow users to create a snooze schedule for rule via API
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: lcawl <lcawley@elastic.co>
Add streams API to documentation as an experimental feature
<img width="2555" alt="Screenshot 2025-03-07 at 11 44 54"
src="https://github.com/user-attachments/assets/f54e5e6e-0c20-4bad-9cff-27747d0f76e2"
/>
There are a couple of changes in here:
* Split streams API in internal and public and mark the public parts as
experimental
* Add the public parts to the Kibana documentation
* Add description and summary
* Adjust the server repository wrapper to pass through summary and
description
# To test
* Generate OAS bundle: `node scripts/capture_oas_snapshot --include-path
/api/streams --update`
* Apply overlays `cd oas_docs && make api-docs`
* Make sure bump.sh is installed (`npm install -g bump-cli`)
* Run for preview: `cd oas_docs && bump preview output/kibana.yaml`
# Open questions
* Does the split into public and internal make sense?
* Is it a problem if this is visible in the user-facing documentation
page before we actually release streams? Or would it be OK if the API is
marked as experimental? (mostly a question for @LucaWintergerst )
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
[META
issue](https://github.com/elastic/security-docs-internal/issues/57)
This PR improves the API documentation for timeline/notes/pinned events
as per the definition in the meta issue.
### Notes
#### Timeline API
- `eventType`:
- is always set to `all`
- looks like it's been superseded by dataView?
- I marked it as `deperecated`
- `eventIdToNoteIds`:
- there's a type mismatch between what the frontend expects and what is
returned
- also it does not seem to be used anymore?
- it needs to be investigated further
#### Notes API
- `GlobalNote`:
- looks like it's not used anymore
- I removed it, nothing broke
- `eventIngested`, `eventTimestamp`, `eventDataView` and `overrideOwner`
are all not used on the patch note endpoint, I removed them. The
`event*` ones I have never seen before to be honest.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
