Depends on https://github.com/elastic/kibana/pull/221610
This PR adds the import index workflow to privileged user monitoring and
API changes required to support it.
### API Enhancements
* **New API for privilege monitoring index creation**: Added a new API
endpoint (`PUT /api/entity_analytics/monitoring/privileges/indices`) to
create indices for privilege monitoring with support for `standard` and
`lookup` modes. This includes the implementation of request and response
schemas (`create_indidex.gen.ts`, `create_indidex.schema.yaml`).
[[1]](diffhunk://#diff-68329bb90dea945f343e1637990d5d05bc159e0aa2511ef1e45d37ed1a6cda51R1-R41)
[[2]](diffhunk://#diff-e979499654a27b3c1930d63c5b1002113c1c3f53f84ce27a4d75a5c492717a96R1-R42)
* **Updated privilege monitoring health response**: Modified the health
response schema to include a `status` field and an optional `error`
object for detailed error handling
(`privilege_monitoring/health.gen.ts`,
`privilege_monitoring/health.schema.yaml`).
[[1]](diffhunk://#diff-00f39a3e65a336eaddf7d3203d1370d910f5ecd2062b6cc21d9c06922c12884eR19-R28)
[[2]](diffhunk://#diff-83afa72b7a1fc48f3cc063e9fb855190d3525228bc0488fb8b871e112b90e961L22-R33)
### Frontend Integration
* **Introduce the create index modal that opens when the create index
button is clicked.
* **Onboarding modal improvements**: Updated the `AddDataSourcePanel`
component to handle index creation more robustly by passing callbacks to
the modal (`add_data_source.tsx`).
* **Error handling in UI**: Enhanced the `PrivilegedUserMonitoring`
component to display error callouts when privilege monitoring data fails
to load (`privileged_user_monitoring/index.tsx`).
[[1]](diffhunk://#diff-273ad32c97dcf15c6c6054fd7c5516d587132674578d25986b235cd174c75789R22-R26)
[[2]](diffhunk://#diff-273ad32c97dcf15c6c6054fd7c5516d587132674578d25986b235cd174c75789R38-R51)
### How to test it?
* Go to the priv mon page with an empty cluster
* Click on the data source by the index button
* Search for available indices, it should return indices with
`user.name.keyword` fields
* Click 'create index' and create a new index
* Choose the created index and click 'Add privileged users'
* You should be redirected to the dashboard (The API is currently not
working)
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
# Overview
This pull request adds capabilities associated with the [Privileged
Access Detection (PAD) anomaly integration
package](https://www.elastic.co/docs/reference/integrations/pad) as a
first-class-citizen into the Entity Analytics Privileged User Monitoring
feature.
# How to test
- Pull this branch into your local machine
- Ensure that the security experimental flag
`privilegeMonitoringEnabled` has been enabled, for example by setting
`xpack.securitySolution.enableExperimental:
[privilegeMonitoringEnabled]` in your `config/kibana.dev.yml` file
- Start Elasticsearch and Kibana
- From the
[security-documents-generator](https://github.com/elastic/security-documents-generator)
repository, run the following command: `yarn start
privileged-user-monitoring` (ensuring your config is pointing to your
locally running Elastic cluster). This will load "source" events that
are anomalous in nature. You can run this command more than once if
desired to upload more than 10 users.
- Grab the CSV file that the generator created in its console output.
- Open the Entity analytics page and navigate to dashboards (by clicking
"Go to dashboards")
- Add the privileged users from the previous step to the privileged
users index using the CSV File Upload option.
> [!NOTE]
> Any errors regarding risk scoring are unrelated to this PR, and are
being resolved separately
- You should see a panel that says "Enable Privileged access detection".
Click "Install", and you'll meet a loading state.
- Once complete, you shouldn't see any results. That's because, even
though we **install** the ML jobs by default, we don't **run** them by
default.
- Click "ML Job Settings", and note that only `pad` jobs should be
displaying in this callout. Feel free to test this callout's links and
filtering options.
- Click "Run job" next to the job called
`pad_linux_high_count_privileged_process_events_by_user`. This is the
job for which we have anomaly data.
- Click away from the callout. **You still shouldn't see data.** That's
because there just aren't any anomalies "today".
- Change the global date filter at the top of the screen to "Last 30
days".
- You should see something similar to this:
<img width="1441" alt="Screenshot 2025-06-16 at 12 50 25 AM"
src="https://github.com/user-attachments/assets/2b3f11f2-f45d-4716-bb8e-79d2b585aa3e"
/>
- Congrats for making it this far! Some things to play around with:
- Click around on the anomaly filters (i.e., click 25-50), and notice
that the results will change.
- Ensure that the ordering of the users is based on the **highest single
anomaly score in any visible bucket**. Meaning, if user `samwise` has
only one anomaly, but its score is 99, and user `frodo` has dozens of
anomalies, but no higher than, say, 80, `samwise` will be on the top of
the list.
- Click the user names to open the appropriate user flyout
- Change the global time filter to change ranges, and ensure the data
shows up appropriately. The buckets will try to roughly show 30 buckets
total, but will have a maximum of 3 hours, meaning if your range is too
small, it may show fewer than 30 buckets. This is intentional behavior,
as the PAD jobs have an anomaly job window of 3 hours.
- Click the button that says "View all in Anomaly Explorer", which uses
the currently selected global time range, and compare results
> [!WARNING]
> Remember that the users in the privileged user monitoring table are
only those that you designated as privileged users in a previous step.
In contrast, the Anomaly Explorer page shows **all** users.
Additionally, note that there may be very slight differences between the
swimlanes, because ES|QL calculates the bucket dates slightly
differently than the Anomaly Explorer. **This should not affect the
results themselves, only the buckets that an individual anomaly score
might fall in.** You might see a single anomaly fall into one visual
bucket instead of another, but the date should be correct.
# Helpful hints
If you'd like an easy way to "reset" the Privileged Access Detection
package and delete its associated jobs (and anomaly data), so that you
can redo the onboarding flow, you can run the below commands (changing
any credentials as necessary):
```shell
curl "http://localhost:5601/api/fleet/epm/packages/pad/0.5.0" \
-X 'DELETE' \
-H 'elastic-api-version: 2023-10-31' \
-H 'kbn-xsrf:true' \
--user elastic:changeme
curl "http://localhost:5601/internal/ml/jobs/delete_jobs" \
-H 'kbn-xsrf:true' \
--user elastic:changeme \
-X 'POST' \
-H 'elastic-api-version: 1' \
-H 'x-elastic-internal-origin:kibana' \
-H 'Content-Type: application/json' \
--data-raw '{
"jobIds": [
"pad_linux_high_count_privileged_process_events_by_user",
"pad_linux_high_median_process_command_line_entropy_by_user",
"pad_linux_rare_process_executed_by_user",
"pad_okta_high_sum_concurrent_sessions_by_user",
"pad_okta_rare_host_name_by_user",
"pad_okta_rare_region_name_by_user",
"pad_okta_rare_source_ip_by_user",
"pad_okta_spike_in_group_application_assignment_changes",
"pad_okta_spike_in_group_lifecycle_changes",
"pad_okta_spike_in_group_membership_changes",
"pad_okta_spike_in_group_privilege_changes",
"pad_okta_spike_in_user_lifecycle_management_changes",
"pad_windows_high_count_group_management_events",
"pad_windows_high_count_special_logon_events",
"pad_windows_high_count_special_privilege_use_events",
"pad_windows_high_count_user_account_management_events",
"pad_windows_rare_device_by_user",
"pad_windows_rare_group_name_by_user",
"pad_windows_rare_privilege_assigned_to_user",
"pad_windows_rare_region_name_by_user",
"pad_windows_rare_source_ip_by_user"
],
"deleteUserAnnotations": true,
"deleteAlertingRules": false
}
'
```
# What's left?
- This PR does not implement "proactive" permissions checks. Instead,
relevant permission issues will cause requests/queries to fail, which
are appropriately shown in the UI for troubleshooting. This proactive
check will be tackled in [this (private) followup
issue](https://github.com/elastic/security-team/issues/12822).
- There is not yet a way to "upgrade" the package directly from the
Privileged User Monitoring screen. This would be a nice addition later
on, instead of requiring users to navigate to the integration page. This
behavior will be tackled in [this (private) followup
issue](https://github.com/elastic/security-team/issues/12823).
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This PR removes the current GET user route as it works with IDs and we
do not use IDs in privmon.
We'd rather remove this before it gets documented and pushed out. We can
revert this PR later if needed
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Mark Hopkin <mark.hopkin@elastic.co>
## Summary
We have [this RFC](https://github.com/elastic/ecs/pull/2493) in, I think
this is a safer bet and might save us a migration in the future:
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Original issue: https://github.com/elastic/security-team/issues/10688
This PR adds a new function to the rules client called
`bulkFillGapsByRuleIds`, that takes a list of rule ids and a date range,
and attempts to fill out the gaps simultaneously, 10 rules at a time
(can be customizable).
For all rules we verify if the user has access to write to the gaps
object, then for each rule we fetch gaps and trigger the backfilling of
its unfilled intervals.
During the execution, we aggregate the errors and the rules that are
skipped in lists and return them along with the scheduling outcomes.
### Error handling
There are 2 error types that are handled throughout the scheduling of a
gaps backfilling for a rule:
- A user doesn't have access the rule
- We fail to backfill gaps
### Skipped rules
A rule can be skipped when it doesn't have unfilled gaps in the given
time range, or when gaps are found, but they are currently "in
progress".
### Endpoint
This PR also updates the existing bulk actions endpoint by adding the
action `fill_gaps`. The endpoint for now should limit the amount of
rules per call to 100.
The bulk actions endpoint returns a summary of the execution which is a
list of counters of rules that succeeded, failed or were skipped.
## How to test?
1. Create several rules that run every 5 seconds and disable them. Leave
them disabled for a couple of minutes and then re-enable them in order
to create some gaps for it. You can confirm that there are gaps by
clicking on the rule, then on the "Execution results" tab, and then
scrolling down to "Gaps".
Alternatively you can use [this
tool](https://github.com/elastic/security-documents-generator) to create
1 rule (5m) with 1000 gaps:
```
yarn start rules --rules 1 -g 1000 -c -i"5m"
```
2. Get the ids of each rule that you created and call the endpoint to
backfill them. You can select a time range that is smaller than the gaps
so that you can call it several times. Here is an example of a curl to
do this
```
curl -X POST "http://localhost:5601/api/detection_engine/rules/_bulk_action?dry_run=false" \
-H "Accept: */*" \
-H "Content-Type: application/json" \
-H "kbn-version: 9.1.0" \
-H "kbn-build-number: 9007199254740991" \
-H "elastic-api-version: 2023-10-31" \
-H "Authorization: Basic $(echo -n 'elastic:changeme' | base64)" \
--data-raw '{
"action": "fill_gaps",
"ids": ["307bdea8-28be-419f-bb25-dac0024f32af"],
"fill_gaps": {
"start_date": "2025-05-09T08:12:09.457Z",
"end_date": "2025-05-09T09:12:09.457Z"
}
}'
```
3. Then you can go to the "Gaps" tab from step 1 and verify that the
gaps are being filled. Additionally, you can see that there are manual
runs scheduled for the rule in order to fill the gaps.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR adds the ability to upload a CSV file with privileged users from
the Entity Analytics pages
## Changes
### Backend
- Added (or completed) the upload CSV route:
`/api/entity_analytics/monitoring/users/_csv`
- Added shared utilities for batching with Node streams
- Added bulk processing actions for the upload
- Parsing users from CSV
- Soft delete for omitted users
- Batch upsert via the bulk API
- Added a check for installing all required privmon resources
### Frontend
- File uploader components
- File validation logic
- Updated EA privmon page to account for the new flow
- Added managing users panels
- open upload flow (same as asset criticality)
## Screen recording
https://github.com/user-attachments/assets/7956f1cf-49e0-4430-8c23-7d6178a15342
## How to test
#### Prerequisite
Make sure you have a CSV file with usernames
Check
[here](https://gist.github.com/tiansivive/0be2f09e1bb380fdde6609a131e929ed)
for a little helper script
Create a few copies where some of the users are deleted, in order to
test soft delete
1. Start up kibana and ES
2. Navigate to Security > Entity Analytics > Privilege User Monitoring
3. Select the `File` option to add data
4. Add one of the CSV files to the open modal and upload
5. Repeat but now upload one of files with the omitted users
Alternatively, testing only the backend only is possible by directly
hitting the API wit curl
```
curl -u elastic:changeme \
-X POST "http://localhost:5601/api/entity_analytics/monitoring/users/_csv" \
-H "kbn-xsrf: true" \
-F "file=@test.csv;type=text/csv"
```
#### Verifying
Easiest way is to use the dev tools to `_search` the privmon users index
with:
```
GET .entity_analytics.monitoring.users-default/_search
```
Look for number of hits and/or use `query` to search for omitted users.
## Remaining work
- [x] API integration tests
- [ ] Batching logic unit tests
- [ ] E2E tests?
---------
Co-authored-by: machadoum <pablo.nevesmachado@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Changes the dashboard endpoint routes from public to internal as they
are still unreleased.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes https://github.com/elastic/kibana/issues/220850
## Summary
Add logic that handles the case of custom integrations that cannot be
installed in the remote because of error
` PackageNotFoundError: [pkgName] package not found in registry`
Currently, when the installation is attempted, we just log the error and
exit. I added some logic that creates an "empty" installation SO in
`epm-packages` with status `install_failed` and populates
the`latest_install_failed_attempts` field so we know what happened.
**NOTE**: we could add this logic in the regular install process as
well, it would be useful for debugging purpose
- I also added some logic to `compareSyncIntegrations` to make this
error a warning and because I also made the logic for the warning more
generic
### Testing
- Install a custom integration in the main cluster, let the
syncIntegrationsTask run and after a while check `GET
.kibana_ingest/_doc/epm-packages:PKGNAME`. You should see the error
saved under `latest_install_failed_attempts`:
<img width="1804" alt="Screenshot 2025-06-10 at 16 20 09"
src="https://github.com/user-attachments/assets/2cffbf01-15f2-4091-bcbd-660fbb390c56"
/>
- In the remote, query `GET
kbn:/api/fleet/remote_synced_integrations/status` and verify that it
returns a warning for the custom integration. Example:
```
"integrations": [
{
"package_name": "agentless_package_links",
"package_version": "0.0.2",
"install_status": {
"main": "installed",
"remote": "not_installed"
},
"updated_at": "2025-05-21T08:55:47.981Z",
"sync_status": "warning",
"warning": {
"title": "agentless_package_links can't be automatically synced",
"message": "This integration must be manually installed on the remote cluster. Automatic updates and remote installs are not supported."
}
},
...
```
- Test the UI - go to Fleet settings and verify that the custom
integration shows a warning:
![Uploading Screenshot 2025-06-11 at 16.26.47.png…]()
### Checklist
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Enable the new tabular experience for managing installed integrations,
including support for bulk integration updates.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Bulk operations support `gap_range`
[issue](https://github.com/elastic/kibana/issues/220772)
### Bug
If a user has 100 rules in total and 20 of them have gaps:
- They click "Show only with gaps" — the UI shows just those 20.
- They hit "Select all" — it says 20 rules are selected.
- But when they perform a bulk action, it ends up applying to **all 100
rules**, not just the 20 with gaps.
This happens because the bulk action uses a query that doesn't include
gap info, so it ends up targeting everything.
### Fix
We now support passing `gaps_range_start` and `gaps_range_end` along
with the bulk action.
On the API side:
- If a query is sent, we use the gap range to find the actual list of
rule IDs with gaps.
- Then we apply the bulk action only to those rules.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Adds a new processor to streams that serves as an escape hatch to
regular Elasticsearch ingest pipelines (entered as JSON).
<img width="1139" alt="Screenshot 2025-05-28 at 15 54 41"
src="https://github.com/user-attachments/assets/67f1f4c4-982e-45d1-ae96-080545c5a0e2"
/>
Some details:
* If `on_failure` or `ignore_failure` are set via the JSON input, the
"outer" definition or ignore_failure flag are ignored
* Expands to multiple processors in the ingest pipeline
* Does minimal validation (needs to be one of a list of known existing
Elasticsearch processors based on the Elasticsearch API types), but
doesn't enforce valid Elasticsearch processors otherwise
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/217490
- Enable feature flag `enableSyncIntegrationsOnRemote`
- Added check to hide sync integrations feature in serverless
- Moved creating the follower index from Fleet setup to output
create/update API and async task (create if does not exist)
- Follower index is not hidden for now, because if hidden, it's not
showing up on the CCR UI
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes#210326
## Summary
The public Maintenance Window API specification was missing from
`oas_docs/output/kibana.yaml`.
The `capture_oas_snapshot.sh` script had an outdated path.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: lcawl <lcawley@elastic.co>
## Summary
* Addresses this comment
https://github.com/elastic/kibana/issues/221056#issuecomment-2917522252,
essentially avoid stating `added in...` if we are generating OAS for a
serverless Kibana
* Refactors `processRouter` and `processVersionedRouter` to take in
object (for easier refactoring)
* Adds test coverage
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes https://github.com/elastic/kibana/issues/1547
Closes https://github.com/elastic/kibana/issues/190342
Closes https://github.com/elastic/kibana/issues/197716
## Summary
This PR adds the ability for collapsible sections to be created and
managed on Dashboards.
https://github.com/user-attachments/assets/c5c046d0-58f1-45e1-88b3-33421f3ec002
> [!NOTE]
> Most of the work for developing collapsible sections occurred in PRs
contained to the `kbn-grid-layout` package (see [this meta
issue](https://github.com/elastic/kibana/issues/190342) to track this
work) - this PR simply makes them available on Dashboards by adding them
as a widget that can be added through the "Add panel" menu. As a result
of this, most work is contained in the Dashboard plugin - changes made
to the `kbn-grid-layout` package only include adding IDs for additional
tests that were added for the Dashboard integration.
### Technical Details
#### Content Management Schema
The content management schema allows for panels and sections to be mixed
within the single `panels` key for a dashboard **without** worrying
about section IDs; for example:
```
{
"panels": [
{
// this is a simplified panel
"gridData": {
"x": 0,
"y": 0,
"w": 12,
"h": 8,
},
"panelConfig": { ... },
},
{
// this is a section
"gridData": {
"y": 9,
},
"collapsed": false,
"title": "Section title",
"panels": [
{
// this is a simplified panel
"gridData": {
"x": 0,
"y": 0,
"w": 24,
"h": 16,
},
"panelConfig": { ... },
},
],
},
]
}
```
#### Saved Object Schema
The dashboard saved object schema, on the other hand, separates out
sections and panels under different keys - this is because, while we are
stuck with panels being stored as `panelJSON`, I didn't want to add much
to this. So, under grid data for each panel, they have an optional
`sectionId` which then links to a section in the `sections` array in the
saved object:
```
{
"panelsJSON": "<...> \"gridData\":{\"i\":\"panelId\",\"y\":0,\"x\":0,\"w\":12,\"h\":8,\"sectionId\":\"someSectionId\"} <...>"
"sections": [
{
"collapsed": false,
"title": "Section title",
"gridData": {
"i": "someSectionId",
"y": 8.
}
}
],
}
```
This allows sections to be serialized **without** being stringified.
This storage also matches how we store this data in runtime using
`layout`.
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Release note
Adds collapsible sections to Dashboard, which allow panels to grouped
into sections that will not load their contents when their assigned
section is collapsed.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Fixes https://github.com/elastic/kibana/issues/209789
## Summary
For `inputs` type integrations
([docs](https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/fleet/dev_docs/input_packages.md)),
we install all the assets when creating the integration policy and not
at integration install time (like for "regular": integrations).
However the clean up of assets doesn't happen when removing the
integration policy and this leaves around orphaned assets that are not
working anymore.
- This PR adds a new endpoint that removes the datastream assets
```
DELETE kbn:/api/fleet/epm/packages/{pkgName}/{pkgVersion}/datastream_assets?packagePolicyId={Id}
```
- The new endpoint is called by the UI when removing the integration
policy;
- Only the datastream assets that match exactly the dataset name are
removed; assets that are common across the integration are kept and can
only be deleted when the whole integration is uninstalled.
Additional changes:
- I did some light refactoring of the functions for inputs-type
integrations to make the code more readable
- Updated the dev_docs about input-type integrations that haven't been
touched for long time
### Testing
- Install an input-type package, for instance "custom logs"
- Check the assets created under the tab `assets`
- Check that the package has only this integration policy
- Remove the integration policy for the package - a warning is shown:
<img width="937" alt="Screenshot 2025-05-09 at 16 58 51"
src="https://github.com/user-attachments/assets/0f86ab38-e0a9-47f5-91f5-71b83e17f2e3"
/>
- Verify that the assets related to the package are cleaned up as well
- Try again but with several integration policies
- In this case the clean up doesn't happen
### Delete assets when there are two integration policies with different
dataset names
Dataset names are`udp.generic` and `udp.test` - in the video I deleted
policy `udp-2` having dataset name `udp.test` and the relative assets
are no longer present:
https://github.com/user-attachments/assets/23350051-1b26-4e52-914d-62f784809c80
### Delete assets when there are two integration policies with same
dataset names
In this case there are two different policies having the same dataset
name `udp.generic`, when deleting the policy there is no warning to
remove the assets. In fact they can be deleted only when there is only
one remaining policy using them:
https://github.com/user-attachments/assets/f75668dd-a4ce-4f5a-ba5d-c99911278dfc
### Checklist
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: jillguyonnet <jill.guyonnet@gmail.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes https://github.com/elastic/kibana/issues/219488
## Summary
Add license gate around remote synced integrations - Only accounts
having Enterprise license will be able to use the new remote synced
integrations feature. This requirement depends on the ccr feature, which
works only on Enterprise licenses, this PR makes sure that this also
checked in Fleet.
### Testing
1. With a license different than enterprise
- Try to create an output that enables synced integrations - it should
fail with a 400:
```
POST kbn:/api/fleet/outputs
{
"name": "new 1",
"type": "remote_elasticsearch",
"hosts": [ "https://lskfgjojg" ],
"is_default": false,
"is_default_monitoring": false,
"sync_integrations": true
}
```
- Verify that the UI elements listed in
https://github.com/elastic/kibana/issues/219488 are not visible
2. With Enterprise license
- Check that the UI elements are visible and the remote sync
integrations features work as expected
### Checklist
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes#217617
Adds a new endpoint `POST kbn:/api/fleet/agents/{agentId}/migrate`
allowing a user to migrate a single agent to another cluster.
Required parameters are: `enrollment_token` and `uri`
- Adds `MIGRATE` as an action type and is reflected in UI
- Includes unit tests as well as integration tests
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
N/A
# Release Note
- Added endpoint allowing a user to migrate an individual agent to
another cluster by specifying the URL and Enrollment Token. Note: tamper
protected and fleet agents can not be migrated and attempting to do so
will return a `403` status code.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Closes https://github.com/elastic/kibana/issues/217683
## Summary
Follow up of https://github.com/elastic/kibana/pull/217144
Handle errors occurring when `sync_uninstalled_integrations` is enabled
on remote outputs; these errors are now saved under
`latest_uninstall_failed_attempts` and will be reported by
`api/fleet/remote_synced_integrations/<output_id>/remote_status` and
`api/fleet/remote_synced_integrations/remote_status`.
- I added a new field in the response of these apis that allows to
understand at a glance the install status of an integration on both
cluster:
```
install_status: {
main: 'installed',
remote: 'not_installed',
}
```
- Added a new "warning" state for synced integrations
- Handled the case when an integration was successfully uninstalled from
both clusters (marked as complete)
- Removed the "throw error" for the case of `outputId` in favour of a
regular error in the response
### Testing
- Follow steps in https://github.com/elastic/kibana/pull/217144
- Check that the errors reported in `latest_uninstall_failed_attempts`
are now visible when querying
`api/fleet/remote_synced_integrations/<output_id>/remote_status` under
the new "warning" field. This can be done also from the UI, checking the
network tab
-
<img width="2111" alt="Screenshot 2025-05-20 at 11 17 13"
src="https://github.com/user-attachments/assets/80a077e7-8b1b-4d04-abe9-0ef0cc44def8"
/>
Response for the failed uninstalled integration:
```
{
"package_name": "akamai",
"package_version": "2.28.0",
"install_status": {
"main": "not_installed",
"remote": "installed"
},
"updated_at": "2025-05-21T09:34:34.492Z",
"sync_status": "warning",
"warning": "Unable to remove package akamai:2.28.0 with existing package policy(s) in use by agent(s) at Fri, 23 May 2025 07:54:41 GMT"
},
```
### UI changes
The integrations uninstalled from the main cluster are now shown with a
greyed out text and the warning is shown on screen as well:
<img width="703" alt="Screenshot 2025-05-23 at 10 37 57"
src="https://github.com/user-attachments/assets/a6900e0b-96cc-4bcc-8f16-db0001f55de3"
/>
### Checklist
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Updated the behaviour of `setXState` per
https://github.com/elastic/kibana/issues/221056.
Partially addresses that issue. In future we need to migrate existing
routes to add that information, this covers the Core provided
capability.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Resolves: https://github.com/elastic/kibana/issues/185932
- Adds a `name` field to the `_meta` object of
`InstallPackageResponseSchema`, which represents the package name.
This addition will appear in the responses for:
- Install from registry: POST
`/api/fleet/epm/packages/<pkgName>/<version?>`
- Install by upload: POST `/api/fleet/epm/packages`
- Create custom integration: POST `/api/fleet/epm/custom_integrations`
## Summary
This PR introduces basic CRUD routes for handling Privileged Users in
Entity Analytics.
The following routes are available:
* CREATE: `POST /api/entity_analytics/monitoring/users`
* GET: `GET /api/entity_analytics/monitoring/users/{id}`
* LIST: `GET /api/entity_analytics/monitoring/users/list`
* UPDATE: `PUT /api/entity_analytics/monitoring/users/{id}`
* DELETE: `DELETE /api/entity_analytics/monitoring/users/{id}`
For CREATE and UPDATE, the request body should be of type:
```
{ "user_name": string, is_monitored: boolean }
```
The reason for snake_case is to align better with the upcoming csv and
json upload work.
This PR already introduces boilerplate code (registering the endpoints
and handlers) for those routes). We might want to change this.
## How to test
1. Start a fresh Kibana instance
2. Enable the `EntityAnalyticsPrivilegeMonitoring` Feature Flag.
3. Initialise the privmon engine with `POST
kbn:/api/entity_analytics/monitoring/engine/init`
4. Test any of the endpoints above
* Make sure to either note down the returned `id`s or simply query the
underlying index: `.entity_analytics.monitoring.users-<your namespace>`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Related to #213024.
This adds the necessary backend schema changes for the investigation
guide.
Note this PR depends on #216292 and should not be merged before it.
---------
Co-authored-by: Panagiota Mitsopoulou <giota85@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Panagiota Mitsopoulou <panagiota.mitsopoulou@elastic.co>
Co-authored-by: Christos Nasikas <xristosnasikas@gmail.com>
Refactors Streams models to use namespaces. `Streams` contains both the
TypeScript interfaces and the validation schemas. In general, there are
four variations of each Stream type:
- `Definition` - the definition that we largely work with in the Streams
client and in the browser.
- `Source` - how the definition is stored in ES. This is a pass-through
at the minute, but separating this from the definition allows us to do
migrations on read and write.
- `GetResponse` - what is returned from `GET /api/streams/{name}`. This
includes the definition, but also other assets that are not part of the
definition, but rather a function of the definition, like Elasticsearch
index templates etc, and attached Kibana assets like Dashboards and
Queries.
- `UpsertRequest` - what is required in `PUT /api/streams/{name}`. This
includes the stream definition, but, importantly, without a name - the
name is not allowed in the request endpoint. Similarly, this allows the
user to update assets like Dashboards and Queries.
Schemas are defined as Validations, which are left/right schemas - that
is, the type on the right is always a refinement of the type on the
left. This allows for more safe transitions than going from any object
to the specific type. Additionally, it includes helpers such as `is`,
`as`, `parse` and `asserts`, which help with both runtime validation and
type refinement.
## Additional changes
- Added special formatting for `AggregateError` in MessageConversion
- Simplified DeepStrict
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
GH issue https://github.com/elastic/security-team/issues/12387
The changes in this PR are based on a
[PR](https://github.com/elastic/kibana/pull/210238) created by
@marshallmain. Initially we wanted to show a warning for users when
frozen is queried, but we have decided not to do that and instead report
the count of frozen indices queried to the execution log, to later be
retrieved as a metric through internal endpoints like the rule execution
results
[endpoint](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/rule_execution_logs/get_rule_execution_results/get_rule_execution_results_route.ts).
When an index is moved to the frozen tier, its name is prefixed with
`partial-` and its `_tier_preference` is set to `data_frozen`. Therefore
we can retrieve all the matching indices using `field_caps`, filter
those that start with the prefix and then check their data tier from the
index settings to make sure it is frozen, since anybody can just create
an index which name is prefixed with `partial-`.
## How was this tested?
- The frozen tier was enabled locally by running my instance of Elastic
with these additional parameters: `-E path.repo="/tmp" -E
xpack.searchable.snapshot.shared_cache.size=20GB`.
- A snapshot repository and an ILM policy were created. The ILM policy
would move an index to frozen after 1 minute using [this
tutorial](https://docs.elastic.dev/security-soution/analyst-experience-team/eng-prod/how-to/configure-local-frozen-tier).
- An index was manually created and populated with a couple of
documents. A rule querying this index was created, and the execution
results endpoint was called to verify that the counter for frozen
indices was zero.
- The ILM policy was assigned to the index, and after waiting for it to
be promoted to frozen, the execution results endpoint was called again
to confirm that the counter was 1 and that the correct index appeared in
the warning message of the execution.
- Another index was manually created, and the same process was repeated
to confirm that the counter was accurate before and after assigning the
ILM policy.
- A similar test was conducted using a data stream to ensure consistent
behavior.
Here is an example of the call to the executions result endpoint 👇 :
```
curl -X GET --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 1" 'c4ddc82a-b964-46c1-8c1a-bc11c9fe1d66/execution/results?start=2025-04-15T12:50:02.000Z&end=2025-04-18T12:51:23.000Z'
```
## In this PR
- Report the count of frozen indices queried
- Expose the counter through the rule execution results endpoint
## Upcoming
- Expose a counter indicating how many frozen indices have been queried
in the summary endpoint (Follow up PR)
- Make sure that this data is exposed on Kibana diagnostics.
- Find out where to document this feature (internally and externally).
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
# Backfill API: Support Multiple Ranges for Scheduling
## Context
In the future, we want to introduce a bulk API for filling gaps. We also
want to avoid creating a separate backfill task for each gap.
To achieve this, this PR updates the scheduling API to support multiple
date ranges.
## Changes
- **No changes** to the mapping of the backfill SO — this only affects
how the API is called.
- Updated related frontend calls to align with the new API structure.
- **Breaking change:**
- The API no longer supports `start` and `end` dates as separate fields.
- Instead, it **requires** `ranges`, where each range must have a
`start` and an `end` date.
- **This is fine because the API is internal and does not affect
external users.**
## Example Usage
```json
fetch("http://localhost:5601/internal/alerting/rules/backfill/_schedule", {
"headers": {
// ..
},
"body": JSON.stringify([
{
"rule_id": "0c0b9539-4a55-4255-a1aa-11651b2af119",
"ranges": [
{
"start": "2025-04-01T10:28:51.204Z",
"end": "2025-04-01T13:28:51.204Z"
}
]
}
]),
"method": "POST"
});
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This PR adds a new feature for streams to control whether the UI and API
is available.
Changes:
* Add the feature with two privileges (`show` and `manage_assets`)
* Can be configured with the classic `none`/`read`/`all` so it's
automatically aligned with serverless editor/viewer permissions
* None also means the app is not shown - to do this, the existing
`status$` observable also looks for at least the `streams.show`
capability
* Only guards changes to the linked dashboards - changes to the
Elasticsearch level are still delegated to the Elasticsearch-level
permissions of the user
* This happens on the UI level (disabled button and dashboard selection
on the dashboard page)
* and on the API level (all endpoints that can change linked dashboards
require the permission)
# Questions
* Not sure about the name `manage_assets` - maybe it should be something
else
* Not sure about how the queries stuff should work - @kdelemme anything
we should do in this area?
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/217154
Improvements to sync integrations status API and error reporting
- Saving custom asset update errors to the package SO in
`latest_custom_asset_install_failed_attempts` field
- Reading these errors in the status API and UI
- Fix sync status calculation: show `FAILED` if one of integrations or
custom assets are in failed state, `SYNCHRONIZING` if one of
integrations or custom assets are in synchronizing state, otherwise show
`COMPLETED` state.
<img width="608" alt="image"
src="https://github.com/user-attachments/assets/15a17690-443b-4ca1-b705-cc92ec7d3b20"
/>
- Reading the `followStats` API to report on fatal errors, found that
the `followInfo` API doesn't report if the connection to the remote
cluster fails. Reproduced this by updating an active Remote Cluster with
an invalid port. The `followInfo` API still reports `active` status.
<img width="612" alt="image"
src="https://github.com/user-attachments/assets/e95ebc62-4ed9-42c2-9954-93d9438b6ece"
/>
```
GET fleet-synced-integrations-ccr-main/_ccr/stats
{
"indices": [
{
"index": "fleet-synced-integrations-ccr-main",
"shards": [
{
"remote_cluster": "main",
"leader_index": "fleet-synced-integrations",
"follower_index": "fleet-synced-integrations-ccr-main",
...
"fatal_exception": {
"type": "exception",
"reason": "java.lang.IllegalArgumentException: port out of range:93001",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "port out of range:93001"
}
}
}
]
}
]
}
```
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
