Depends on https://github.com/elastic/kibana/pull/221610
This PR adds the import index workflow to privileged user monitoring and
API changes required to support it.
### API Enhancements
* **New API for privilege monitoring index creation**: Added a new API
endpoint (`PUT /api/entity_analytics/monitoring/privileges/indices`) to
create indices for privilege monitoring with support for `standard` and
`lookup` modes. This includes the implementation of request and response
schemas (`create_indidex.gen.ts`, `create_indidex.schema.yaml`).
[[1]](diffhunk://#diff-68329bb90dea945f343e1637990d5d05bc159e0aa2511ef1e45d37ed1a6cda51R1-R41)
[[2]](diffhunk://#diff-e979499654a27b3c1930d63c5b1002113c1c3f53f84ce27a4d75a5c492717a96R1-R42)
* **Updated privilege monitoring health response**: Modified the health
response schema to include a `status` field and an optional `error`
object for detailed error handling
(`privilege_monitoring/health.gen.ts`,
`privilege_monitoring/health.schema.yaml`).
[[1]](diffhunk://#diff-00f39a3e65a336eaddf7d3203d1370d910f5ecd2062b6cc21d9c06922c12884eR19-R28)
[[2]](diffhunk://#diff-83afa72b7a1fc48f3cc063e9fb855190d3525228bc0488fb8b871e112b90e961L22-R33)
### Frontend Integration
* **Introduce the create index modal that opens when the create index
button is clicked.
* **Onboarding modal improvements**: Updated the `AddDataSourcePanel`
component to handle index creation more robustly by passing callbacks to
the modal (`add_data_source.tsx`).
* **Error handling in UI**: Enhanced the `PrivilegedUserMonitoring`
component to display error callouts when privilege monitoring data fails
to load (`privileged_user_monitoring/index.tsx`).
[[1]](diffhunk://#diff-273ad32c97dcf15c6c6054fd7c5516d587132674578d25986b235cd174c75789R22-R26)
[[2]](diffhunk://#diff-273ad32c97dcf15c6c6054fd7c5516d587132674578d25986b235cd174c75789R38-R51)
### How to test it?
* Go to the priv mon page with an empty cluster
* Click on the data source by the index button
* Search for available indices, it should return indices with
`user.name.keyword` fields
* Click 'create index' and create a new index
* Choose the created index and click 'Add privileged users'
* You should be redirected to the dashboard (The API is currently not
working)
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
# Overview
This pull request adds capabilities associated with the [Privileged
Access Detection (PAD) anomaly integration
package](https://www.elastic.co/docs/reference/integrations/pad) as a
first-class-citizen into the Entity Analytics Privileged User Monitoring
feature.
# How to test
- Pull this branch into your local machine
- Ensure that the security experimental flag
`privilegeMonitoringEnabled` has been enabled, for example by setting
`xpack.securitySolution.enableExperimental:
[privilegeMonitoringEnabled]` in your `config/kibana.dev.yml` file
- Start Elasticsearch and Kibana
- From the
[security-documents-generator](https://github.com/elastic/security-documents-generator)
repository, run the following command: `yarn start
privileged-user-monitoring` (ensuring your config is pointing to your
locally running Elastic cluster). This will load "source" events that
are anomalous in nature. You can run this command more than once if
desired to upload more than 10 users.
- Grab the CSV file that the generator created in its console output.
- Open the Entity analytics page and navigate to dashboards (by clicking
"Go to dashboards")
- Add the privileged users from the previous step to the privileged
users index using the CSV File Upload option.
> [!NOTE]
> Any errors regarding risk scoring are unrelated to this PR, and are
being resolved separately
- You should see a panel that says "Enable Privileged access detection".
Click "Install", and you'll meet a loading state.
- Once complete, you shouldn't see any results. That's because, even
though we **install** the ML jobs by default, we don't **run** them by
default.
- Click "ML Job Settings", and note that only `pad` jobs should be
displaying in this callout. Feel free to test this callout's links and
filtering options.
- Click "Run job" next to the job called
`pad_linux_high_count_privileged_process_events_by_user`. This is the
job for which we have anomaly data.
- Click away from the callout. **You still shouldn't see data.** That's
because there just aren't any anomalies "today".
- Change the global date filter at the top of the screen to "Last 30
days".
- You should see something similar to this:
<img width="1441" alt="Screenshot 2025-06-16 at 12 50 25 AM"
src="https://github.com/user-attachments/assets/2b3f11f2-f45d-4716-bb8e-79d2b585aa3e"
/>
- Congrats for making it this far! Some things to play around with:
- Click around on the anomaly filters (i.e., click 25-50), and notice
that the results will change.
- Ensure that the ordering of the users is based on the **highest single
anomaly score in any visible bucket**. Meaning, if user `samwise` has
only one anomaly, but its score is 99, and user `frodo` has dozens of
anomalies, but no higher than, say, 80, `samwise` will be on the top of
the list.
- Click the user names to open the appropriate user flyout
- Change the global time filter to change ranges, and ensure the data
shows up appropriately. The buckets will try to roughly show 30 buckets
total, but will have a maximum of 3 hours, meaning if your range is too
small, it may show fewer than 30 buckets. This is intentional behavior,
as the PAD jobs have an anomaly job window of 3 hours.
- Click the button that says "View all in Anomaly Explorer", which uses
the currently selected global time range, and compare results
> [!WARNING]
> Remember that the users in the privileged user monitoring table are
only those that you designated as privileged users in a previous step.
In contrast, the Anomaly Explorer page shows **all** users.
Additionally, note that there may be very slight differences between the
swimlanes, because ES|QL calculates the bucket dates slightly
differently than the Anomaly Explorer. **This should not affect the
results themselves, only the buckets that an individual anomaly score
might fall in.** You might see a single anomaly fall into one visual
bucket instead of another, but the date should be correct.
# Helpful hints
If you'd like an easy way to "reset" the Privileged Access Detection
package and delete its associated jobs (and anomaly data), so that you
can redo the onboarding flow, you can run the below commands (changing
any credentials as necessary):
```shell
curl "http://localhost:5601/api/fleet/epm/packages/pad/0.5.0" \
-X 'DELETE' \
-H 'elastic-api-version: 2023-10-31' \
-H 'kbn-xsrf:true' \
--user elastic:changeme
curl "http://localhost:5601/internal/ml/jobs/delete_jobs" \
-H 'kbn-xsrf:true' \
--user elastic:changeme \
-X 'POST' \
-H 'elastic-api-version: 1' \
-H 'x-elastic-internal-origin:kibana' \
-H 'Content-Type: application/json' \
--data-raw '{
"jobIds": [
"pad_linux_high_count_privileged_process_events_by_user",
"pad_linux_high_median_process_command_line_entropy_by_user",
"pad_linux_rare_process_executed_by_user",
"pad_okta_high_sum_concurrent_sessions_by_user",
"pad_okta_rare_host_name_by_user",
"pad_okta_rare_region_name_by_user",
"pad_okta_rare_source_ip_by_user",
"pad_okta_spike_in_group_application_assignment_changes",
"pad_okta_spike_in_group_lifecycle_changes",
"pad_okta_spike_in_group_membership_changes",
"pad_okta_spike_in_group_privilege_changes",
"pad_okta_spike_in_user_lifecycle_management_changes",
"pad_windows_high_count_group_management_events",
"pad_windows_high_count_special_logon_events",
"pad_windows_high_count_special_privilege_use_events",
"pad_windows_high_count_user_account_management_events",
"pad_windows_rare_device_by_user",
"pad_windows_rare_group_name_by_user",
"pad_windows_rare_privilege_assigned_to_user",
"pad_windows_rare_region_name_by_user",
"pad_windows_rare_source_ip_by_user"
],
"deleteUserAnnotations": true,
"deleteAlertingRules": false
}
'
```
# What's left?
- This PR does not implement "proactive" permissions checks. Instead,
relevant permission issues will cause requests/queries to fail, which
are appropriately shown in the UI for troubleshooting. This proactive
check will be tackled in [this (private) followup
issue](https://github.com/elastic/security-team/issues/12822).
- There is not yet a way to "upgrade" the package directly from the
Privileged User Monitoring screen. This would be a nice addition later
on, instead of requiring users to navigate to the integration page. This
behavior will be tackled in [this (private) followup
issue](https://github.com/elastic/security-team/issues/12823).
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This PR removes the current GET user route as it works with IDs and we
do not use IDs in privmon.
We'd rather remove this before it gets documented and pushed out. We can
revert this PR later if needed
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Mark Hopkin <mark.hopkin@elastic.co>
## Summary
This PR adds the ability to upload a CSV file with privileged users from
the Entity Analytics pages
## Changes
### Backend
- Added (or completed) the upload CSV route:
`/api/entity_analytics/monitoring/users/_csv`
- Added shared utilities for batching with Node streams
- Added bulk processing actions for the upload
- Parsing users from CSV
- Soft delete for omitted users
- Batch upsert via the bulk API
- Added a check for installing all required privmon resources
### Frontend
- File uploader components
- File validation logic
- Updated EA privmon page to account for the new flow
- Added managing users panels
- open upload flow (same as asset criticality)
## Screen recording
https://github.com/user-attachments/assets/7956f1cf-49e0-4430-8c23-7d6178a15342
## How to test
#### Prerequisite
Make sure you have a CSV file with usernames
Check
[here](https://gist.github.com/tiansivive/0be2f09e1bb380fdde6609a131e929ed)
for a little helper script
Create a few copies where some of the users are deleted, in order to
test soft delete
1. Start up kibana and ES
2. Navigate to Security > Entity Analytics > Privilege User Monitoring
3. Select the `File` option to add data
4. Add one of the CSV files to the open modal and upload
5. Repeat but now upload one of files with the omitted users
Alternatively, testing only the backend only is possible by directly
hitting the API wit curl
```
curl -u elastic:changeme \
-X POST "http://localhost:5601/api/entity_analytics/monitoring/users/_csv" \
-H "kbn-xsrf: true" \
-F "file=@test.csv;type=text/csv"
```
#### Verifying
Easiest way is to use the dev tools to `_search` the privmon users index
with:
```
GET .entity_analytics.monitoring.users-default/_search
```
Look for number of hits and/or use `query` to search for omitted users.
## Remaining work
- [x] API integration tests
- [ ] Batching logic unit tests
- [ ] E2E tests?
---------
Co-authored-by: machadoum <pablo.nevesmachado@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Customers who enabled risk scoring in a version below 8.18 and then
upgraded to 8.18 or 9.0, will find risk scoring stops working in 0-30
days, this is because the ingest pipeline to set event ingested is not
installed, but is specified on the risk scoring index, causing all
writes to fail.
This PR fixes this by installing the ingest pipeline in all spaces where
either:
- risk scoring is set up (the risk-engine-configuration saved object is
present)
- the asset criticality index exists
This PR also increases the internal versioning of the asset criticality
and risk scoring indices, to trigger the migration which will:
- add the pipeline to the asset criticality index
- add the pipeline to a risk scoring index which hasnt been rolled over
- remove the pipeline from the risk scoring latest index if it is set
(if risk scoring was set up in 9.0 or 8.18 for the first time) we remove
this because the latest index should not modify the risk scoring
documents.
The result is that asset criticality and risk scoring should work and
event.ingested should correctly be set.
## Other important technical changes
- whenevr we increase the risk scoring internal index version, this will
now trigger a rollover on the data stream
- asset criticality and risk scoring index migrations now support
settings changes
## test steps
- on 8.17 initialise risk engine and generate some risk scores
- upgrade to 8.18.0 or 9.0.0
- rollover the risk score latest datastream dev tool `POST
/risk-score.risk-score-default/_rollover`
- verify risk scoring has broken, easy way is by assigning asset
criticality in the entity flyout
- upgrade to 9.0.3 or 8.18.3
- verify risk scoring and asset criticality work
## Test steps for local env:
- start kibana
- checkout SDG branch
https://github.com/elastic/security-documents-generator/pull/153
- run `yarn start risk-score-ingest-bug` AS SUPERUSER
- observe asset criticality assignment results in an error / is not
performed
- restart kibana
- observe the ingest pipeline has been created and it works again
dev tools verification:
```
# the pipeline should have been created
GET /_ingest/pipeline/entity_analytics_create_eventIngest_from_timestamp-pipeline-default
# the mapping version should be 4 and the default pipeline should be set
GET /.asset-criticality.asset-criticality-default
# the mapping version should be 4 and the default pipeline should be set
GET /risk-score.risk-score-default
# the default pipeline should be set
GET /_index_template/.risk-score.risk-score-default-index-template
# the default pipeline should NOT be set
GET /risk-score.risk-score-latest-default
```
Other verifications:
- risk scoring should work
- asset criticality assignment should work
- ???
---------
Co-authored-by: machadoum <pablo.nevesmachado@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
issue: https://github.com/elastic/security-team/issues/12309
Introduces the integrations card in the SIEM migrations topic, after the
"start rule migration" card.
Requirements:
- The same integrations card UI as the rest of the onboarding topics.
- When no migration is complete, show the same recommended integrations
as the default topic.
- When some migration is complete, show the integrations detected by the
rule migrations (all of them).
- Render a label next to the title showing the number of migration rules
linked to the integration (both prebuilt and custom rules)
- Sort the integrations list by the number of rules linked.
- As the last integration item in the list, render a card to link the
integration auto-import
### Screenshots
When no migration has been executed yet:
When some migrations have been completed:
Auto import card is appended at the end:
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Resolves https://github.com/elastic/security-team/issues/12483
This PR changes REST API Endpoints scheme to align with
https://github.com/elastic/security-team/issues/12483. Below is the
summary of changes done.
### API Scheme changes
The REST API scheme has been changed to reflect
https://github.com/elastic/security-team/issues/12483. This is pretty
much self explanatory as defined in below openapi schema yaml :
-
[x-pack/solutions/security/plugins/security_solution/common/siem_migrations/model/api/rules/rule_migration.schema.yaml](https://github.com/elastic/kibana/pull/219597/files#diff-3025af9eca156f3308474e2b42808da1531423457b7791daf6660db95a53b978)
### Introduction of Delete Migration API
This PR also adds `DELETE` method on route
`/rules/siem_migrations/{migration_id}` for deleting a migrations.
Deleting a migration does below operations:
- Stops a migration if it is running
- Deletes the rules, resources related to migration and migration
document itself.
### File Reorganizations
Directly structure has been changed a little bit to reflect the
endpoint. There is a sub-directory called `rules` which deals with only
`rules` of the migration and the root directly only contains the
endpoints related to the migration.
#### Before
```
//siem_migrations/rules/api
├── create.ts
├── get.ts
├── start.ts
├── update.ts
├──
├──
├──
```
#### After
```
//siem_migrations/rules/api
├── create.ts
├── delete.ts
├── get.ts
├── rules
│ ├── add.ts
│ ├── get.ts
│ └── update.ts
├──
├──
├──
```
## Migration Strategy
### TL,DR;
```mermaid
flowchart TD
StartM[Start Migration] --> isMigExists{Does Migration Index Exists}
isMigExists -->|Yes|FetchMDoc[Fetch Migration Docs]
isMigExists -->|No|CreateMIndex[Create MigrationIndex]
CreateMIndex --> FetchMDoc
FetchMDoc --> FetchMRules[Fetch Migration Stats Rules index]
FetchMRules --> FilterMigration{Filter Migration Docs not in Migration Index}
FilterMigration --> |is Empty|END[END]
FilterMigration --> |is Not Empty| CreateMDocs[Create Migration Docs]
CreateMDocs --> END
```
At the time of merging this PR, the Migration indices can be in 3
states:
### There are migrations created after
https://github.com/elastic/kibana/pull/216164 and this means that there
are `some` migrations existing in
`.kibana-siem-migrations-migrations-<space_id>` and migrations created
before above mentioned PR will only exist in
`.kibana-siem-migrations-rules-<space_id>`.
In this case `migrateRuleMigrationIndex` will create migration in below
steps:
1. Look for **all** migration Documents in
`.kibana-siem-migrations-migrations-<space_id>`
2. Get **all** Migrations stats from
`.kibana-siem-migrations-rules-<space_id>` which includes below
properties
- migration_id : will help in reconciling the migration id in
.kibana-siem-migrations-migrations-<space_id>` index
- created_at : Date on which migration_id was created.
- created_by: User who created the migrations.
3. A new document with above migration will be added to
`.kibana-siem-migrations-migrations-<space_id>`.
4. Now both `.kibana-siem-migrations-migrations-<space_id>` and
`.kibana-siem-migrations-rules-<space_id>` will be in sync.
### Alternatively, there are no migration created after
https://github.com/elastic/kibana/pull/216164. In that case, there is a
possibility that `.kibana-siem-migrations-migrations-<space_id>`, will
not even exist.
In this case `migrateRuleMigrationIndex` will create migration in below
steps:
1. Create the `.kibana-siem-migrations-migrations-<space_id>` index.
2. Do steps mentioned in above scenario.
### Once the migrations has been run successfully, both
`.kibana-siem-migrations-migrations-<space_id>` index and
`.kibana-siem-migrations-rules-<space_id>` will be in sync.
1. In this case, migration will not run, since it tries to filter the
migrations by `id` which exist in
`kibana-siem-migrations-rules-<space_id>` but do not exist in
`kibana-siem-migrations-migrations-<space_id>`
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR introduces basic CRUD routes for handling Privileged Users in
Entity Analytics.
The following routes are available:
* CREATE: `POST /api/entity_analytics/monitoring/users`
* GET: `GET /api/entity_analytics/monitoring/users/{id}`
* LIST: `GET /api/entity_analytics/monitoring/users/list`
* UPDATE: `PUT /api/entity_analytics/monitoring/users/{id}`
* DELETE: `DELETE /api/entity_analytics/monitoring/users/{id}`
For CREATE and UPDATE, the request body should be of type:
```
{ "user_name": string, is_monitored: boolean }
```
The reason for snake_case is to align better with the upcoming csv and
json upload work.
This PR already introduces boilerplate code (registering the endpoints
and handlers) for those routes). We might want to change this.
## How to test
1. Start a fresh Kibana instance
2. Enable the `EntityAnalyticsPrivilegeMonitoring` Feature Flag.
3. Initialise the privmon engine with `POST
kbn:/api/entity_analytics/monitoring/engine/init`
4. Test any of the endpoints above
* Make sure to either note down the returned `id`s or simply query the
underlying index: `.entity_analytics.monitoring.users-<your namespace>`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Bootstrap Privileged User Monitoring page. This page is hidden behind
`privilegeMonitoringEnabled` flag.
### Included
* Add the Privileged User Monitoring page content according to design
* Link integrations to the integrations page
* Find index modal
* New API to search for compatible indices
* It also renames the navigation title to only have the first letter
capitalised.
### Not Included
* The navigation is already implemented by
https://github.com/elastic/kibana/pull/217180
* The video introduction
* The final API call in the "choose index" is out of scope for this
issue.
* The CSV upload functionality is entirely out of scope for this ticket.
* The "Sample Dashboard"
* The link to docs
### How to test it?
* Enable `privilegeMonitoringEnabled` flag.
* Start kibana.
* Use the menu to navigate to the Priv User monitoring page
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR introduces the first building blocks for the [Entity Analytics
Privileged
Monitoring](https://github.com/elastic/security-team/issues/9971).
We follow the approach used in the Entity Store and add a new "Engine",
which consists of the following components:
* Public API
* INIT and HEALTH routes
* Kibana task
* Privilege Monitoring Data Client
* Engine Saved Object
* API key manager
* Related storage indices
* Feature Flag: `privilegeMonitoringEnabled` set to `false` by default.
* API integration test configuration
* only tests that the health endpoint is available
* Auditing and Telemetry
## Testing steps
1. Make sure to add `privilegeMonitoringEnabled` to your
`kibana.dev.yaml`
2. In devtools, ensure the API is working with `GET
kbn:/api/entity_analytics/monitoring/privileges/health`
3. Start the engine with: `POST
kbn:/api/entity_analytics/monitoring/engine/init`
4. Look for `DEBUG` logs mentioning the
`entity_analytics:monitoring:privileges:engine` task
---------
Co-authored-by: CAWilson94 <charlotte.wilson@elastic.co>
Co-authored-by: Charlotte Alexandra Wilson <CAWilson94@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
**Resolves: https://github.com/elastic/security-docs/issues/3019**
## Summary
In this PR I am improving documentation for the bulk_actions endpoint.
The focus is on improving descriptions of parameters and providing
examples for all important situations, especially setting / adding
actions.
# Testing
1. cd x-pack/solutions/security/plugins/security_solution
2. yarn openapi:bundle:detections
3. Take the bundled file
(docs/openapi/ess/security_solution_detections_api_2023_10_31.bundled.schema.yaml)
and load it into bump.sh console to see the changes.
4. Compare the changes with the [Legacy
documentation](https://www.elastic.co/guide/en/security/current/rule-api-overview.html)
You can also use this [link](https://bump.sh/jkelas/doc/kibana_wip/)
where I deployed the generated bundled doc.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
**Partially resolves: #211808**
## Summary
This is the first part of the migration effort, containing changes for:
- CRUD endpoints
- BULK Actions
- Export / Import Rule
- Find Rule
- List Tags
- Get Status
- Install Rule
I migrated the examples and the description of the fields. Some of the
fields contained description that was very similar to the legacy, then I
didn't change it. I only modified the descriptions where it was
valuable.
I also discovered some problems. For example the value for the 'query'
field, always shows 'EQL query to execute'. I reported this to the docs
team, in the 'next-api-reference' channel.
Another issue was with 'related_integrations field', which also didn't
show the long description. I also wrote about it to the docs team
[here](https://elastic.slack.com/archives/C05UL5YC06B/p1740137094701209).
In this PR I decided to try moving the description one level up, where
it renders properly.
# Testing
1. cd x-pack/solutions/security/plugins/security_solution
2. yarn openapi:bundle:detections
3. Take the bundled file
(docs/openapi/ess/security_solution_detections_api_2023_10_31.bundled.schema.yaml)
and load it into bump.sh console to see the changes.
4. Compare the changes with the [Legacy
documentation](https://www.elastic.co/guide/en/security/current/rule-api-overview.html)
You can also use this [link](https://bump.sh/jkelas/doc/kibana_wip/)
where I deployed the generated bundled doc.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
**Partially addresses:** #211808,
https://github.com/elastic/security-docs/issues/5981 (internal)
**Resolves: #208329**
## Summary
This is the second part of the migration effort, containing changes for:
- BULK CRUD (removing, for v.9.0)
The PR also contains changes for ticket #208329 - as changes for
removing of dead code for handling Bulk CRUD endpoints had to be
combined together with removing the schema files for Bulk CRUD
endpoints.
This PR will be backported only to versions for Kibana v9
# Testing
1. cd x-pack/solutions/security/plugins/security_solution
2. yarn openapi:bundle:detections
3. Take the bundled file
(docs/openapi/ess/security_solution_detections_api_2023_10_31.bundled.schema.yaml)
and load it into bump.sh console to see the changes.
4. Compare the changes with the [Legacy
documentation](https://www.elastic.co/guide/en/security/current/rule-api-overview.html)
You can also use this [link](https://bump.sh/jkelas2/doc/kibana_wip2/)
where I deployed the generated bundled doc.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
[META
issue](https://github.com/elastic/security-docs-internal/issues/57)
This PR improves the API documentation for timeline/notes/pinned events
as per the definition in the meta issue.
### Notes
#### Timeline API
- `eventType`:
- is always set to `all`
- looks like it's been superseded by dataView?
- I marked it as `deperecated`
- `eventIdToNoteIds`:
- there's a type mismatch between what the frontend expects and what is
returned
- also it does not seem to be used anymore?
- it needs to be investigated further
#### Notes API
- `GlobalNote`:
- looks like it's not used anymore
- I removed it, nothing broke
- `eventIngested`, `eventTimestamp`, `eventDataView` and `overrideOwner`
are all not used on the patch note endpoint, I removed them. The
`event*` ones I have never seen before to be honest.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## For reviewers:
Only `*.schema.yml` files were edited (excluding
`*.bundled.schema.yml`). Rest of the changes comes from auto generation
and can be ignored.
## Description
Part of DW team effort - elastic/security-team#11804
This PR aligns the property/schema descriptions and examples in
AsciiDocs with OpenAPI schemas. The primary goal of this PR was not to
extend or enhance the documentation but to migrate from one system to
another.
Ascii docs -
https://www.elastic.co/guide/en/security/8.17/management-api-overview.html
OpenApi generated docs -
https://www.elastic.co/docs/api/doc/kibana/operation/operation-endpointgetactionslist
Changes:
Copied missing property descriptions from AsciiDoc to OpenApi properties
Copied existing AsciiDoc examples for both requests and responses
Fixed falsy query object in some GET requests - in OpenApi it was
defined as an object, not as path query params.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Paul Tavares <56442535+paul-tavares@users.noreply.github.com>
Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com>
## Summary
Implements the access controls for SIEM rule migrations.
## API changes
- All API routes have been secured with "SIEM Migration" feature checks
- Start migration API route now checks if the user has privileges to use
the connector ID received
## UI changes
### Onboarding SIEM migrations
- AI Connector selection
- Actions & Connectors: Read -> This privilege allows reading and
selecting a connector
Otherwise, we show a callout with the missing privileges:
- Create a migration
- Security All -> Main Security read & write access
- Siem Migrations All -> new feature under the Security catalog
- Actions & Connectors: Read -> This privilege allows connector
execution for LLM calls
Otherwise, we show a callout with the missing privileges:
### Rule Translations page
- Minimum privileges to make the page accessible (read access):
- Security Read -> Main Security read access
- Siem Migrations All -> new feature under the Security catalog
Otherwise, we hide the link in the navigation and display the generic
empty state if accessed:
- To successfully install rules the following privileges are also
required (write access):
- Security All -> Main Security read & write access
- Index privileges for `.alerts*` pattern: _read, write,
view_index_metadata, manage_
- Index privileges for `lookup_*` pattern: _read_
Otherwise, we show a callout at the top of the page, this callout is
consistent with the one displayed on the Detection Rules page
(`/app/security/rules`)
- To retry rule translations (upload missing macros/lookups or retry
errors)
- Actions & Connectors: Read -> This privilege allows connector
execution for LLM calls
Otherwise, when attempted, we show a toast with the missing privilege.
## Other changes
- Technical preview label
- No connector selected toast
https://github.com/user-attachments/assets/e4900129-ae9c-413f-9a41-f7dca452e71d
## Fixes
- [Fixed] Not possible to select a connector when no connector is
selected:
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
# Summary
As part of the effort to add missing content for Security APIs, this PR
introduces a few missing request, response, and parameter examples for
Detection Engine Alert and migration APIs.
## Summary
[Internal link](https://github.com/elastic/security-team/issues/10820)
to the feature details
These changes add a functionality which enables related integrations
functionality for migration rules:
* related integration are shown in the migration rules table
* user can navigate to the integration page to see instructions about
installation process
### Other tasks and fixes
* Default sorting in the table (by `Stats` => by `Author` => by
`Severity` => by `Updated`)
> [!NOTE]
> This feature needs `siemMigrationsEnabled` experimental flag enabled
to work.
## Screen recording
<img width="1838" alt="Screenshot 2024-12-17 at 19 26 47"
src="https://github.com/user-attachments/assets/c1ed9d5d-e237-4dfe-b144-a80adbf46cd3"
/>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
[Internal link](https://github.com/elastic/security-team/issues/10820)
to the feature details
These changes add a functionality which allows user to retry failed
migration rules.
### Other tasks and fixes
* Integrated `MigrationReadyPanel` and `MigrationProgressPanel` to show
migration's `ready` and `running` states
* Migration stats pooling issue caused by waiting while there are no
pending migrations left. If any other operation triggers `startPooling`
during the waiting it will be ignored and thus latest stats will never
come back.
> [!NOTE]
> This feature needs `siemMigrationsEnabled` experimental flag enabled
to work.
### Testing note
1. Make sure you have a SIEM migration with failed rules
2. Open that migration via `Security > Rules > SIEM Rules Migrations >
{#MIGRATION_WITH_FAILED_RULES}`
3. You should see a `Reprocess rules (#)` button which triggers failed
rules reprocessing
## Screen recording
https://github.com/user-attachments/assets/d33dc4a0-1791-4869-aa8d-b0322b5f19c3
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
[Internal link](https://github.com/elastic/security-team/issues/10820)
to the feature details
These changes add functionality that allows to display matched prebuilt
rules details.
### New route
There is a new route
`/internal/siem_migrations/rules/{migration_id}/prebuilt_rules` that
will return all prebuilt rules matched by translated rules within a
specific migration.
### UI changes
The rule migration details flyout was updated to display matched
prebuilt rule data in both `Translation` and `Overview` tabs.
https://github.com/user-attachments/assets/3da49653-e0ab-4d8b-892e-dd05cf73743b
### Other changes
Also, as part of this PR, batching of a rule installation/creation was
added.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Sergi Massaneda <sergi.massaneda@gmail.com>
## Summary
Part of: https://github.com/elastic/security-team/issues/10667
Implementation of the Onboarding card to create migrations using the
flyout
> [!NOTE]
> This feature needs `siemMigrationsEnabled` experimental flag enabled
to work. Otherwise only the default topic will be available and the
topic selector won't be displayed.
<img width="1547" alt="image"
src="https://github.com/user-attachments/assets/f43c60a0-0631-44d8-ba82-7f16971ecc24">
#### To do in part 2:
- Complete implementation of migration finished panel: chart and stats
- Improve implementation of migration ready panel: should only allow
open the flyout
- Implement missing steps in the flyout: Macros and Lookups
### Test
Enable experimental flag
Use file:
[1733135547_1420.json](1733135547_1420.json)
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
### Summary
This PR introduces two internal API routes:
1. `/internal/api/endpoint/workflow_isnights`
2. `/internal/api/endpoint/workflow_isnights/{insightId}`
### Details
- The first route (`/internal/api/endpoint/workflow_isnights`) will be
used with the `securityWorkflowInsightsService.fetch` method to retrieve
stored insights.
- The second route
(`/internal/api/endpoint/workflow_isnights/{insightId}`) will work with
the `securityWorkflowInsightsService.update` method to update existing
insights.
---------
Co-authored-by: Joey F. Poon <joey.poon@elastic.co>
## Summary
[Internal link](https://github.com/elastic/security-team/issues/10820)
to the feature details
With these changes we add pagination functionality for the rules
migration table. This way we will improve the performance within the
page.
Also, added as part of these PR:
* moved `install` and `install_translated` routes to the `rules/api`
folder; before those were located in `rules/api/rules` and made
confusion
* a new `translation_stats` route to return stats for the specific
migration about the translated rules, like `total` number of the rules,
and number of `prebuilt`, `custom` and `installable` rules
* add `Updated` table column
* small UI fixes:
* use correct icon for "SIEM rule migration"
* do not remove "Install translated rules" button and rather disable it
when there are no installable rules
* do not allow user to update translation status via UI
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
[Internal link](https://github.com/elastic/security-team/issues/10820)
to the feature details
With these changes we two new routes:
* `/internal/siem_migrations/rules/install`: allows to install a
specific set of migration rules
* `/internal/siem_migrations/rules/install_translated`: allows to
install all translated rules in specified migration
Also we connect these two new API calls with the "Install" button within
the "migration rules" table and the "Install translated rules" button on
the "SIEM migration rules" page.
### Screenshots
https://github.com/user-attachments/assets/29390d07-eab5-4157-8958-1e3f8459db09
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Sergi Massaneda <sergi.massaneda@gmail.com>
## Summary
* Add two tabs to the Entity Store page
* The import entities tab has all the bulk upload content
* The status tab has the new content created on this PR
* Move the "clear entity store data" button to the header according to
design mockups.
* Delete unused stats route
* Rename `enablement` API docs to `enable`
* Add a new parameter to the status API (`withComponents`)
* Should I make it snake cased?
### import entities tab
### status tab
## How to test it
- Open security solution app with data
- Go to entity store page
- You shouldn't see the new tab because the engine is disabled
- Enable the engine and wait
- Click on the new tab that showed up
- It should list user and host engine components, and everything should
be installed
- Delete or misconfigure some of the resources, the new status should be
reflected on the tab.
## TODO:
- [x] Rebase main after https://github.com/elastic/kibana/pull/199762 is
merged
- [x] Remove temporary status hook
- [x] Fix the"clear entity data" button. It should re-fetch the status
API.
### Checklist
Reviewers should verify this PR satisfies this list as well.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Changes in this PR:
* Added `update` route to handle bulk rule migrations docs updates
* Exposed `id` field in `RuleMigration` object needed for ES bulk update
operation
* Updated SIEM migrations schemas to use `NonEmptyString` when it is
needed
## Testing locally
Enable the flag
```
xpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']
```
Create and start a rule migration. Then use `update` API to updated
corresponding docs.
cURL request examples:
<details>
<summary>Rules migration `create` POST request</summary>
```
curl --location --request POST 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1' \
--header 'Content-Type: application/json' \
--data '[
{
"id": "f8c325ea-506e-4105-8ccf-da1492e90115",
"vendor": "splunk",
"title": "Linux Auditd Add User Account Type",
"description": "The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.",
"query": "sourcetype=\"linux:audit\" type=ADD_USER \n| rename hostname as dest \n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \n| `security_content_ctime(firstTime)` \n| `security_content_ctime(lastTime)`\n| search *",
"query_language":"spl",
"mitre_attack_ids": [
"T1136"
]
},
{
"id": "7b87c556-0ca4-47e0-b84c-6cd62a0a3e90",
"vendor": "splunk",
"title": "Linux Auditd Change File Owner To Root",
"description": "The following analytic detects the use of the '\''chown'\'' command to change a file owner to '\''root'\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.",
"query": "`linux_auditd` `linux_auditd_normalized_proctitle_process`\r\n| rename host as dest \r\n| where LIKE (process_exec, \"%chown %root%\") \r\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \r\n| `security_content_ctime(firstTime)` \r\n| `security_content_ctime(lastTime)`\r\n| `linux_auditd_change_file_owner_to_root_filter`",
"query_language": "spl",
"mitre_attack_ids": [
"T1222"
]
}
]'
```
</details>
<details>
<summary>Rules migration `start` task request</summary>
- Assuming the connector `azureOpenAiGPT4o` is already created in the
local environment.
- Using the {{`migration_id`}} from the first POST request response
```
curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/start' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1' \
--header 'Content-Type: application/json' \
--data '{
"connectorId": "azureOpenAiGPT4o"
}'
```
</details>
<details>
<summary>Rules migration rules documents request</summary>
- Using the {{`migration_id`}} from the first POST request response.
```
curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1'
```
</details>
<details>
<summary>Rules migration `update` PUT request</summary>
- Using the {{`rule_migration_id_1`}} and {{`rule_migration_id_2`}} from
previous GET request response
```
curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1'
--data '[
{
"comments": [
"## Migration Summary\n- The `FROM` command is used to select the `logs-*` index pattern.\n- The `RENAME` command is used to rename the `host` field to `dest`.\n- The `WHERE` command filters the rows where `process_exec` contains the pattern `*chown *root*`.\n- The `STATS` command is used to aggregate the data, counting the number of occurrences and finding the minimum and maximum timestamps, grouped by `process_exec`, `proctitle`, `normalized_proctitle_delimiter`, and `dest`.\n- The macros `security_content_ctime` and `linux_auditd_change_file_owner_to_root_filter` are placeholders for the corresponding Splunk macros.",
"Additional comment 2.0"
],
"translation_result": "full",
"id": "{{rule_migration_id_1}}"
},
{
"created_by": "elastic2.0",
"elastic_rule": {
"severity": "high",
"title": "Linux Auditd Change File Owner To Root (UPDATED)"
},
"id": "{{rule_migration_id_2}}"
}
]'
```
</details>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR adds 2 new endpoints regarding enablement of the Entity Store:
* `api/entity_store/enable`, which initializes entity engines for both
`user` and `host` entities
* `api/entity_store/status`, which computes a global store status based
on the individual engine status
In addition, running initialization of multiple engines in parallel is
now allowed.
### How to test
1. Use dev tools to call `POST kbn:/api/entity_store/enable`
2. Check that two engines were created and that the status is
`installing` by calling `GET kbn:/api/entity_store/status`
3. Wait a few seconds and keep calling the `status` endpoint. Once
initialization finishes, the status should switch to `running`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
- This adds api removals for deprecated API's.
- Also updates OAS documentations and api types.
- Also updates/removes test.
These changes are intended to be for `9.0` and Serverless GA.
API's removed:
- `/api/endpoint/isolate`
- `/api/endpoint/unisolate`
- `/api/endpoint/policy/summaries`
- `/api/endpoint/suggestions/{suggestion_type}`
- `/api/endpoint/action_log/{agent_id}`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
It implements the background task to execute the rule migrations and the
API to manage them. It also contains a basic implementation of the
langGraph agent workflow that will perform the migration using
generative AI.
> [!NOTE]
> This feature needs `siemMigrationsEnabled` experimental flag enabled
to work. Otherwise, the new API routes won't be registered, and the
`SiemRuleMigrationsService` _setup_ won't be called. So no migration
task code can be reached, and no data stream/template will be installed
to ES.
### The rule migration task implementation:
- Retrieve a batch of N rule migration documents (50 rules initially, we
may change that later) with `status: pending`.
- Update those documents to `status: processing`.
- Execute the migration for each of the N migrations in parallel.
- If there is any error update the document with `status: error`.
- For each rule migration that finishes we set the result to the
storage, and also update `status: finished`.
- When all the batch of rules is finished the task will check if there
are still migration documents with `status: pending` if so it will
process the next batch with a delay (10 seconds initially, we may change
that later).
- If the task is stopped (via API call or server shut-down), we do a
bulk update for all the `status: processing` documents back to `status:
pending`.
### Task API
- `POST /internal/siem_migrations/rules` (implemented
[here](https://github.com/elastic/security-team/issues/10654)) ->
Creates the migration on the backend and stores the original rules. It
returns the `migration_id`
- `GET /internal/siem_migrations/rules/stats` -> Retrieves the stats for
all the existing migrations, aggregated by `migration_id`.
- `GET /internal/siem_migrations/rules/{migration_id}` -> Retrieves all
the migration rule documents of a specific migration.
- `PUT /internal/siem_migrations/rules/{migration_id}/start` -> Starts
the background task for a specific migration.
- `GET /internal/siem_migrations/rules/{migration_id}/stats` ->
Retrieves the stats of a specific migration task. The UI will do polling
to this endpoint.
- `PUT /internal/siem_migrations/rules/{migration_id}/stop` -> Stops the
execution of a specific migration running task. When a migration is
stopped, the executing task is aborted and all the rules in the batch
being processed are moved back to pending, all finished rules will
remain stored. When the Kibana server shuts down all the running
migrations are stopped automatically. To resume the migration we can
call `{migration_id}/start` again and it will take it from the same
rules batch it was left.
#### Stats (UI polling) response example:
```
{
"status": "running",
"rules": {
"total": 34,
"finished": 20,
"pending": 4,
"processing": 10,
"failed": 0
},
"last_updated_at": "2024-10-29T15:04:49.618Z"
}
```
### LLM agent Graph
The initial implementation of the agent graph that is executed per rule:
The first node tries to match the original rule with an Elastic prebuilt
rule. If it does not succeed, the second node will try to translate the
query as a custom rule using the ES|QL knowledge base, this composes
previous PoCs:
- https://github.com/elastic/kibana/pull/193900
- https://github.com/elastic/kibana/pull/196651
## Testing locally
Enable the flag
```
xpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']
```
cURL request examples:
<details>
<summary>Rules migration `create` POST request</summary>
```
curl --location --request POST 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1' \
--header 'Content-Type: application/json' \
--data '[
{
"id": "f8c325ea-506e-4105-8ccf-da1492e90115",
"vendor": "splunk",
"title": "Linux Auditd Add User Account Type",
"description": "The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.",
"query": "sourcetype=\"linux:audit\" type=ADD_USER \n| rename hostname as dest \n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \n| `security_content_ctime(firstTime)` \n| `security_content_ctime(lastTime)`\n| search *",
"query_language":"spl",
"mitre_attack_ids": [
"T1136"
]
},
{
"id": "7b87c556-0ca4-47e0-b84c-6cd62a0a3e90",
"vendor": "splunk",
"title": "Linux Auditd Change File Owner To Root",
"description": "The following analytic detects the use of the '\''chown'\'' command to change a file owner to '\''root'\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.",
"query": "`linux_auditd` `linux_auditd_normalized_proctitle_process`\r\n| rename host as dest \r\n| where LIKE (process_exec, \"%chown %root%\") \r\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \r\n| `security_content_ctime(firstTime)` \r\n| `security_content_ctime(lastTime)`\r\n| `linux_auditd_change_file_owner_to_root_filter`",
"query_language": "spl",
"mitre_attack_ids": [
"T1222"
]
}
]'
```
</details>
<details>
<summary>Rules migration `start` task request</summary>
- Assuming the connector `azureOpenAiGPT4o` is already created in the
local environment.
- Using the {{`migration_id`}} from the first POST request response
```
curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/start' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1' \
--header 'Content-Type: application/json' \
--data '{
"connectorId": "azureOpenAiGPT4o"
}'
```
</details>
<details>
<summary>Rules migration `stop` task request</summary>
- Using the {{`migration_id`}} from the first POST request response.
```
curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/stop' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1'
```
</details>
<details>
<summary>Rules migration task `stats` request</summary>
- Using the {{`migration_id`}} from the first POST request response.
```
curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/stats' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1'
```
</details>
<details>
<summary>Rules migration rules documents request</summary>
- Using the {{`migration_id`}} from the first POST request response.
```
curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1'
```
</details>
<details>
<summary>Rules migration all stats request</summary>
```
curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/stats' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1'
```
</details>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
issue: https://github.com/elastic/security-team/issues/10654?reload=1
Implements the persistence layer for the rule migrations from other
vendors, as part of the SIEM Rule migrations effort.
### Changes
- Schemas created for `SiemRuleMigration` document entity, along with
`ElasticRule` and `OriginalRule`.
- New API `/internal/siem_migrations/rules` was created:
- `POST` -> Receives an array of (original) rules and stores them with
`status: pending` to be processed. Responds with the `migration_id` that
will be used to start the migration background task (implementation
details here: https://github.com/elastic/security-team/issues/10850).
- `GET` -> (to be implemented later)
- New `SiemMigrationsService` added to the `securitySolution` route
context, to encapsulate all operations related to SIEM migrations (We
start with _rule_ migrations, but there are more "kinds" of SIEM
migrations in the pipeline: _dashboards_, _saved queries_...). It
contains:
- `SiemRuleMigrationsService` to encapsulate all operations related to
SIEM rule migrations.
- `RuleMigrationsDataStream` class to manage the
`.kibana.siem-rule-migrations-<spaceId>` data stream operations using
`DataStreamSpacesAdapter`.
- It exposes a client with abstracted operations that are exposed to the
API routes:
- `create`: indexes an array of _SiemRuleMigration_ documents to the
data stream
- `search`: searches _SiemRuleMigration_ documents by specific terms.
> [!NOTE]
> Without `siemMigrationsEnabled` experimental flag the new API route
won't be registered, and the `SiemRuleMigrationsService` _setup_ won't
be called, so no index/component template will be installed to ES.
### Testing locally
Enable the flag
```
xpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']
```
<details>
<summary>Example curl request</summary>
```
curl --location 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1' \
--header 'Content-Type: application/json' \
--data '[
{
"id": "f8c325ea-506e-4105-8ccf-da1492e90115",
"vendor": "splunk",
"title": "Linux Auditd Add User Account Type",
"description": "The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.",
"query": "sourcetype=\"linux:audit\" type=ADD_USER \n| rename hostname as dest \n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \n| `security_content_ctime(firstTime)` \n| `security_content_ctime(lastTime)`\n| search *",
"query_language":"spl",
"mitre_attack_ids": [
"T1136"
]
},
{
"id": "7b87c556-0ca4-47e0-b84c-6cd62a0a3e90",
"vendor": "splunk",
"title": "Linux Auditd Change File Owner To Root",
"description": "The following analytic detects the use of the '\''chown'\'' command to change a file owner to '\''root'\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.",
"query": "`linux_auditd` `linux_auditd_normalized_proctitle_process`\r\n| rename host as dest \r\n| where LIKE (process_exec, \"%chown %root%\") \r\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \r\n| `security_content_ctime(firstTime)` \r\n| `security_content_ctime(lastTime)`\r\n| `linux_auditd_change_file_owner_to_root_filter`",
"query_language": "spl",
"mitre_attack_ids": [
"T1222"
]
}
]'
```
</details>
The newly created documents can be retrieved using Kibana DevTools
console:
```
GET .kibana.siem-rule-migrations-default/_search
```
### Screenshots
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
* Update the Entity Store to retrieve indices from the security solution
data view.
* Create a new API that updates all installed entity engine indices
(`api/entity_store/engines/apply_dataview_indices`)
### How to test it?
* Install the entity store
* Check if the transform index has the security solutions data view
indices
* Call `apply_dataview_indices` API; it should not return changes
* Update the security solution data view indices
* Call `apply_dataview_indices` API and if the API response contains the
updated indices
* Check if the transform index also got updated
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
*Epic:** https://github.com/elastic/security-team/issues/9401 (internal)
## Summary
This PR includes Security Solution OpenAPI domain bundles into the production OpenAPI Kibana bundle. The result Kibana bundler is expected to be published to Bump.sh manually by @lcawl.
## Summary
fixes: https://github.com/elastic/security-team/issues/10235
fixes: https://github.com/elastic/security-team/issues/10237
This is the final PR for migrating over all timeline-related schemas and
types to the new generated zod schemas from our OpenAPI specs. (see
https://github.com/elastic/security-team/issues/10110)
On top of moving to the new schemas/types, this PR also cleans up usage
of now outdated types.
I'm aware of the size of this PR but rest assured, the changes are easy
to review and for most teams, only a handful of files need to be
reviewed:
```markdown
### elastic/security-defend-workflows
* x-pack/test/security_solution_endpoint/apps/endpoint/endpoint_solution_integrations.ts
### elastic/security-detection-rule-management
* x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/get_prebuilt_rules_and_timelines_status/get_prebuilt_rules_and_timelines_status_route.ts
* x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/perform_timelines_installation.ts
### elastic/security-detections-response
* x-pack/test/security_solution_cypress/cypress/objects/timeline.ts
### elastic/security-engineering-productivity
* x-pack/test/security_solution_cypress/cypress/objects/timeline.ts
* x-pack/test/security_solution_cypress/cypress/tasks/api_calls/timelines.ts
```
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
### Summary
This PR adds support for Kibana Spaces in the generated
`securitySolutionApi` service for integration tests.
Users can now pass an extra parameter `kibanaSpace` when calling a route
from the service. The provided space will be prefixed to the API's path.
If no argument is provided, it is default'ed to `'default'`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
1. Create a new public API endpoint : `DELETE
/api/risk_score/engine/dangerously_delete_data`
## Test cases Result
```
PASS x-pack/plugins/security_solution/server/lib/entity_analytics/risk_engine/routes/delete.test.ts (7.017 s)
risk engine cleanup route
invokes the risk engine cleanup route
✓ should call the router with the correct route and handler (71 ms)
✓ returns a 200 when cleanup is successful (64 ms)
✓ returns a 500 when cleanup is unsuccessful (57 ms)
✓ returns a 500 when cleanup is unsuccessful with multiple errors (53 ms)
when task manager is unavailable
✓ returns a 400 when task manager is unavailable (55 ms)
when user does not have the required privileges
✓ returns a 403 when user does not have the required privileges (88 ms)
```
### API Responses
## When multiple errors encountered
```
{
"risk_engine_cleanup": false,
"errors": [
{
"seq": 1,
"error": "resource_not_found_exception\n\tRoot causes:\n\t\tresource_not_found_exception: Transform with id [risk_score_latest_transform_default] could not be found"
},
{
"seq": 2,
"error": "index_not_found_exception\n\tRoot causes:\n\t\tindex_not_found_exception: no such index [risk-score.risk-score-default]"
},
{
"seq": 3,
"error": "index_template_missing_exception\n\tRoot causes:\n\t\tindex_template_missing_exception: index_template [.risk-score.risk-score-default-index-template] missing"
},
{
"seq": 4,
"error": "resource_not_found_exception\n\tRoot causes:\n\t\tresource_not_found_exception: .risk-score-mappings"
}
],
"status_code": 500
}
```
## Success
```
{
"risk_engine_cleanup": true
}
```
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### OpenAPI spec
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
Adds basic end to end tests for the engine CRUD routes.
I also noticed there was some inconsistency in the naming of the API
routes which I have fixed.
---------
Co-authored-by: machadoum <pablo.nevesmachado@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Tiago Vila Verde <tiago.vilaverde@elastic.co>
## Summary
Resolves https://github.com/elastic/security-docs-internal/issues/39 by
improving the Entity Analytics API docs content. Adds missing and
improves existing operation summaries and operation descriptions to
adhere to our [OAS
standards](450494532/API+reference+docs).
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR introduces the following API routes for listing Entity Store
"entities":
<meta charset="utf-8"><b style="font-weight:normal;"
id="docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb"><div
dir="ltr" style="margin-left:-0.75pt;" align="left">
List Entities | GET /api/entity_store/entities/list
-- | --
</div></b>
The PR includes the following:
- The OpenAPI schemas for the route
- The actual Kibana side endpoint
- Add searchEntities function to the `EntityStoreDataClient`
### How to test
1. Add some host/user data
* Easiest is to use
[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)
2. Make sure to add `entityStoreEnabled` under
`xpack.securitySolution.enableExperimental` in your `kibana.dev.yml`
3. In kibana dev tools or your terminal, call the `INIT` route for
either `user` or `host`.
4. You should now see 2 transforms in kibana. Make sure to re-trigger
them if needed so they process the documents.
5. Call the new API, and it should return entities
Implements https://github.com/elastic/security-team/issues/10517
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Resolves https://github.com/elastic/security-docs-internal/issues/35 by
improving the Timeline API docs content. Adds missing and improves
existing operation summaries and operation descriptions to adhere to our
[OAS
standards](450494532/API+reference+docs).
---------
Co-authored-by: Jatin Kathuria <jtn.kathuria@gmail.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR introduces the following API routes for setting up Entity Store
"engines":
<meta charset="utf-8"><b style="font-weight:normal;"
id="docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb"><div
dir="ltr" style="margin-left:-0.75pt;" align="left">
Initialise Engine | POST /api/entity_store/engines/<entity_type>/init
-- | --
Start Engine | POST /api/entity_store/engines/<entity_type>/start
Stop Engine | POST /api/entity_store/engines/<entity_type>/stop
Delete Engine | DELETE /api/entity_store/engines/<entity_type>
Get engine | GET /api/entity_store/engines/<entity_type>
List Engines | GET /api/entity_store/engines
</div></b>
The PR includes the following:
- Adding the `EntityManager` plugin (see elastic/obs-entities) as a
dependency of the Security Solution
- The OpenAPI schemas for the new routes
- The actual Kibana side endpoints
- A `Saved Object` to track the installed engines
- A new `EntityStoreDataClient`
- A new feature flag `entityStoreEngineRoutesEnabled`
### How to test
1. Add some host/user data
* Easiest is to use
[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)
2. Make sure to add `entityStoreEngineRoutesEnabled` under
`xpack.securitySolution.enableExperimental` in your `kibana.dev.yml`
3. In kibana dev tools or your terminal, call the `INIT` route for
either `user` or `host`.
4. You should now see 2 transforms in kibana. Make sure to re-trigger
them if needed so they process the documents.
5. Check that new entities have been observed by querying the new
entities index via:
* `GET .entities.v1.latest.ea*/_search`
6. Check the other endpoints are working (`START`, `STOP`, `LIST`, etc)
7. Calling `DELETE` should remove the transforms
Implements https://github.com/elastic/security-team/issues/10230
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
* Create new API `/internal/risk_score/engine/schedule_now`
* Fix storybook, it was broken due to some mock changes
* Update status API to return task status
* Add schedule risk engine panel to bulk asset criticality final step
### How to test it?
* Enable asset criticality in stack management / Advanced settings
* Enable the risk engine
* Open the asset criticality page inside the Manage section
* Upload a valid file
* Verify that the new panel is present and that you can schedule the
risk engine run by pressing the button
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Mark Hopkin <mark.hopkin@elastic.co>
Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
