[Entity Analytics][Privmon] CSV upload (#221798)

## Summary
 
This PR adds the ability to upload a CSV file with privileged users from
the Entity Analytics pages

## Changes

### Backend

- Added (or completed) the upload CSV route:
`/api/entity_analytics/monitoring/users/_csv`
- Added shared utilities for batching with Node streams
- Added bulk processing actions for the upload
  - Parsing users from CSV
  - Soft delete for omitted users 
  - Batch upsert via the bulk API
- Added a check for installing all required privmon resources

### Frontend

- File uploader components
- File validation logic
- Updated EA privmon page to account for the new flow
- Added managing users panels 
  - open upload flow (same as asset criticality)

## Screen recording


https://github.com/user-attachments/assets/7956f1cf-49e0-4430-8c23-7d6178a15342

## How to test

#### Prerequisite

Make sure you have a CSV file with usernames
Check
[here](https://gist.github.com/tiansivive/0be2f09e1bb380fdde6609a131e929ed)
for a little helper script

Create a few copies where some of the users are deleted, in order to
test soft delete

1. Start up kibana and ES
2. Navigate to Security > Entity Analytics > Privilege User Monitoring
3. Select the `File` option to add data
4. Add one of the CSV files to the open modal and upload
5. Repeat but now upload one of files with the omitted users 

Alternatively, testing only the backend only is possible by directly
hitting the API wit curl
```
curl -u elastic:changeme \
  -X POST "http://localhost:5601/api/entity_analytics/monitoring/users/_csv" \
  -H "kbn-xsrf: true" \
  -F "file=@test.csv;type=text/csv"
```

#### Verifying

Easiest way is to use the dev tools to `_search` the privmon users index
with:
```
GET .entity_analytics.monitoring.users-default/_search
```

Look for number of hits and/or use `query` to search for omitted users. 


## Remaining work

- [x] API integration tests
- [ ] Batching logic unit tests
- [ ] E2E tests?

---------

Co-authored-by: machadoum <pablo.nevesmachado@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This commit is contained in:
Tiago Vila Verde 2025-06-18 15:23:03 +02:00 committed by GitHub
parent 5e28a4200f
commit a8a7574c66
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
55 changed files with 3065 additions and 389 deletions

View file

@ -11146,58 +11146,48 @@ paths:
- Security Entity Analytics API
/api/entity_analytics/monitoring/users/_csv:
post:
operationId: BulkUploadUsersCSV
operationId: PrivmonBulkUploadUsersCSV
requestBody:
content:
text/csv:
schema:
type: string
description: CSV file containing users to upsert
required: true
responses:
'200':
content:
application/json:
schema:
type: object
properties:
upserted_count:
type: integer
description: Successful response
summary: Upsert multiple monitored users via CSV upload
tags:
- Security Entity Analytics API
/api/entity_analytics/monitoring/users/_json:
post:
operationId: BulkUploadUsersJSON
requestBody:
content:
text/json:
multipart/form-data:
schema:
type: object
properties:
users:
items:
type: object
properties:
is_monitored:
type: boolean
user_name:
type: string
type: array
description: JSON file containing users to upsert
required: true
file:
description: The CSV file to upload.
format: binary
type: string
required:
- file
responses:
'200':
content:
application/json:
schema:
example:
errors:
- index: 1
message: Invalid monitored field
username: john.doe
stats:
failed: 1
successful: 1
total: 2
type: object
properties:
upserted_count:
type: integer
description: Successful response
summary: Upsert multiple monitored users via JSON upload
errors:
items:
$ref: '#/components/schemas/Security_Entity_Analytics_API_PrivmonUserCsvUploadErrorItem'
type: array
stats:
$ref: '#/components/schemas/Security_Entity_Analytics_API_PrivmonUserCsvUploadStats'
required:
- errors
- stats
description: Bulk upload successful
'413':
description: File too large
summary: Upsert multiple monitored users via CSV upload
tags:
- Security Entity Analytics API
/api/entity_analytics/monitoring/users/{id}:
@ -66988,6 +66978,34 @@ components:
required:
- type
- status
Security_Entity_Analytics_API_PrivmonUserCsvUploadErrorItem:
type: object
properties:
index:
nullable: true
type: integer
message:
type: string
username:
nullable: true
type: string
required:
- message
- index
- username
Security_Entity_Analytics_API_PrivmonUserCsvUploadStats:
type: object
properties:
failed:
type: integer
successful:
type: integer
total:
type: integer
required:
- successful
- failed
- total
Security_Entity_Analytics_API_RiskEngineScheduleNowErrorResponse:
type: object
properties:

View file

@ -13305,58 +13305,48 @@ paths:
- Security Entity Analytics API
/api/entity_analytics/monitoring/users/_csv:
post:
operationId: BulkUploadUsersCSV
operationId: PrivmonBulkUploadUsersCSV
requestBody:
content:
text/csv:
schema:
type: string
description: CSV file containing users to upsert
required: true
responses:
'200':
content:
application/json:
schema:
type: object
properties:
upserted_count:
type: integer
description: Successful response
summary: Upsert multiple monitored users via CSV upload
tags:
- Security Entity Analytics API
/api/entity_analytics/monitoring/users/_json:
post:
operationId: BulkUploadUsersJSON
requestBody:
content:
text/json:
multipart/form-data:
schema:
type: object
properties:
users:
items:
type: object
properties:
is_monitored:
type: boolean
user_name:
type: string
type: array
description: JSON file containing users to upsert
required: true
file:
description: The CSV file to upload.
format: binary
type: string
required:
- file
responses:
'200':
content:
application/json:
schema:
example:
errors:
- index: 1
message: Invalid monitored field
username: john.doe
stats:
failed: 1
successful: 1
total: 2
type: object
properties:
upserted_count:
type: integer
description: Successful response
summary: Upsert multiple monitored users via JSON upload
errors:
items:
$ref: '#/components/schemas/Security_Entity_Analytics_API_PrivmonUserCsvUploadErrorItem'
type: array
stats:
$ref: '#/components/schemas/Security_Entity_Analytics_API_PrivmonUserCsvUploadStats'
required:
- errors
- stats
description: Bulk upload successful
'413':
description: File too large
summary: Upsert multiple monitored users via CSV upload
tags:
- Security Entity Analytics API
/api/entity_analytics/monitoring/users/{id}:
@ -76488,6 +76478,34 @@ components:
required:
- type
- status
Security_Entity_Analytics_API_PrivmonUserCsvUploadErrorItem:
type: object
properties:
index:
nullable: true
type: integer
message:
type: string
username:
nullable: true
type: string
required:
- message
- index
- username
Security_Entity_Analytics_API_PrivmonUserCsvUploadStats:
type: object
properties:
failed:
type: integer
successful:
type: integer
total:
type: integer
required:
- successful
- failed
- total
Security_Entity_Analytics_API_RiskEngineScheduleNowErrorResponse:
type: object
properties: