## Summary
[Internal link](https://github.com/elastic/security-team/issues/10820)
to the feature details
With these changes we two new routes:
* `/internal/siem_migrations/rules/install`: allows to install a
specific set of migration rules
* `/internal/siem_migrations/rules/install_translated`: allows to
install all translated rules in specified migration
Also we connect these two new API calls with the "Install" button within
the "migration rules" table and the "Install translated rules" button on
the "SIEM migration rules" page.
### Screenshots
https://github.com/user-attachments/assets/29390d07-eab5-4157-8958-1e3f8459db09
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Sergi Massaneda <sergi.massaneda@gmail.com>
## Summary
* Add two tabs to the Entity Store page
* The import entities tab has all the bulk upload content
* The status tab has the new content created on this PR
* Move the "clear entity store data" button to the header according to
design mockups.
* Delete unused stats route
* Rename `enablement` API docs to `enable`
* Add a new parameter to the status API (`withComponents`)
* Should I make it snake cased?
### import entities tab
### status tab
## How to test it
- Open security solution app with data
- Go to entity store page
- You shouldn't see the new tab because the engine is disabled
- Enable the engine and wait
- Click on the new tab that showed up
- It should list user and host engine components, and everything should
be installed
- Delete or misconfigure some of the resources, the new status should be
reflected on the tab.
## TODO:
- [x] Rebase main after https://github.com/elastic/kibana/pull/199762 is
merged
- [x] Remove temporary status hook
- [x] Fix the"clear entity data" button. It should re-fetch the status
API.
### Checklist
Reviewers should verify this PR satisfies this list as well.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
The timeline API endpoints are currently implemented from a mix of HTTP
and GraphQL practices. Since GraphQL has been removed for a long time
now, we should make sure the endpoints conform to HTTP best practices.
This will allow us to simplify the API- and client-logic. Further,
third-parties accessing these APIs will have an easier time integrating.
### Usage of HTTP status codes
Depending on the error, the API endpoints currently return a `200` with
`{ code: 404, message: '(...)' }` or an actual HTTP error response with
e.g. `403` and the message in the body. The practice of returning 200s
with embedded error codes comes from GraphQL, where error codes are
always embedded.
Example of a current HTTP response of a failed timeline request:
```
HTTP status: 200
HTTP body:
{
"error_code": 409,
"messsage": "there was a conflict"
}
```
Going forward, all endpoints should return proper error codes and embed
the error messages in the response body.
```
HTTP status: 409
HTTP body:
{
"messsage": "there was a conflict"
}
```
### Removal of `{}` responses
Some timeline endpoints might return empty objects in case they were not
able to resolve/retrieve some SOs. The empty object implies a `404`
response. This creates complications on the client that now have to
provide extra logic for how to interpret empty objects.
Example of a current request of one of the endpoints that allows empty
responses.
```
HTTP status: 200
{}
```
The absence of an object, on some of the listed endpoints, indicates a
404 or the top-level or embedded saved object.
Going forward, the endpoints should not return empty objects and instead
return the proper HTTP error code (e.g. `404`) or a success code.
```
HTTP status: 404
```
### No more nested bodies
Another relic of the GraphQL time is the nesting of request bodies like
this:
```
HTTP status: 200
HTTP body:
{
"data": {
"persistTimeline": {
(actual timeline object)
}
}
}
```
Combined with sometimes returning empty objects and potentially
returning a status code in the body, makes it overly complicated for
clients to reason about the response.
Going forward, the actual object(s) should be returned as a top-level
JSON object, omitting `data.persistX`.
```
HTTP status: 200
HTTP body:
{
(actual timeline object)
}
```
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Changes in this PR:
* Added `update` route to handle bulk rule migrations docs updates
* Exposed `id` field in `RuleMigration` object needed for ES bulk update
operation
* Updated SIEM migrations schemas to use `NonEmptyString` when it is
needed
## Testing locally
Enable the flag
```
xpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']
```
Create and start a rule migration. Then use `update` API to updated
corresponding docs.
cURL request examples:
<details>
<summary>Rules migration `create` POST request</summary>
```
curl --location --request POST 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1' \
--header 'Content-Type: application/json' \
--data '[
{
"id": "f8c325ea-506e-4105-8ccf-da1492e90115",
"vendor": "splunk",
"title": "Linux Auditd Add User Account Type",
"description": "The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.",
"query": "sourcetype=\"linux:audit\" type=ADD_USER \n| rename hostname as dest \n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \n| `security_content_ctime(firstTime)` \n| `security_content_ctime(lastTime)`\n| search *",
"query_language":"spl",
"mitre_attack_ids": [
"T1136"
]
},
{
"id": "7b87c556-0ca4-47e0-b84c-6cd62a0a3e90",
"vendor": "splunk",
"title": "Linux Auditd Change File Owner To Root",
"description": "The following analytic detects the use of the '\''chown'\'' command to change a file owner to '\''root'\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.",
"query": "`linux_auditd` `linux_auditd_normalized_proctitle_process`\r\n| rename host as dest \r\n| where LIKE (process_exec, \"%chown %root%\") \r\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \r\n| `security_content_ctime(firstTime)` \r\n| `security_content_ctime(lastTime)`\r\n| `linux_auditd_change_file_owner_to_root_filter`",
"query_language": "spl",
"mitre_attack_ids": [
"T1222"
]
}
]'
```
</details>
<details>
<summary>Rules migration `start` task request</summary>
- Assuming the connector `azureOpenAiGPT4o` is already created in the
local environment.
- Using the {{`migration_id`}} from the first POST request response
```
curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/start' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1' \
--header 'Content-Type: application/json' \
--data '{
"connectorId": "azureOpenAiGPT4o"
}'
```
</details>
<details>
<summary>Rules migration rules documents request</summary>
- Using the {{`migration_id`}} from the first POST request response.
```
curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1'
```
</details>
<details>
<summary>Rules migration `update` PUT request</summary>
- Using the {{`rule_migration_id_1`}} and {{`rule_migration_id_2`}} from
previous GET request response
```
curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1'
--data '[
{
"comments": [
"## Migration Summary\n- The `FROM` command is used to select the `logs-*` index pattern.\n- The `RENAME` command is used to rename the `host` field to `dest`.\n- The `WHERE` command filters the rows where `process_exec` contains the pattern `*chown *root*`.\n- The `STATS` command is used to aggregate the data, counting the number of occurrences and finding the minimum and maximum timestamps, grouped by `process_exec`, `proctitle`, `normalized_proctitle_delimiter`, and `dest`.\n- The macros `security_content_ctime` and `linux_auditd_change_file_owner_to_root_filter` are placeholders for the corresponding Splunk macros.",
"Additional comment 2.0"
],
"translation_result": "full",
"id": "{{rule_migration_id_1}}"
},
{
"created_by": "elastic2.0",
"elastic_rule": {
"severity": "high",
"title": "Linux Auditd Change File Owner To Root (UPDATED)"
},
"id": "{{rule_migration_id_2}}"
}
]'
```
</details>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR adds 2 new endpoints regarding enablement of the Entity Store:
* `api/entity_store/enable`, which initializes entity engines for both
`user` and `host` entities
* `api/entity_store/status`, which computes a global store status based
on the individual engine status
In addition, running initialization of multiple engines in parallel is
now allowed.
### How to test
1. Use dev tools to call `POST kbn:/api/entity_store/enable`
2. Check that two engines were created and that the status is
`installing` by calling `GET kbn:/api/entity_store/status`
3. Wait a few seconds and keep calling the `status` endpoint. Once
initialization finishes, the status should switch to `running`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
- This adds api removals for deprecated API's.
- Also updates OAS documentations and api types.
- Also updates/removes test.
These changes are intended to be for `9.0` and Serverless GA.
API's removed:
- `/api/endpoint/isolate`
- `/api/endpoint/unisolate`
- `/api/endpoint/policy/summaries`
- `/api/endpoint/suggestions/{suggestion_type}`
- `/api/endpoint/action_log/{agent_id}`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
It implements the background task to execute the rule migrations and the
API to manage them. It also contains a basic implementation of the
langGraph agent workflow that will perform the migration using
generative AI.
> [!NOTE]
> This feature needs `siemMigrationsEnabled` experimental flag enabled
to work. Otherwise, the new API routes won't be registered, and the
`SiemRuleMigrationsService` _setup_ won't be called. So no migration
task code can be reached, and no data stream/template will be installed
to ES.
### The rule migration task implementation:
- Retrieve a batch of N rule migration documents (50 rules initially, we
may change that later) with `status: pending`.
- Update those documents to `status: processing`.
- Execute the migration for each of the N migrations in parallel.
- If there is any error update the document with `status: error`.
- For each rule migration that finishes we set the result to the
storage, and also update `status: finished`.
- When all the batch of rules is finished the task will check if there
are still migration documents with `status: pending` if so it will
process the next batch with a delay (10 seconds initially, we may change
that later).
- If the task is stopped (via API call or server shut-down), we do a
bulk update for all the `status: processing` documents back to `status:
pending`.
### Task API
- `POST /internal/siem_migrations/rules` (implemented
[here](https://github.com/elastic/security-team/issues/10654)) ->
Creates the migration on the backend and stores the original rules. It
returns the `migration_id`
- `GET /internal/siem_migrations/rules/stats` -> Retrieves the stats for
all the existing migrations, aggregated by `migration_id`.
- `GET /internal/siem_migrations/rules/{migration_id}` -> Retrieves all
the migration rule documents of a specific migration.
- `PUT /internal/siem_migrations/rules/{migration_id}/start` -> Starts
the background task for a specific migration.
- `GET /internal/siem_migrations/rules/{migration_id}/stats` ->
Retrieves the stats of a specific migration task. The UI will do polling
to this endpoint.
- `PUT /internal/siem_migrations/rules/{migration_id}/stop` -> Stops the
execution of a specific migration running task. When a migration is
stopped, the executing task is aborted and all the rules in the batch
being processed are moved back to pending, all finished rules will
remain stored. When the Kibana server shuts down all the running
migrations are stopped automatically. To resume the migration we can
call `{migration_id}/start` again and it will take it from the same
rules batch it was left.
#### Stats (UI polling) response example:
```
{
"status": "running",
"rules": {
"total": 34,
"finished": 20,
"pending": 4,
"processing": 10,
"failed": 0
},
"last_updated_at": "2024-10-29T15:04:49.618Z"
}
```
### LLM agent Graph
The initial implementation of the agent graph that is executed per rule:
The first node tries to match the original rule with an Elastic prebuilt
rule. If it does not succeed, the second node will try to translate the
query as a custom rule using the ES|QL knowledge base, this composes
previous PoCs:
- https://github.com/elastic/kibana/pull/193900
- https://github.com/elastic/kibana/pull/196651
## Testing locally
Enable the flag
```
xpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']
```
cURL request examples:
<details>
<summary>Rules migration `create` POST request</summary>
```
curl --location --request POST 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1' \
--header 'Content-Type: application/json' \
--data '[
{
"id": "f8c325ea-506e-4105-8ccf-da1492e90115",
"vendor": "splunk",
"title": "Linux Auditd Add User Account Type",
"description": "The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.",
"query": "sourcetype=\"linux:audit\" type=ADD_USER \n| rename hostname as dest \n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \n| `security_content_ctime(firstTime)` \n| `security_content_ctime(lastTime)`\n| search *",
"query_language":"spl",
"mitre_attack_ids": [
"T1136"
]
},
{
"id": "7b87c556-0ca4-47e0-b84c-6cd62a0a3e90",
"vendor": "splunk",
"title": "Linux Auditd Change File Owner To Root",
"description": "The following analytic detects the use of the '\''chown'\'' command to change a file owner to '\''root'\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.",
"query": "`linux_auditd` `linux_auditd_normalized_proctitle_process`\r\n| rename host as dest \r\n| where LIKE (process_exec, \"%chown %root%\") \r\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \r\n| `security_content_ctime(firstTime)` \r\n| `security_content_ctime(lastTime)`\r\n| `linux_auditd_change_file_owner_to_root_filter`",
"query_language": "spl",
"mitre_attack_ids": [
"T1222"
]
}
]'
```
</details>
<details>
<summary>Rules migration `start` task request</summary>
- Assuming the connector `azureOpenAiGPT4o` is already created in the
local environment.
- Using the {{`migration_id`}} from the first POST request response
```
curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/start' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1' \
--header 'Content-Type: application/json' \
--data '{
"connectorId": "azureOpenAiGPT4o"
}'
```
</details>
<details>
<summary>Rules migration `stop` task request</summary>
- Using the {{`migration_id`}} from the first POST request response.
```
curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/stop' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1'
```
</details>
<details>
<summary>Rules migration task `stats` request</summary>
- Using the {{`migration_id`}} from the first POST request response.
```
curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/stats' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1'
```
</details>
<details>
<summary>Rules migration rules documents request</summary>
- Using the {{`migration_id`}} from the first POST request response.
```
curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1'
```
</details>
<details>
<summary>Rules migration all stats request</summary>
```
curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/stats' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1'
```
</details>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
issue: https://github.com/elastic/security-team/issues/10654?reload=1
Implements the persistence layer for the rule migrations from other
vendors, as part of the SIEM Rule migrations effort.
### Changes
- Schemas created for `SiemRuleMigration` document entity, along with
`ElasticRule` and `OriginalRule`.
- New API `/internal/siem_migrations/rules` was created:
- `POST` -> Receives an array of (original) rules and stores them with
`status: pending` to be processed. Responds with the `migration_id` that
will be used to start the migration background task (implementation
details here: https://github.com/elastic/security-team/issues/10850).
- `GET` -> (to be implemented later)
- New `SiemMigrationsService` added to the `securitySolution` route
context, to encapsulate all operations related to SIEM migrations (We
start with _rule_ migrations, but there are more "kinds" of SIEM
migrations in the pipeline: _dashboards_, _saved queries_...). It
contains:
- `SiemRuleMigrationsService` to encapsulate all operations related to
SIEM rule migrations.
- `RuleMigrationsDataStream` class to manage the
`.kibana.siem-rule-migrations-<spaceId>` data stream operations using
`DataStreamSpacesAdapter`.
- It exposes a client with abstracted operations that are exposed to the
API routes:
- `create`: indexes an array of _SiemRuleMigration_ documents to the
data stream
- `search`: searches _SiemRuleMigration_ documents by specific terms.
> [!NOTE]
> Without `siemMigrationsEnabled` experimental flag the new API route
won't be registered, and the `SiemRuleMigrationsService` _setup_ won't
be called, so no index/component template will be installed to ES.
### Testing locally
Enable the flag
```
xpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']
```
<details>
<summary>Example curl request</summary>
```
curl --location 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1' \
--header 'Content-Type: application/json' \
--data '[
{
"id": "f8c325ea-506e-4105-8ccf-da1492e90115",
"vendor": "splunk",
"title": "Linux Auditd Add User Account Type",
"description": "The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.",
"query": "sourcetype=\"linux:audit\" type=ADD_USER \n| rename hostname as dest \n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \n| `security_content_ctime(firstTime)` \n| `security_content_ctime(lastTime)`\n| search *",
"query_language":"spl",
"mitre_attack_ids": [
"T1136"
]
},
{
"id": "7b87c556-0ca4-47e0-b84c-6cd62a0a3e90",
"vendor": "splunk",
"title": "Linux Auditd Change File Owner To Root",
"description": "The following analytic detects the use of the '\''chown'\'' command to change a file owner to '\''root'\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.",
"query": "`linux_auditd` `linux_auditd_normalized_proctitle_process`\r\n| rename host as dest \r\n| where LIKE (process_exec, \"%chown %root%\") \r\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \r\n| `security_content_ctime(firstTime)` \r\n| `security_content_ctime(lastTime)`\r\n| `linux_auditd_change_file_owner_to_root_filter`",
"query_language": "spl",
"mitre_attack_ids": [
"T1222"
]
}
]'
```
</details>
The newly created documents can be retrieved using Kibana DevTools
console:
```
GET .kibana.siem-rule-migrations-default/_search
```
### Screenshots
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
* Update the Entity Store to retrieve indices from the security solution
data view.
* Create a new API that updates all installed entity engine indices
(`api/entity_store/engines/apply_dataview_indices`)
### How to test it?
* Install the entity store
* Check if the transform index has the security solutions data view
indices
* Call `apply_dataview_indices` API; it should not return changes
* Update the security solution data view indices
* Call `apply_dataview_indices` API and if the API response contains the
updated indices
* Check if the transform index also got updated
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
*Epic:** https://github.com/elastic/security-team/issues/9401 (internal)
## Summary
This PR includes Security Solution OpenAPI domain bundles into the production OpenAPI Kibana bundle. The result Kibana bundler is expected to be published to Bump.sh manually by @lcawl.
## Summary
fixes: https://github.com/elastic/security-team/issues/10235
fixes: https://github.com/elastic/security-team/issues/10237
This is the final PR for migrating over all timeline-related schemas and
types to the new generated zod schemas from our OpenAPI specs. (see
https://github.com/elastic/security-team/issues/10110)
On top of moving to the new schemas/types, this PR also cleans up usage
of now outdated types.
I'm aware of the size of this PR but rest assured, the changes are easy
to review and for most teams, only a handful of files need to be
reviewed:
```markdown
### elastic/security-defend-workflows
* x-pack/test/security_solution_endpoint/apps/endpoint/endpoint_solution_integrations.ts
### elastic/security-detection-rule-management
* x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/get_prebuilt_rules_and_timelines_status/get_prebuilt_rules_and_timelines_status_route.ts
* x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/perform_timelines_installation.ts
### elastic/security-detections-response
* x-pack/test/security_solution_cypress/cypress/objects/timeline.ts
### elastic/security-engineering-productivity
* x-pack/test/security_solution_cypress/cypress/objects/timeline.ts
* x-pack/test/security_solution_cypress/cypress/tasks/api_calls/timelines.ts
```
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
1. Create a new public API endpoint : `DELETE
/api/risk_score/engine/dangerously_delete_data`
## Test cases Result
```
PASS x-pack/plugins/security_solution/server/lib/entity_analytics/risk_engine/routes/delete.test.ts (7.017 s)
risk engine cleanup route
invokes the risk engine cleanup route
✓ should call the router with the correct route and handler (71 ms)
✓ returns a 200 when cleanup is successful (64 ms)
✓ returns a 500 when cleanup is unsuccessful (57 ms)
✓ returns a 500 when cleanup is unsuccessful with multiple errors (53 ms)
when task manager is unavailable
✓ returns a 400 when task manager is unavailable (55 ms)
when user does not have the required privileges
✓ returns a 403 when user does not have the required privileges (88 ms)
```
### API Responses
## When multiple errors encountered
```
{
"risk_engine_cleanup": false,
"errors": [
{
"seq": 1,
"error": "resource_not_found_exception\n\tRoot causes:\n\t\tresource_not_found_exception: Transform with id [risk_score_latest_transform_default] could not be found"
},
{
"seq": 2,
"error": "index_not_found_exception\n\tRoot causes:\n\t\tindex_not_found_exception: no such index [risk-score.risk-score-default]"
},
{
"seq": 3,
"error": "index_template_missing_exception\n\tRoot causes:\n\t\tindex_template_missing_exception: index_template [.risk-score.risk-score-default-index-template] missing"
},
{
"seq": 4,
"error": "resource_not_found_exception\n\tRoot causes:\n\t\tresource_not_found_exception: .risk-score-mappings"
}
],
"status_code": 500
}
```
## Success
```
{
"risk_engine_cleanup": true
}
```
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### OpenAPI spec
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
Adds basic end to end tests for the engine CRUD routes.
I also noticed there was some inconsistency in the naming of the API
routes which I have fixed.
---------
Co-authored-by: machadoum <pablo.nevesmachado@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Tiago Vila Verde <tiago.vilaverde@elastic.co>
## Summary
Fixes: https://github.com/elastic/security-team/issues/10133
Migrates some timeline routes to use the newly generated OpenAPI types.
The changes mostly affect pinned event and note routes to keep the
changes small. Routes that actually accept and return timeline objects
will come in a next step.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Resolves https://github.com/elastic/security-docs-internal/issues/39 by
improving the Entity Analytics API docs content. Adds missing and
improves existing operation summaries and operation descriptions to
adhere to our [OAS
standards](450494532/API+reference+docs).
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR introduces the following API routes for listing Entity Store
"entities":
<meta charset="utf-8"><b style="font-weight:normal;"
id="docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb"><div
dir="ltr" style="margin-left:-0.75pt;" align="left">
List Entities | GET /api/entity_store/entities/list
-- | --
</div></b>
The PR includes the following:
- The OpenAPI schemas for the route
- The actual Kibana side endpoint
- Add searchEntities function to the `EntityStoreDataClient`
### How to test
1. Add some host/user data
* Easiest is to use
[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)
2. Make sure to add `entityStoreEnabled` under
`xpack.securitySolution.enableExperimental` in your `kibana.dev.yml`
3. In kibana dev tools or your terminal, call the `INIT` route for
either `user` or `host`.
4. You should now see 2 transforms in kibana. Make sure to re-trigger
them if needed so they process the documents.
5. Call the new API, and it should return entities
Implements https://github.com/elastic/security-team/issues/10517
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Resolves https://github.com/elastic/security-docs-internal/issues/35 by
improving the Timeline API docs content. Adds missing and improves
existing operation summaries and operation descriptions to adhere to our
[OAS
standards](450494532/API+reference+docs).
---------
Co-authored-by: Jatin Kathuria <jtn.kathuria@gmail.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR introduces the following API routes for setting up Entity Store
"engines":
<meta charset="utf-8"><b style="font-weight:normal;"
id="docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb"><div
dir="ltr" style="margin-left:-0.75pt;" align="left">
Initialise Engine | POST /api/entity_store/engines/<entity_type>/init
-- | --
Start Engine | POST /api/entity_store/engines/<entity_type>/start
Stop Engine | POST /api/entity_store/engines/<entity_type>/stop
Delete Engine | DELETE /api/entity_store/engines/<entity_type>
Get engine | GET /api/entity_store/engines/<entity_type>
List Engines | GET /api/entity_store/engines
</div></b>
The PR includes the following:
- Adding the `EntityManager` plugin (see elastic/obs-entities) as a
dependency of the Security Solution
- The OpenAPI schemas for the new routes
- The actual Kibana side endpoints
- A `Saved Object` to track the installed engines
- A new `EntityStoreDataClient`
- A new feature flag `entityStoreEngineRoutesEnabled`
### How to test
1. Add some host/user data
* Easiest is to use
[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)
2. Make sure to add `entityStoreEngineRoutesEnabled` under
`xpack.securitySolution.enableExperimental` in your `kibana.dev.yml`
3. In kibana dev tools or your terminal, call the `INIT` route for
either `user` or `host`.
4. You should now see 2 transforms in kibana. Make sure to re-trigger
them if needed so they process the documents.
5. Check that new entities have been observed by querying the new
entities index via:
* `GET .entities.v1.latest.ea*/_search`
6. Check the other endpoints are working (`START`, `STOP`, `LIST`, etc)
7. Calling `DELETE` should remove the transforms
Implements https://github.com/elastic/security-team/issues/10230
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Creates CLI script tooling for building data, rules, exceptions, and
lists in any (local, cloud, serverless) environment for manual testing.
The initial commits here add generated clients for accessing security
solution, exceptions, and lists APIs and a placeholder script where
those clients are set up for use. See README for more details.
Much of the code in this PR is auto-generated clients. The hand written
code is intended to be primarily in `quickstart/modules/`, where we can
add wrapper code to simplify the process for common test environment
setup. For example, `createValueListException` takes an array of items
and some metadata and automatically creates a new value list and an
exception that references that value list. `/modules/data/` contains
functions to generate documents of arbitrary size, and we can add more
functions to create various other types of documents.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
