15 commits

Author	SHA1	Message	Date
Ievgen Sorokopud	33263b25c2	[8.x] [Rules migration] Add rules migrations update route (#11209) (#200815) (#200910) ... # Backport This will backport the following commits from `main` to `8.x`: - [[Rules migration] Add rules migrations update route (#11209) (#200815)](https://github.com/elastic/kibana/pull/200815) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Ievgen Sorokopud","email":"ievgen.sorokopud@elastic.co"},"sourceCommit":{"committedDate":"2024-11-20T12:36:37Z","message":"[Rules migration] Add rules migrations update route (#11209) (#200815)\n\n## Summary\r\n\r\nChanges in this PR:\r\n* Added `update` route to handle bulk rule migrations docs updates\r\n* Exposed `id` field in `RuleMigration` object needed for ES bulk update\r\noperation\r\n* Updated SIEM migrations schemas to use `NonEmptyString` when it is\r\nneeded\r\n\r\n## Testing locally\r\n\r\nEnable the flag\r\n```\r\nxpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']\r\n```\r\n\r\nCreate and start a rule migration. Then use `update` API to updated\r\ncorresponding docs.\r\n\r\ncURL request examples:\r\n\r\n<details>\r\n <summary>Rules migration `create` POST request</summary>\r\n\r\n```\r\ncurl --location --request POST 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \\\r\n--header 'Content-Type: application/json' \\\r\n--data '[\r\n {\r\n \"id\": \"f8c325ea-506e-4105-8ccf-da1492e90115\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Add User Account Type\",\r\n \"description\": \"The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.\",\r\n \"query\": \"sourcetype=\\\"linux:audit\\\" type=ADD_USER \\n| rename hostname as dest \\n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \\n| `security_content_ctime(firstTime)` \\n| `security_content_ctime(lastTime)`\\n| search *\",\r\n \"query_language\":\"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1136\"\r\n ]\r\n },\r\n {\r\n \"id\": \"7b87c556-0ca4-47e0-b84c-6cd62a0a3e90\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Change File Owner To Root\",\r\n \"description\": \"The following analytic detects the use of the '\\''chown'\\'' command to change a file owner to '\\''root'\\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.\",\r\n \"query\": \"`linux_auditd` `linux_auditd_normalized_proctitle_process`\\r\\n| rename host as dest \\r\\n| where LIKE (process_exec, \\\"%chown %root%\\\") \\r\\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \\r\\n| `security_content_ctime(firstTime)` \\r\\n| `security_content_ctime(lastTime)`\\r\\n| `linux_auditd_change_file_owner_to_root_filter`\",\r\n \"query_language\": \"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1222\"\r\n ]\r\n }\r\n]'\r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration `start` task request</summary>\r\n\r\n- Assuming the connector `azureOpenAiGPT4o` is already created in the\r\nlocal environment.\r\n- Using the {{`migration_id`}} from the first POST request response\r\n\r\n```\r\ncurl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/start' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \\\r\n--header 'Content-Type: application/json' \\\r\n--data '{\r\n \"connectorId\": \"azureOpenAiGPT4o\"\r\n}'\r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration rules documents request</summary>\r\n\r\n- Using the {{`migration_id`}} from the first POST request response.\r\n\r\n```\r\ncurl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration `update` PUT request</summary>\r\n\r\n- Using the {{`rule_migration_id_1`}} and {{`rule_migration_id_2`}} from\r\nprevious GET request response\r\n\r\n```\r\ncurl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \r\n--data '[\r\n {\r\n \"comments\": [\r\n \"## Migration Summary\\n- The `FROM` command is used to select the `logs-*` index pattern.\\n- The `RENAME` command is used to rename the `host` field to `dest`.\\n- The `WHERE` command filters the rows where `process_exec` contains the pattern `*chown *root*`.\\n- The `STATS` command is used to aggregate the data, counting the number of occurrences and finding the minimum and maximum timestamps, grouped by `process_exec`, `proctitle`, `normalized_proctitle_delimiter`, and `dest`.\\n- The macros `security_content_ctime` and `linux_auditd_change_file_owner_to_root_filter` are placeholders for the corresponding Splunk macros.\",\r\n \"Additional comment 2.0\"\r\n ],\r\n \"translation_result\": \"full\",\r\n \"id\": \"{{rule_migration_id_1}}\"\r\n },\r\n {\r\n \"created_by\": \"elastic2.0\",\r\n \"elastic_rule\": {\r\n \"severity\": \"high\",\r\n \"title\": \"Linux Auditd Change File Owner To Root (UPDATED)\"\r\n },\r\n \"id\": \"{{rule_migration_id_2}}\"\r\n }\r\n]'\r\n```\r\n</details>\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"f6ac2cf8603ca633070e719f69b4fcef45ea92cb","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Threat Hunting","Team: SecuritySolution","backport:prev-minor"],"number":200815,"url":"https://github.com/elastic/kibana/pull/200815","mergeCommit":{"message":"[Rules migration] Add rules migrations update route (#11209) (#200815)\n\n## Summary\r\n\r\nChanges in this PR:\r\n* Added `update` route to handle bulk rule migrations docs updates\r\n* Exposed `id` field in `RuleMigration` object needed for ES bulk update\r\noperation\r\n* Updated SIEM migrations schemas to use `NonEmptyString` when it is\r\nneeded\r\n\r\n## Testing locally\r\n\r\nEnable the flag\r\n```\r\nxpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']\r\n```\r\n\r\nCreate and start a rule migration. Then use `update` API to updated\r\ncorresponding docs.\r\n\r\ncURL request examples:\r\n\r\n<details>\r\n <summary>Rules migration `create` POST request</summary>\r\n\r\n```\r\ncurl --location --request POST 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \\\r\n--header 'Content-Type: application/json' \\\r\n--data '[\r\n {\r\n \"id\": \"f8c325ea-506e-4105-8ccf-da1492e90115\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Add User Account Type\",\r\n \"description\": \"The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.\",\r\n \"query\": \"sourcetype=\\\"linux:audit\\\" type=ADD_USER \\n| rename hostname as dest \\n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \\n| `security_content_ctime(firstTime)` \\n| `security_content_ctime(lastTime)`\\n| search *\",\r\n \"query_language\":\"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1136\"\r\n ]\r\n },\r\n {\r\n \"id\": \"7b87c556-0ca4-47e0-b84c-6cd62a0a3e90\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Change File Owner To Root\",\r\n \"description\": \"The following analytic detects the use of the '\\''chown'\\'' command to change a file owner to '\\''root'\\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.\",\r\n \"query\": \"`linux_auditd` `linux_auditd_normalized_proctitle_process`\\r\\n| rename host as dest \\r\\n| where LIKE (process_exec, \\\"%chown %root%\\\") \\r\\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \\r\\n| `security_content_ctime(firstTime)` \\r\\n| `security_content_ctime(lastTime)`\\r\\n| `linux_auditd_change_file_owner_to_root_filter`\",\r\n \"query_language\": \"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1222\"\r\n ]\r\n }\r\n]'\r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration `start` task request</summary>\r\n\r\n- Assuming the connector `azureOpenAiGPT4o` is already created in the\r\nlocal environment.\r\n- Using the {{`migration_id`}} from the first POST request response\r\n\r\n```\r\ncurl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/start' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \\\r\n--header 'Content-Type: application/json' \\\r\n--data '{\r\n \"connectorId\": \"azureOpenAiGPT4o\"\r\n}'\r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration rules documents request</summary>\r\n\r\n- Using the {{`migration_id`}} from the first POST request response.\r\n\r\n```\r\ncurl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration `update` PUT request</summary>\r\n\r\n- Using the {{`rule_migration_id_1`}} and {{`rule_migration_id_2`}} from\r\nprevious GET request response\r\n\r\n```\r\ncurl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \r\n--data '[\r\n {\r\n \"comments\": [\r\n \"## Migration Summary\\n- The `FROM` command is used to select the `logs-*` index pattern.\\n- The `RENAME` command is used to rename the `host` field to `dest`.\\n- The `WHERE` command filters the rows where `process_exec` contains the pattern `*chown *root*`.\\n- The `STATS` command is used to aggregate the data, counting the number of occurrences and finding the minimum and maximum timestamps, grouped by `process_exec`, `proctitle`, `normalized_proctitle_delimiter`, and `dest`.\\n- The macros `security_content_ctime` and `linux_auditd_change_file_owner_to_root_filter` are placeholders for the corresponding Splunk macros.\",\r\n \"Additional comment 2.0\"\r\n ],\r\n \"translation_result\": \"full\",\r\n \"id\": \"{{rule_migration_id_1}}\"\r\n },\r\n {\r\n \"created_by\": \"elastic2.0\",\r\n \"elastic_rule\": {\r\n \"severity\": \"high\",\r\n \"title\": \"Linux Auditd Change File Owner To Root (UPDATED)\"\r\n },\r\n \"id\": \"{{rule_migration_id_2}}\"\r\n }\r\n]'\r\n```\r\n</details>\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"f6ac2cf8603ca633070e719f69b4fcef45ea92cb"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/200815","number":200815,"mergeCommit":{"message":"[Rules migration] Add rules migrations update route (#11209) (#200815)\n\n## Summary\r\n\r\nChanges in this PR:\r\n* Added `update` route to handle bulk rule migrations docs updates\r\n* Exposed `id` field in `RuleMigration` object needed for ES bulk update\r\noperation\r\n* Updated SIEM migrations schemas to use `NonEmptyString` when it is\r\nneeded\r\n\r\n## Testing locally\r\n\r\nEnable the flag\r\n```\r\nxpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']\r\n```\r\n\r\nCreate and start a rule migration. Then use `update` API to updated\r\ncorresponding docs.\r\n\r\ncURL request examples:\r\n\r\n<details>\r\n <summary>Rules migration `create` POST request</summary>\r\n\r\n```\r\ncurl --location --request POST 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \\\r\n--header 'Content-Type: application/json' \\\r\n--data '[\r\n {\r\n \"id\": \"f8c325ea-506e-4105-8ccf-da1492e90115\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Add User Account Type\",\r\n \"description\": \"The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.\",\r\n \"query\": \"sourcetype=\\\"linux:audit\\\" type=ADD_USER \\n| rename hostname as dest \\n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \\n| `security_content_ctime(firstTime)` \\n| `security_content_ctime(lastTime)`\\n| search *\",\r\n \"query_language\":\"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1136\"\r\n ]\r\n },\r\n {\r\n \"id\": \"7b87c556-0ca4-47e0-b84c-6cd62a0a3e90\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Change File Owner To Root\",\r\n \"description\": \"The following analytic detects the use of the '\\''chown'\\'' command to change a file owner to '\\''root'\\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.\",\r\n \"query\": \"`linux_auditd` `linux_auditd_normalized_proctitle_process`\\r\\n| rename host as dest \\r\\n| where LIKE (process_exec, \\\"%chown %root%\\\") \\r\\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \\r\\n| `security_content_ctime(firstTime)` \\r\\n| `security_content_ctime(lastTime)`\\r\\n| `linux_auditd_change_file_owner_to_root_filter`\",\r\n \"query_language\": \"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1222\"\r\n ]\r\n }\r\n]'\r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration `start` task request</summary>\r\n\r\n- Assuming the connector `azureOpenAiGPT4o` is already created in the\r\nlocal environment.\r\n- Using the {{`migration_id`}} from the first POST request response\r\n\r\n```\r\ncurl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/start' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \\\r\n--header 'Content-Type: application/json' \\\r\n--data '{\r\n \"connectorId\": \"azureOpenAiGPT4o\"\r\n}'\r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration rules documents request</summary>\r\n\r\n- Using the {{`migration_id`}} from the first POST request response.\r\n\r\n```\r\ncurl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration `update` PUT request</summary>\r\n\r\n- Using the {{`rule_migration_id_1`}} and {{`rule_migration_id_2`}} from\r\nprevious GET request response\r\n\r\n```\r\ncurl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \r\n--data '[\r\n {\r\n \"comments\": [\r\n \"## Migration Summary\\n- The `FROM` command is used to select the `logs-*` index pattern.\\n- The `RENAME` command is used to rename the `host` field to `dest`.\\n- The `WHERE` command filters the rows where `process_exec` contains the pattern `*chown *root*`.\\n- The `STATS` command is used to aggregate the data, counting the number of occurrences and finding the minimum and maximum timestamps, grouped by `process_exec`, `proctitle`, `normalized_proctitle_delimiter`, and `dest`.\\n- The macros `security_content_ctime` and `linux_auditd_change_file_owner_to_root_filter` are placeholders for the corresponding Splunk macros.\",\r\n \"Additional comment 2.0\"\r\n ],\r\n \"translation_result\": \"full\",\r\n \"id\": \"{{rule_migration_id_1}}\"\r\n },\r\n {\r\n \"created_by\": \"elastic2.0\",\r\n \"elastic_rule\": {\r\n \"severity\": \"high\",\r\n \"title\": \"Linux Auditd Change File Owner To Root (UPDATED)\"\r\n },\r\n \"id\": \"{{rule_migration_id_2}}\"\r\n }\r\n]'\r\n```\r\n</details>\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"f6ac2cf8603ca633070e719f69b4fcef45ea92cb"}}]}] BACKPORT-->	2024-11-20 08:40:07 -06:00
Kibana Machine	017b92f051	[8.x] [SecuritySolution][SIEM migrations] Add macros and lookups support in the API (#199370) (#200644) ... # Backport This will backport the following commits from `main` to `8.x`: - [[SecuritySolution][SIEM migrations] Add macros and lookups support in the API (#199370)](https://github.com/elastic/kibana/pull/199370) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Sergi Massaneda","email":"sergi.massaneda@elastic.co"},"sourceCommit":{"committedDate":"2024-11-18T19:47:32Z","message":"[SecuritySolution][SIEM migrations] Add macros and lookups support in the API (#199370)","sha":"4f3bbe8d30a962ddb4e9cd5c2d207dabaa063ffb","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Threat Hunting","backport:prev-minor","v8.18.0"],"title":"[SecuritySolution][SIEM migrations] Add macros and lookups support in the API","number":199370,"url":"https://github.com/elastic/kibana/pull/199370","mergeCommit":{"message":"[SecuritySolution][SIEM migrations] Add macros and lookups support in the API (#199370)","sha":"4f3bbe8d30a962ddb4e9cd5c2d207dabaa063ffb"}},"sourceBranch":"main","suggestedTargetBranches":["8.18"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/199370","number":199370,"mergeCommit":{"message":"[SecuritySolution][SIEM migrations] Add macros and lookups support in the API (#199370)","sha":"4f3bbe8d30a962ddb4e9cd5c2d207dabaa063ffb"}},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Sergi Massaneda <sergi.massaneda@elastic.co>	2024-11-18 16:17:47 -06:00
Pablo Machado	c8d1228ff8	[8.x] [SecuritySolution] Check user permissions before initialising entity engine (#198661) (#199162) ... # Backport This will backport the following commits from `main` to `8.x`: - [[SecuritySolution] Check user permissions before initialising entity engine (#198661)](https://github.com/elastic/kibana/pull/198661) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Pablo Machado","email":"pablo.nevesmachado@elastic.co"},"sourceCommit":{"committedDate":"2024-11-06T10:23:30Z","message":"[SecuritySolution] Check user permissions before initialising entity engine (#198661)\n\n## Summary\r\n\r\n* Create privileges API for the Entity Store\r\n* Create missing privileges callout\r\n* Add missing Entity Store privileges callout to Entity Store \r\n* Add missing Entity Store privileges callout to Dashboard\r\n\r\n\r\n![Screenshot 2024-11-04 at 16 16\r\n03](30cdb096-24cd-4a1c-a20b-abbbece865d7\r\n\r\n### Update:\r\n\r\nI added a \"Line clamp\" and \"Read More\" button as requested by Mark:\r\n\r\n\r\n\r\n### Checklist\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [x] Any UI touched in this PR is usable by keyboard only (learn more\r\nabout [keyboard accessibility](https://webaim.org/techniques/keyboard/))","sha":"0e3b83b595906b42fc386e19451759399ee3e74e","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement","v9.0.0","Team: SecuritySolution","Theme: entity_analytics","Feature:Entity Analytics","Team:Entity Analytics","backport:version","v8.18.0"],"number":198661,"url":"https://github.com/elastic/kibana/pull/198661","mergeCommit":{"message":"[SecuritySolution] Check user permissions before initialising entity engine (#198661)\n\n## Summary\r\n\r\n* Create privileges API for the Entity Store\r\n* Create missing privileges callout\r\n* Add missing Entity Store privileges callout to Entity Store \r\n* Add missing Entity Store privileges callout to Dashboard\r\n\r\n\r\n![Screenshot 2024-11-04 at 16 16\r\n03](30cdb096-24cd-4a1c-a20b-abbbece865d7\r\n\r\n### Update:\r\n\r\nI added a \"Line clamp\" and \"Read More\" button as requested by Mark:\r\n\r\n\r\n\r\n### Checklist\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [x] Any UI touched in this PR is usable by keyboard only (learn more\r\nabout [keyboard accessibility](https://webaim.org/techniques/keyboard/))","sha":"0e3b83b595906b42fc386e19451759399ee3e74e"}},"sourceBranch":"main","suggestedTargetBranches":["8.18"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/198661","number":198661,"mergeCommit":{"message":"[SecuritySolution] Check user permissions before initialising entity engine (#198661)\n\n## Summary\r\n\r\n* Create privileges API for the Entity Store\r\n* Create missing privileges callout\r\n* Add missing Entity Store privileges callout to Entity Store \r\n* Add missing Entity Store privileges callout to Dashboard\r\n\r\n\r\n![Screenshot 2024-11-04 at 16 16\r\n03](30cdb096-24cd-4a1c-a20b-abbbece865d7\r\n\r\n### Update:\r\n\r\nI added a \"Line clamp\" and \"Read More\" button as requested by Mark:\r\n\r\n\r\n\r\n### Checklist\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [x] Any UI touched in this PR is usable by keyboard only (learn more\r\nabout [keyboard accessibility](https://webaim.org/techniques/keyboard/))","sha":"0e3b83b595906b42fc386e19451759399ee3e74e"}},{"branch":"8.18","label":"v8.18.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->	2024-11-07 10:05:49 -06:00
Sergi Massaneda	de6da8aa28	[8.x] [SecuritySolution][SIEM migrations] Implement background task API (#197997) (#199209) ... # Backport This will backport the following commits from `main` to `8.x`: - [[SecuritySolution][SIEM migrations] Implement background task API (#197997)](https://github.com/elastic/kibana/pull/197997) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Sergi Massaneda","email":"sergi.massaneda@elastic.co"},"sourceCommit":{"committedDate":"2024-11-06T17:25:24Z","message":"[SecuritySolution][SIEM migrations] Implement background task API (#197997)\n\n## Summary\r\n\r\nIt implements the background task to execute the rule migrations and the\r\nAPI to manage them. It also contains a basic implementation of the\r\nlangGraph agent workflow that will perform the migration using\r\ngenerative AI.\r\n\r\n> [!NOTE] \r\n> This feature needs `siemMigrationsEnabled` experimental flag enabled\r\nto work. Otherwise, the new API routes won't be registered, and the\r\n`SiemRuleMigrationsService` _setup_ won't be called. So no migration\r\ntask code can be reached, and no data stream/template will be installed\r\nto ES.\r\n\r\n### The rule migration task implementation:\r\n\r\n- Retrieve a batch of N rule migration documents (50 rules initially, we\r\nmay change that later) with `status: pending`.\r\n- Update those documents to `status: processing`.\r\n- Execute the migration for each of the N migrations in parallel.\r\n- If there is any error update the document with `status: error`.\r\n- For each rule migration that finishes we set the result to the\r\nstorage, and also update `status: finished`.\r\n- When all the batch of rules is finished the task will check if there\r\nare still migration documents with `status: pending` if so it will\r\nprocess the next batch with a delay (10 seconds initially, we may change\r\nthat later).\r\n- If the task is stopped (via API call or server shut-down), we do a\r\nbulk update for all the `status: processing` documents back to `status:\r\npending`.\r\n\r\n### Task API\r\n\r\n- `POST /internal/siem_migrations/rules` (implemented\r\n[here](https://github.com/elastic/security-team/issues/10654)) ->\r\nCreates the migration on the backend and stores the original rules. It\r\nreturns the `migration_id`\r\n- `GET /internal/siem_migrations/rules/stats` -> Retrieves the stats for\r\nall the existing migrations, aggregated by `migration_id`.\r\n- `GET /internal/siem_migrations/rules/{migration_id}` -> Retrieves all\r\nthe migration rule documents of a specific migration.\r\n- `PUT /internal/siem_migrations/rules/{migration_id}/start` -> Starts\r\nthe background task for a specific migration.\r\n- `GET /internal/siem_migrations/rules/{migration_id}/stats` ->\r\nRetrieves the stats of a specific migration task. The UI will do polling\r\nto this endpoint.\r\n- `PUT /internal/siem_migrations/rules/{migration_id}/stop` -> Stops the\r\nexecution of a specific migration running task. When a migration is\r\nstopped, the executing task is aborted and all the rules in the batch\r\nbeing processed are moved back to pending, all finished rules will\r\nremain stored. When the Kibana server shuts down all the running\r\nmigrations are stopped automatically. To resume the migration we can\r\ncall `{migration_id}/start` again and it will take it from the same\r\nrules batch it was left.\r\n\r\n#### Stats (UI polling) response example:\r\n```\r\n{\r\n \"status\": \"running\",\r\n \"rules\": {\r\n \"total\": 34,\r\n \"finished\": 20,\r\n \"pending\": 4,\r\n \"processing\": 10,\r\n \"failed\": 0\r\n },\r\n \"last_updated_at\": \"2024-10-29T15:04:49.618Z\"\r\n}\r\n```\r\n\r\n### LLM agent Graph\r\n\r\nThe initial implementation of the agent graph that is executed per rule:\r\n\r\n\r\n\r\nThe first node tries to match the original rule with an Elastic prebuilt\r\nrule. If it does not succeed, the second node will try to translate the\r\nquery as a custom rule using the ES|QL knowledge base, this composes\r\nprevious PoCs:\r\n- https://github.com/elastic/kibana/pull/193900\r\n- https://github.com/elastic/kibana/pull/196651\r\n\r\n\r\n\r\n## Testing locally\r\n\r\nEnable the flag\r\n```\r\nxpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']\r\n```\r\n\r\ncURL request examples:\r\n\r\n<details>\r\n <summary>Rules migration `create` POST request</summary>\r\n\r\n```\r\ncurl --location --request POST 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \\\r\n--header 'Content-Type: application/json' \\\r\n--data '[\r\n {\r\n \"id\": \"f8c325ea-506e-4105-8ccf-da1492e90115\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Add User Account Type\",\r\n \"description\": \"The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.\",\r\n \"query\": \"sourcetype=\\\"linux:audit\\\" type=ADD_USER \\n| rename hostname as dest \\n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \\n| `security_content_ctime(firstTime)` \\n| `security_content_ctime(lastTime)`\\n| search *\",\r\n \"query_language\":\"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1136\"\r\n ]\r\n },\r\n {\r\n \"id\": \"7b87c556-0ca4-47e0-b84c-6cd62a0a3e90\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Change File Owner To Root\",\r\n \"description\": \"The following analytic detects the use of the '\\''chown'\\'' command to change a file owner to '\\''root'\\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.\",\r\n \"query\": \"`linux_auditd` `linux_auditd_normalized_proctitle_process`\\r\\n| rename host as dest \\r\\n| where LIKE (process_exec, \\\"%chown %root%\\\") \\r\\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \\r\\n| `security_content_ctime(firstTime)` \\r\\n| `security_content_ctime(lastTime)`\\r\\n| `linux_auditd_change_file_owner_to_root_filter`\",\r\n \"query_language\": \"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1222\"\r\n ]\r\n }\r\n]'\r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration `start` task request</summary>\r\n\r\n- Assuming the connector `azureOpenAiGPT4o` is already created in the\r\nlocal environment.\r\n- Using the {{`migration_id`}} from the first POST request response\r\n\r\n```\r\ncurl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/start' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \\\r\n--header 'Content-Type: application/json' \\\r\n--data '{\r\n \"connectorId\": \"azureOpenAiGPT4o\"\r\n}'\r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration `stop` task request</summary>\r\n\r\n- Using the {{`migration_id`}} from the first POST request response.\r\n\r\n```\r\ncurl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/stop' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \r\n```\r\n</details>\r\n\r\n\r\n<details>\r\n <summary>Rules migration task `stats` request</summary>\r\n\r\n- Using the {{`migration_id`}} from the first POST request response.\r\n\r\n```\r\ncurl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/stats' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration rules documents request</summary>\r\n\r\n- Using the {{`migration_id`}} from the first POST request response.\r\n\r\n```\r\ncurl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration all stats request</summary>\r\n\r\n```\r\ncurl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/stats' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \r\n```\r\n</details>\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"cc66320e970443cede6b9c9a4ab67fb16062e1a4","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Threat Hunting","Team: SecuritySolution","backport:prev-minor","v8.18.0"],"number":197997,"url":"https://github.com/elastic/kibana/pull/197997","mergeCommit":{"message":"[SecuritySolution][SIEM migrations] Implement background task API (#197997)\n\n## Summary\r\n\r\nIt implements the background task to execute the rule migrations and the\r\nAPI to manage them. It also contains a basic implementation of the\r\nlangGraph agent workflow that will perform the migration using\r\ngenerative AI.\r\n\r\n> [!NOTE] \r\n> This feature needs `siemMigrationsEnabled` experimental flag enabled\r\nto work. Otherwise, the new API routes won't be registered, and the\r\n`SiemRuleMigrationsService` _setup_ won't be called. So no migration\r\ntask code can be reached, and no data stream/template will be installed\r\nto ES.\r\n\r\n### The rule migration task implementation:\r\n\r\n- Retrieve a batch of N rule migration documents (50 rules initially, we\r\nmay change that later) with `status: pending`.\r\n- Update those documents to `status: processing`.\r\n- Execute the migration for each of the N migrations in parallel.\r\n- If there is any error update the document with `status: error`.\r\n- For each rule migration that finishes we set the result to the\r\nstorage, and also update `status: finished`.\r\n- When all the batch of rules is finished the task will check if there\r\nare still migration documents with `status: pending` if so it will\r\nprocess the next batch with a delay (10 seconds initially, we may change\r\nthat later).\r\n- If the task is stopped (via API call or server shut-down), we do a\r\nbulk update for all the `status: processing` documents back to `status:\r\npending`.\r\n\r\n### Task API\r\n\r\n- `POST /internal/siem_migrations/rules` (implemented\r\n[here](https://github.com/elastic/security-team/issues/10654)) ->\r\nCreates the migration on the backend and stores the original rules. It\r\nreturns the `migration_id`\r\n- `GET /internal/siem_migrations/rules/stats` -> Retrieves the stats for\r\nall the existing migrations, aggregated by `migration_id`.\r\n- `GET /internal/siem_migrations/rules/{migration_id}` -> Retrieves all\r\nthe migration rule documents of a specific migration.\r\n- `PUT /internal/siem_migrations/rules/{migration_id}/start` -> Starts\r\nthe background task for a specific migration.\r\n- `GET /internal/siem_migrations/rules/{migration_id}/stats` ->\r\nRetrieves the stats of a specific migration task. The UI will do polling\r\nto this endpoint.\r\n- `PUT /internal/siem_migrations/rules/{migration_id}/stop` -> Stops the\r\nexecution of a specific migration running task. When a migration is\r\nstopped, the executing task is aborted and all the rules in the batch\r\nbeing processed are moved back to pending, all finished rules will\r\nremain stored. When the Kibana server shuts down all the running\r\nmigrations are stopped automatically. To resume the migration we can\r\ncall `{migration_id}/start` again and it will take it from the same\r\nrules batch it was left.\r\n\r\n#### Stats (UI polling) response example:\r\n```\r\n{\r\n \"status\": \"running\",\r\n \"rules\": {\r\n \"total\": 34,\r\n \"finished\": 20,\r\n \"pending\": 4,\r\n \"processing\": 10,\r\n \"failed\": 0\r\n },\r\n \"last_updated_at\": \"2024-10-29T15:04:49.618Z\"\r\n}\r\n```\r\n\r\n### LLM agent Graph\r\n\r\nThe initial implementation of the agent graph that is executed per rule:\r\n\r\n\r\n\r\nThe first node tries to match the original rule with an Elastic prebuilt\r\nrule. If it does not succeed, the second node will try to translate the\r\nquery as a custom rule using the ES|QL knowledge base, this composes\r\nprevious PoCs:\r\n- https://github.com/elastic/kibana/pull/193900\r\n- https://github.com/elastic/kibana/pull/196651\r\n\r\n\r\n\r\n## Testing locally\r\n\r\nEnable the flag\r\n```\r\nxpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']\r\n```\r\n\r\ncURL request examples:\r\n\r\n<details>\r\n <summary>Rules migration `create` POST request</summary>\r\n\r\n```\r\ncurl --location --request POST 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \\\r\n--header 'Content-Type: application/json' \\\r\n--data '[\r\n {\r\n \"id\": \"f8c325ea-506e-4105-8ccf-da1492e90115\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Add User Account Type\",\r\n \"description\": \"The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.\",\r\n \"query\": \"sourcetype=\\\"linux:audit\\\" type=ADD_USER \\n| rename hostname as dest \\n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \\n| `security_content_ctime(firstTime)` \\n| `security_content_ctime(lastTime)`\\n| search *\",\r\n \"query_language\":\"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1136\"\r\n ]\r\n },\r\n {\r\n \"id\": \"7b87c556-0ca4-47e0-b84c-6cd62a0a3e90\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Change File Owner To Root\",\r\n \"description\": \"The following analytic detects the use of the '\\''chown'\\'' command to change a file owner to '\\''root'\\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.\",\r\n \"query\": \"`linux_auditd` `linux_auditd_normalized_proctitle_process`\\r\\n| rename host as dest \\r\\n| where LIKE (process_exec, \\\"%chown %root%\\\") \\r\\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \\r\\n| `security_content_ctime(firstTime)` \\r\\n| `security_content_ctime(lastTime)`\\r\\n| `linux_auditd_change_file_owner_to_root_filter`\",\r\n \"query_language\": \"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1222\"\r\n ]\r\n }\r\n]'\r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration `start` task request</summary>\r\n\r\n- Assuming the connector `azureOpenAiGPT4o` is already created in the\r\nlocal environment.\r\n- Using the {{`migration_id`}} from the first POST request response\r\n\r\n```\r\ncurl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/start' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \\\r\n--header 'Content-Type: application/json' \\\r\n--data '{\r\n \"connectorId\": \"azureOpenAiGPT4o\"\r\n}'\r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration `stop` task request</summary>\r\n\r\n- Using the {{`migration_id`}} from the first POST request response.\r\n\r\n```\r\ncurl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/stop' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \r\n```\r\n</details>\r\n\r\n\r\n<details>\r\n <summary>Rules migration task `stats` request</summary>\r\n\r\n- Using the {{`migration_id`}} from the first POST request response.\r\n\r\n```\r\ncurl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/stats' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration rules documents request</summary>\r\n\r\n- Using the {{`migration_id`}} from the first POST request response.\r\n\r\n```\r\ncurl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration all stats request</summary>\r\n\r\n```\r\ncurl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/stats' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \r\n```\r\n</details>\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"cc66320e970443cede6b9c9a4ab67fb16062e1a4"}},"sourceBranch":"main","suggestedTargetBranches":["8.18"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/197997","number":197997,"mergeCommit":{"message":"[SecuritySolution][SIEM migrations] Implement background task API (#197997)\n\n## Summary\r\n\r\nIt implements the background task to execute the rule migrations and the\r\nAPI to manage them. It also contains a basic implementation of the\r\nlangGraph agent workflow that will perform the migration using\r\ngenerative AI.\r\n\r\n> [!NOTE] \r\n> This feature needs `siemMigrationsEnabled` experimental flag enabled\r\nto work. Otherwise, the new API routes won't be registered, and the\r\n`SiemRuleMigrationsService` _setup_ won't be called. So no migration\r\ntask code can be reached, and no data stream/template will be installed\r\nto ES.\r\n\r\n### The rule migration task implementation:\r\n\r\n- Retrieve a batch of N rule migration documents (50 rules initially, we\r\nmay change that later) with `status: pending`.\r\n- Update those documents to `status: processing`.\r\n- Execute the migration for each of the N migrations in parallel.\r\n- If there is any error update the document with `status: error`.\r\n- For each rule migration that finishes we set the result to the\r\nstorage, and also update `status: finished`.\r\n- When all the batch of rules is finished the task will check if there\r\nare still migration documents with `status: pending` if so it will\r\nprocess the next batch with a delay (10 seconds initially, we may change\r\nthat later).\r\n- If the task is stopped (via API call or server shut-down), we do a\r\nbulk update for all the `status: processing` documents back to `status:\r\npending`.\r\n\r\n### Task API\r\n\r\n- `POST /internal/siem_migrations/rules` (implemented\r\n[here](https://github.com/elastic/security-team/issues/10654)) ->\r\nCreates the migration on the backend and stores the original rules. It\r\nreturns the `migration_id`\r\n- `GET /internal/siem_migrations/rules/stats` -> Retrieves the stats for\r\nall the existing migrations, aggregated by `migration_id`.\r\n- `GET /internal/siem_migrations/rules/{migration_id}` -> Retrieves all\r\nthe migration rule documents of a specific migration.\r\n- `PUT /internal/siem_migrations/rules/{migration_id}/start` -> Starts\r\nthe background task for a specific migration.\r\n- `GET /internal/siem_migrations/rules/{migration_id}/stats` ->\r\nRetrieves the stats of a specific migration task. The UI will do polling\r\nto this endpoint.\r\n- `PUT /internal/siem_migrations/rules/{migration_id}/stop` -> Stops the\r\nexecution of a specific migration running task. When a migration is\r\nstopped, the executing task is aborted and all the rules in the batch\r\nbeing processed are moved back to pending, all finished rules will\r\nremain stored. When the Kibana server shuts down all the running\r\nmigrations are stopped automatically. To resume the migration we can\r\ncall `{migration_id}/start` again and it will take it from the same\r\nrules batch it was left.\r\n\r\n#### Stats (UI polling) response example:\r\n```\r\n{\r\n \"status\": \"running\",\r\n \"rules\": {\r\n \"total\": 34,\r\n \"finished\": 20,\r\n \"pending\": 4,\r\n \"processing\": 10,\r\n \"failed\": 0\r\n },\r\n \"last_updated_at\": \"2024-10-29T15:04:49.618Z\"\r\n}\r\n```\r\n\r\n### LLM agent Graph\r\n\r\nThe initial implementation of the agent graph that is executed per rule:\r\n\r\n\r\n\r\nThe first node tries to match the original rule with an Elastic prebuilt\r\nrule. If it does not succeed, the second node will try to translate the\r\nquery as a custom rule using the ES|QL knowledge base, this composes\r\nprevious PoCs:\r\n- https://github.com/elastic/kibana/pull/193900\r\n- https://github.com/elastic/kibana/pull/196651\r\n\r\n\r\n\r\n## Testing locally\r\n\r\nEnable the flag\r\n```\r\nxpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']\r\n```\r\n\r\ncURL request examples:\r\n\r\n<details>\r\n <summary>Rules migration `create` POST request</summary>\r\n\r\n```\r\ncurl --location --request POST 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \\\r\n--header 'Content-Type: application/json' \\\r\n--data '[\r\n {\r\n \"id\": \"f8c325ea-506e-4105-8ccf-da1492e90115\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Add User Account Type\",\r\n \"description\": \"The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.\",\r\n \"query\": \"sourcetype=\\\"linux:audit\\\" type=ADD_USER \\n| rename hostname as dest \\n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \\n| `security_content_ctime(firstTime)` \\n| `security_content_ctime(lastTime)`\\n| search *\",\r\n \"query_language\":\"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1136\"\r\n ]\r\n },\r\n {\r\n \"id\": \"7b87c556-0ca4-47e0-b84c-6cd62a0a3e90\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Change File Owner To Root\",\r\n \"description\": \"The following analytic detects the use of the '\\''chown'\\'' command to change a file owner to '\\''root'\\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.\",\r\n \"query\": \"`linux_auditd` `linux_auditd_normalized_proctitle_process`\\r\\n| rename host as dest \\r\\n| where LIKE (process_exec, \\\"%chown %root%\\\") \\r\\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \\r\\n| `security_content_ctime(firstTime)` \\r\\n| `security_content_ctime(lastTime)`\\r\\n| `linux_auditd_change_file_owner_to_root_filter`\",\r\n \"query_language\": \"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1222\"\r\n ]\r\n }\r\n]'\r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration `start` task request</summary>\r\n\r\n- Assuming the connector `azureOpenAiGPT4o` is already created in the\r\nlocal environment.\r\n- Using the {{`migration_id`}} from the first POST request response\r\n\r\n```\r\ncurl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/start' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \\\r\n--header 'Content-Type: application/json' \\\r\n--data '{\r\n \"connectorId\": \"azureOpenAiGPT4o\"\r\n}'\r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration `stop` task request</summary>\r\n\r\n- Using the {{`migration_id`}} from the first POST request response.\r\n\r\n```\r\ncurl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/stop' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \r\n```\r\n</details>\r\n\r\n\r\n<details>\r\n <summary>Rules migration task `stats` request</summary>\r\n\r\n- Using the {{`migration_id`}} from the first POST request response.\r\n\r\n```\r\ncurl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/stats' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration rules documents request</summary>\r\n\r\n- Using the {{`migration_id`}} from the first POST request response.\r\n\r\n```\r\ncurl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \r\n```\r\n</details>\r\n\r\n<details>\r\n <summary>Rules migration all stats request</summary>\r\n\r\n```\r\ncurl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/stats' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \r\n```\r\n</details>\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"cc66320e970443cede6b9c9a4ab67fb16062e1a4"}},{"branch":"8.18","label":"v8.18.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->	2024-11-07 10:41:13 +00:00
Sergi Massaneda	177854df2a	[8.x] [SecuritySolution][SIEM Migrations] Rule migrations storage (#197032) (#197603) ... # Backport This will backport the following commits from `main` to `8.x`: - [[SecuritySolution][SIEM Migrations] Rule migrations storage (#197032)](https://github.com/elastic/kibana/pull/197032) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Sergi Massaneda","email":"sergi.massaneda@elastic.co"},"sourceCommit":{"committedDate":"2024-10-24T09:56:08Z","message":"[SecuritySolution][SIEM Migrations] Rule migrations storage (#197032)\n\n## Summary\r\n\r\nissue: https://github.com/elastic/security-team/issues/10654?reload=1\r\n\r\nImplements the persistence layer for the rule migrations from other\r\nvendors, as part of the SIEM Rule migrations effort.\r\n\r\n### Changes\r\n\r\n- Schemas created for `SiemRuleMigration` document entity, along with\r\n`ElasticRule` and `OriginalRule`.\r\n\r\n- New API `/internal/siem_migrations/rules` was created:\r\n- `POST` -> Receives an array of (original) rules and stores them with\r\n`status: pending` to be processed. Responds with the `migration_id` that\r\nwill be used to start the migration background task (implementation\r\ndetails here: https://github.com/elastic/security-team/issues/10850).\r\n - `GET` -> (to be implemented later)\r\n\r\n- New `SiemMigrationsService` added to the `securitySolution` route\r\ncontext, to encapsulate all operations related to SIEM migrations (We\r\nstart with _rule_ migrations, but there are more \"kinds\" of SIEM\r\nmigrations in the pipeline: _dashboards_, _saved queries_...). It\r\ncontains:\r\n\r\n- `SiemRuleMigrationsService` to encapsulate all operations related to\r\nSIEM rule migrations.\r\n- `RuleMigrationsDataStream` class to manage the\r\n`.kibana.siem-rule-migrations-<spaceId>` data stream operations using\r\n`DataStreamSpacesAdapter`.\r\n- It exposes a client with abstracted operations that are exposed to the\r\nAPI routes:\r\n- `create`: indexes an array of _SiemRuleMigration_ documents to the\r\ndata stream\r\n- `search`: searches _SiemRuleMigration_ documents by specific terms.\r\n\r\n> [!NOTE] \r\n> Without `siemMigrationsEnabled` experimental flag the new API route\r\nwon't be registered, and the `SiemRuleMigrationsService` _setup_ won't\r\nbe called, so no index/component template will be installed to ES.\r\n\r\n### Testing locally\r\n\r\nEnable the flag\r\n```\r\nxpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']\r\n```\r\n\r\n<details>\r\n <summary>Example curl request</summary>\r\n\r\n```\r\ncurl --location 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \\\r\n--header 'Content-Type: application/json' \\\r\n--data '[\r\n {\r\n \"id\": \"f8c325ea-506e-4105-8ccf-da1492e90115\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Add User Account Type\",\r\n \"description\": \"The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.\",\r\n \"query\": \"sourcetype=\\\"linux:audit\\\" type=ADD_USER \\n| rename hostname as dest \\n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \\n| `security_content_ctime(firstTime)` \\n| `security_content_ctime(lastTime)`\\n| search *\",\r\n \"query_language\":\"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1136\"\r\n ]\r\n },\r\n {\r\n \"id\": \"7b87c556-0ca4-47e0-b84c-6cd62a0a3e90\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Change File Owner To Root\",\r\n \"description\": \"The following analytic detects the use of the '\\''chown'\\'' command to change a file owner to '\\''root'\\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.\",\r\n \"query\": \"`linux_auditd` `linux_auditd_normalized_proctitle_process`\\r\\n| rename host as dest \\r\\n| where LIKE (process_exec, \\\"%chown %root%\\\") \\r\\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \\r\\n| `security_content_ctime(firstTime)` \\r\\n| `security_content_ctime(lastTime)`\\r\\n| `linux_auditd_change_file_owner_to_root_filter`\",\r\n \"query_language\": \"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1222\"\r\n ]\r\n }\r\n]'\r\n```\r\n</details>\r\n\r\nThe newly created documents can be retrieved using Kibana DevTools\r\nconsole:\r\n```\r\nGET .kibana.siem-rule-migrations-default/_search\r\n```\r\n\r\n### Screenshots\r\n\r\n\r\n\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"d7109d67810cedfce1ad2bf9e8fd826b20aee06b","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Threat Hunting","Team: SecuritySolution","backport:prev-minor","8.18 candidate","v8.17.0"],"number":197032,"url":"https://github.com/elastic/kibana/pull/197032","mergeCommit":{"message":"[SecuritySolution][SIEM Migrations] Rule migrations storage (#197032)\n\n## Summary\r\n\r\nissue: https://github.com/elastic/security-team/issues/10654?reload=1\r\n\r\nImplements the persistence layer for the rule migrations from other\r\nvendors, as part of the SIEM Rule migrations effort.\r\n\r\n### Changes\r\n\r\n- Schemas created for `SiemRuleMigration` document entity, along with\r\n`ElasticRule` and `OriginalRule`.\r\n\r\n- New API `/internal/siem_migrations/rules` was created:\r\n- `POST` -> Receives an array of (original) rules and stores them with\r\n`status: pending` to be processed. Responds with the `migration_id` that\r\nwill be used to start the migration background task (implementation\r\ndetails here: https://github.com/elastic/security-team/issues/10850).\r\n - `GET` -> (to be implemented later)\r\n\r\n- New `SiemMigrationsService` added to the `securitySolution` route\r\ncontext, to encapsulate all operations related to SIEM migrations (We\r\nstart with _rule_ migrations, but there are more \"kinds\" of SIEM\r\nmigrations in the pipeline: _dashboards_, _saved queries_...). It\r\ncontains:\r\n\r\n- `SiemRuleMigrationsService` to encapsulate all operations related to\r\nSIEM rule migrations.\r\n- `RuleMigrationsDataStream` class to manage the\r\n`.kibana.siem-rule-migrations-<spaceId>` data stream operations using\r\n`DataStreamSpacesAdapter`.\r\n- It exposes a client with abstracted operations that are exposed to the\r\nAPI routes:\r\n- `create`: indexes an array of _SiemRuleMigration_ documents to the\r\ndata stream\r\n- `search`: searches _SiemRuleMigration_ documents by specific terms.\r\n\r\n> [!NOTE] \r\n> Without `siemMigrationsEnabled` experimental flag the new API route\r\nwon't be registered, and the `SiemRuleMigrationsService` _setup_ won't\r\nbe called, so no index/component template will be installed to ES.\r\n\r\n### Testing locally\r\n\r\nEnable the flag\r\n```\r\nxpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']\r\n```\r\n\r\n<details>\r\n <summary>Example curl request</summary>\r\n\r\n```\r\ncurl --location 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \\\r\n--header 'Content-Type: application/json' \\\r\n--data '[\r\n {\r\n \"id\": \"f8c325ea-506e-4105-8ccf-da1492e90115\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Add User Account Type\",\r\n \"description\": \"The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.\",\r\n \"query\": \"sourcetype=\\\"linux:audit\\\" type=ADD_USER \\n| rename hostname as dest \\n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \\n| `security_content_ctime(firstTime)` \\n| `security_content_ctime(lastTime)`\\n| search *\",\r\n \"query_language\":\"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1136\"\r\n ]\r\n },\r\n {\r\n \"id\": \"7b87c556-0ca4-47e0-b84c-6cd62a0a3e90\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Change File Owner To Root\",\r\n \"description\": \"The following analytic detects the use of the '\\''chown'\\'' command to change a file owner to '\\''root'\\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.\",\r\n \"query\": \"`linux_auditd` `linux_auditd_normalized_proctitle_process`\\r\\n| rename host as dest \\r\\n| where LIKE (process_exec, \\\"%chown %root%\\\") \\r\\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \\r\\n| `security_content_ctime(firstTime)` \\r\\n| `security_content_ctime(lastTime)`\\r\\n| `linux_auditd_change_file_owner_to_root_filter`\",\r\n \"query_language\": \"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1222\"\r\n ]\r\n }\r\n]'\r\n```\r\n</details>\r\n\r\nThe newly created documents can be retrieved using Kibana DevTools\r\nconsole:\r\n```\r\nGET .kibana.siem-rule-migrations-default/_search\r\n```\r\n\r\n### Screenshots\r\n\r\n\r\n\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"d7109d67810cedfce1ad2bf9e8fd826b20aee06b"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/197032","number":197032,"mergeCommit":{"message":"[SecuritySolution][SIEM Migrations] Rule migrations storage (#197032)\n\n## Summary\r\n\r\nissue: https://github.com/elastic/security-team/issues/10654?reload=1\r\n\r\nImplements the persistence layer for the rule migrations from other\r\nvendors, as part of the SIEM Rule migrations effort.\r\n\r\n### Changes\r\n\r\n- Schemas created for `SiemRuleMigration` document entity, along with\r\n`ElasticRule` and `OriginalRule`.\r\n\r\n- New API `/internal/siem_migrations/rules` was created:\r\n- `POST` -> Receives an array of (original) rules and stores them with\r\n`status: pending` to be processed. Responds with the `migration_id` that\r\nwill be used to start the migration background task (implementation\r\ndetails here: https://github.com/elastic/security-team/issues/10850).\r\n - `GET` -> (to be implemented later)\r\n\r\n- New `SiemMigrationsService` added to the `securitySolution` route\r\ncontext, to encapsulate all operations related to SIEM migrations (We\r\nstart with _rule_ migrations, but there are more \"kinds\" of SIEM\r\nmigrations in the pipeline: _dashboards_, _saved queries_...). It\r\ncontains:\r\n\r\n- `SiemRuleMigrationsService` to encapsulate all operations related to\r\nSIEM rule migrations.\r\n- `RuleMigrationsDataStream` class to manage the\r\n`.kibana.siem-rule-migrations-<spaceId>` data stream operations using\r\n`DataStreamSpacesAdapter`.\r\n- It exposes a client with abstracted operations that are exposed to the\r\nAPI routes:\r\n- `create`: indexes an array of _SiemRuleMigration_ documents to the\r\ndata stream\r\n- `search`: searches _SiemRuleMigration_ documents by specific terms.\r\n\r\n> [!NOTE] \r\n> Without `siemMigrationsEnabled` experimental flag the new API route\r\nwon't be registered, and the `SiemRuleMigrationsService` _setup_ won't\r\nbe called, so no index/component template will be installed to ES.\r\n\r\n### Testing locally\r\n\r\nEnable the flag\r\n```\r\nxpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']\r\n```\r\n\r\n<details>\r\n <summary>Example curl request</summary>\r\n\r\n```\r\ncurl --location 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \\\r\n--header 'Content-Type: application/json' \\\r\n--data '[\r\n {\r\n \"id\": \"f8c325ea-506e-4105-8ccf-da1492e90115\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Add User Account Type\",\r\n \"description\": \"The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.\",\r\n \"query\": \"sourcetype=\\\"linux:audit\\\" type=ADD_USER \\n| rename hostname as dest \\n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \\n| `security_content_ctime(firstTime)` \\n| `security_content_ctime(lastTime)`\\n| search *\",\r\n \"query_language\":\"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1136\"\r\n ]\r\n },\r\n {\r\n \"id\": \"7b87c556-0ca4-47e0-b84c-6cd62a0a3e90\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Change File Owner To Root\",\r\n \"description\": \"The following analytic detects the use of the '\\''chown'\\'' command to change a file owner to '\\''root'\\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.\",\r\n \"query\": \"`linux_auditd` `linux_auditd_normalized_proctitle_process`\\r\\n| rename host as dest \\r\\n| where LIKE (process_exec, \\\"%chown %root%\\\") \\r\\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \\r\\n| `security_content_ctime(firstTime)` \\r\\n| `security_content_ctime(lastTime)`\\r\\n| `linux_auditd_change_file_owner_to_root_filter`\",\r\n \"query_language\": \"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1222\"\r\n ]\r\n }\r\n]'\r\n```\r\n</details>\r\n\r\nThe newly created documents can be retrieved using Kibana DevTools\r\nconsole:\r\n```\r\nGET .kibana.siem-rule-migrations-default/_search\r\n```\r\n\r\n### Screenshots\r\n\r\n\r\n\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"d7109d67810cedfce1ad2bf9e8fd826b20aee06b"}},{"branch":"8.x","label":"v8.17.0","labelRegex":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->	2024-10-24 07:34:55 -05:00
Kibana Machine	ff7b33c65a	[8.x] [SecuritySolution] Load entity store indices from security solution data view (#195862) (#196209) ... # Backport This will backport the following commits from `main` to `8.x`: - [[SecuritySolution] Load entity store indices from security solution data view (#195862)](https://github.com/elastic/kibana/pull/195862) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Pablo Machado","email":"pablo.nevesmachado@elastic.co"},"sourceCommit":{"committedDate":"2024-10-14T20:56:58Z","message":"[SecuritySolution] Load entity store indices from security solution data view (#195862)\n\n## Summary\r\n\r\n* Update the Entity Store to retrieve indices from the security solution\r\ndata view.\r\n* Create a new API that updates all installed entity engine indices\r\n(`api/entity_store/engines/apply_dataview_indices`)\r\n\r\n\r\n### How to test it?\r\n* Install the entity store\r\n* Check if the transform index has the security solutions data view\r\nindices\r\n* Call `apply_dataview_indices` API; it should not return changes\r\n* Update the security solution data view indices\r\n* Call `apply_dataview_indices` API and if the API response contains the\r\nupdated indices\r\n* Check if the transform index also got updated\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"489c0901ffd335879d9652424ab15ef9f39cc4cb","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement","v9.0.0","Team: SecuritySolution","backport:prev-minor","Theme: entity_analytics","Feature:Entity Analytics","Team:Entity Analytics"],"title":"[SecuritySolution] Load entity store indices from security solution data view","number":195862,"url":"https://github.com/elastic/kibana/pull/195862","mergeCommit":{"message":"[SecuritySolution] Load entity store indices from security solution data view (#195862)\n\n## Summary\r\n\r\n* Update the Entity Store to retrieve indices from the security solution\r\ndata view.\r\n* Create a new API that updates all installed entity engine indices\r\n(`api/entity_store/engines/apply_dataview_indices`)\r\n\r\n\r\n### How to test it?\r\n* Install the entity store\r\n* Check if the transform index has the security solutions data view\r\nindices\r\n* Call `apply_dataview_indices` API; it should not return changes\r\n* Update the security solution data view indices\r\n* Call `apply_dataview_indices` API and if the API response contains the\r\nupdated indices\r\n* Check if the transform index also got updated\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"489c0901ffd335879d9652424ab15ef9f39cc4cb"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/195862","number":195862,"mergeCommit":{"message":"[SecuritySolution] Load entity store indices from security solution data view (#195862)\n\n## Summary\r\n\r\n* Update the Entity Store to retrieve indices from the security solution\r\ndata view.\r\n* Create a new API that updates all installed entity engine indices\r\n(`api/entity_store/engines/apply_dataview_indices`)\r\n\r\n\r\n### How to test it?\r\n* Install the entity store\r\n* Check if the transform index has the security solutions data view\r\nindices\r\n* Call `apply_dataview_indices` API; it should not return changes\r\n* Update the security solution data view indices\r\n* Call `apply_dataview_indices` API and if the API response contains the\r\nupdated indices\r\n* Check if the transform index also got updated\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"489c0901ffd335879d9652424ab15ef9f39cc4cb"}}]}] BACKPORT--> Co-authored-by: Pablo Machado <pablo.nevesmachado@elastic.co>	2024-10-15 06:14:35 -05:00
Maxim Palenov	d120e66c4b	[8.x] [HTTP/OAS] Include Security Solution domain OAS to production docs (#194132) (#195221) ... # Backport This will backport the following commits from `main` to `8.x`: - [[HTTP/OAS] Include Security Solution domain OAS to production docs (#194132)](https://github.com/elastic/kibana/pull/194132) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Maxim Palenov","email":"maxim.palenov@elastic.co"},"sourceCommit":{"committedDate":"2024-10-04T19:34:25Z","message":"[HTTP/OAS] Include Security Solution domain OAS to production docs (#194132)\n\n*Epic:** https://github.com/elastic/security-team/issues/9401 (internal)\r\n\r\n## Summary\r\n\r\nThis PR includes Security Solution OpenAPI domain bundles into the production OpenAPI Kibana bundle. The result Kibana bundler is expected to be published to Bump.sh manually by @lcawl.","sha":"102297ca151d56c8a7da36c14c72386b4cd225ca","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","docs","Team: SecuritySolution","backport:prev-minor","Feature:OAS","v8.16.0"],"number":194132,"url":"https://github.com/elastic/kibana/pull/194132","mergeCommit":{"message":"[HTTP/OAS] Include Security Solution domain OAS to production docs (#194132)\n\n*Epic:** https://github.com/elastic/security-team/issues/9401 (internal)\r\n\r\n## Summary\r\n\r\nThis PR includes Security Solution OpenAPI domain bundles into the production OpenAPI Kibana bundle. The result Kibana bundler is expected to be published to Bump.sh manually by @lcawl.","sha":"102297ca151d56c8a7da36c14c72386b4cd225ca"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/194132","number":194132,"mergeCommit":{"message":"[HTTP/OAS] Include Security Solution domain OAS to production docs (#194132)\n\n*Epic:** https://github.com/elastic/security-team/issues/9401 (internal)\r\n\r\n## Summary\r\n\r\nThis PR includes Security Solution OpenAPI domain bundles into the production OpenAPI Kibana bundle. The result Kibana bundler is expected to be published to Bump.sh manually by @lcawl.","sha":"102297ca151d56c8a7da36c14c72386b4cd225ca"}},{"branch":"8.x","label":"v8.16.0","labelRegex":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->	2024-10-07 09:29:06 -07:00
Jan Monschke	1186ed4504	[8.x] [Threat Hunting Investigations] Migrate all timeline routes to OpenAPI types (#190238) (#194606) ... # Backport This will backport the following commits from `main` to `8.x`: - [[Threat Hunting Investigations] Migrate all timeline routes to OpenAPI types (#190238)](https://github.com/elastic/kibana/pull/190238) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Jan Monschke","email":"jan.monschke@elastic.co"},"sourceCommit":{"committedDate":"2024-09-30T18:55:23Z","message":"[Threat Hunting Investigations] Migrate all timeline routes to OpenAPI types (#190238)\n\n## Summary\r\n\r\nfixes: https://github.com/elastic/security-team/issues/10235\r\nfixes: https://github.com/elastic/security-team/issues/10237\r\n\r\nThis is the final PR for migrating over all timeline-related schemas and\r\ntypes to the new generated zod schemas from our OpenAPI specs. (see\r\nhttps://github.com/elastic/security-team/issues/10110)\r\nOn top of moving to the new schemas/types, this PR also cleans up usage\r\nof now outdated types.\r\n\r\nI'm aware of the size of this PR but rest assured, the changes are easy\r\nto review and for most teams, only a handful of files need to be\r\nreviewed:\r\n\r\n```markdown\r\n### elastic/security-defend-workflows\r\n\r\n* x-pack/test/security_solution_endpoint/apps/endpoint/endpoint_solution_integrations.ts\r\n\r\n### elastic/security-detection-rule-management\r\n\r\n* x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/get_prebuilt_rules_and_timelines_status/get_prebuilt_rules_and_timelines_status_route.ts\r\n* x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/perform_timelines_installation.ts\r\n\r\n### elastic/security-detections-response\r\n\r\n* x-pack/test/security_solution_cypress/cypress/objects/timeline.ts\r\n\r\n### elastic/security-engineering-productivity\r\n\r\n* x-pack/test/security_solution_cypress/cypress/objects/timeline.ts\r\n* x-pack/test/security_solution_cypress/cypress/tasks/api_calls/timelines.ts\r\n```\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"00789609ad663efffd7b3997ca773fe3ea5511e2","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:skip","v9.0.0","Team:Threat Hunting:Investigations","v8.16.0"],"number":190238,"url":"https://github.com/elastic/kibana/pull/190238","mergeCommit":{"message":"[Threat Hunting Investigations] Migrate all timeline routes to OpenAPI types (#190238)\n\n## Summary\r\n\r\nfixes: https://github.com/elastic/security-team/issues/10235\r\nfixes: https://github.com/elastic/security-team/issues/10237\r\n\r\nThis is the final PR for migrating over all timeline-related schemas and\r\ntypes to the new generated zod schemas from our OpenAPI specs. (see\r\nhttps://github.com/elastic/security-team/issues/10110)\r\nOn top of moving to the new schemas/types, this PR also cleans up usage\r\nof now outdated types.\r\n\r\nI'm aware of the size of this PR but rest assured, the changes are easy\r\nto review and for most teams, only a handful of files need to be\r\nreviewed:\r\n\r\n```markdown\r\n### elastic/security-defend-workflows\r\n\r\n* x-pack/test/security_solution_endpoint/apps/endpoint/endpoint_solution_integrations.ts\r\n\r\n### elastic/security-detection-rule-management\r\n\r\n* x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/get_prebuilt_rules_and_timelines_status/get_prebuilt_rules_and_timelines_status_route.ts\r\n* x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/perform_timelines_installation.ts\r\n\r\n### elastic/security-detections-response\r\n\r\n* x-pack/test/security_solution_cypress/cypress/objects/timeline.ts\r\n\r\n### elastic/security-engineering-productivity\r\n\r\n* x-pack/test/security_solution_cypress/cypress/objects/timeline.ts\r\n* x-pack/test/security_solution_cypress/cypress/tasks/api_calls/timelines.ts\r\n```\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"00789609ad663efffd7b3997ca773fe3ea5511e2"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/190238","number":190238,"mergeCommit":{"message":"[Threat Hunting Investigations] Migrate all timeline routes to OpenAPI types (#190238)\n\n## Summary\r\n\r\nfixes: https://github.com/elastic/security-team/issues/10235\r\nfixes: https://github.com/elastic/security-team/issues/10237\r\n\r\nThis is the final PR for migrating over all timeline-related schemas and\r\ntypes to the new generated zod schemas from our OpenAPI specs. (see\r\nhttps://github.com/elastic/security-team/issues/10110)\r\nOn top of moving to the new schemas/types, this PR also cleans up usage\r\nof now outdated types.\r\n\r\nI'm aware of the size of this PR but rest assured, the changes are easy\r\nto review and for most teams, only a handful of files need to be\r\nreviewed:\r\n\r\n```markdown\r\n### elastic/security-defend-workflows\r\n\r\n* x-pack/test/security_solution_endpoint/apps/endpoint/endpoint_solution_integrations.ts\r\n\r\n### elastic/security-detection-rule-management\r\n\r\n* x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/get_prebuilt_rules_and_timelines_status/get_prebuilt_rules_and_timelines_status_route.ts\r\n* x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/perform_timelines_installation.ts\r\n\r\n### elastic/security-detections-response\r\n\r\n* x-pack/test/security_solution_cypress/cypress/objects/timeline.ts\r\n\r\n### elastic/security-engineering-productivity\r\n\r\n* x-pack/test/security_solution_cypress/cypress/objects/timeline.ts\r\n* x-pack/test/security_solution_cypress/cypress/tasks/api_calls/timelines.ts\r\n```\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"00789609ad663efffd7b3997ca773fe3ea5511e2"}},{"branch":"8.x","label":"v8.16.0","labelRegex":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->	2024-10-01 12:06:41 -05:00
Jan Monschke	45b67b9326	[8.x] [Threat Hunting Investigations] Use OpenAPI types in more timeline routes (#189977) (#194581) ... # Backport This will backport the following commits from `main` to `8.x`: - [[Threat Hunting Investigations] Use OpenAPI types in more timeline routes (#189977)](https://github.com/elastic/kibana/pull/189977) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Jan Monschke","email":"jan.monschke@elastic.co"},"sourceCommit":{"committedDate":"2024-09-24T06:49:44Z","message":"[Threat Hunting Investigations] Use OpenAPI types in more timeline routes (#189977)\n\n## Summary\r\n\r\nFixes: https://github.com/elastic/security-team/issues/10133\r\n\r\nMigrates some timeline routes to use the newly generated OpenAPI types.\r\nThe changes mostly affect pinned event and note routes to keep the\r\nchanges small. Routes that actually accept and return timeline objects\r\nwill come in a next step.\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"caad89426c0e790a54cfb6055f09decc3fd9733b","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:skip","v9.0.0","Team:Threat Hunting:Investigations","v8.16.0"],"number":189977,"url":"https://github.com/elastic/kibana/pull/189977","mergeCommit":{"message":"[Threat Hunting Investigations] Use OpenAPI types in more timeline routes (#189977)\n\n## Summary\r\n\r\nFixes: https://github.com/elastic/security-team/issues/10133\r\n\r\nMigrates some timeline routes to use the newly generated OpenAPI types.\r\nThe changes mostly affect pinned event and note routes to keep the\r\nchanges small. Routes that actually accept and return timeline objects\r\nwill come in a next step.\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"caad89426c0e790a54cfb6055f09decc3fd9733b"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/189977","number":189977,"mergeCommit":{"message":"[Threat Hunting Investigations] Use OpenAPI types in more timeline routes (#189977)\n\n## Summary\r\n\r\nFixes: https://github.com/elastic/security-team/issues/10133\r\n\r\nMigrates some timeline routes to use the newly generated OpenAPI types.\r\nThe changes mostly affect pinned event and note routes to keep the\r\nchanges small. Routes that actually accept and return timeline objects\r\nwill come in a next step.\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"caad89426c0e790a54cfb6055f09decc3fd9733b"}},{"branch":"8.x","label":"v8.16.0","labelRegex":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->	2024-10-01 10:17:51 -05:00
Abhishek Bhatia	906556a881	[8.x] [Entity Analytics] New API endpoint to cleanup the risk engine installation and data (#191843) (#193989) ... # Backport This will backport the following commits from `main` to `8.x`: - [[Entity Analytics] New API endpoint to cleanup the risk engine installation and data (#191843)](https://github.com/elastic/kibana/pull/191843) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Abhishek Bhatia","email":"117628830+abhishekbhatia1710@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-09-25T11:35:24Z","message":"[Entity Analytics] New API endpoint to cleanup the risk engine installation and data (#191843)\n\n## Summary\r\n\r\n1. Create a new public API endpoint : `DELETE\r\n/api/risk_score/engine/dangerously_delete_data`\r\n\r\n\r\n## Test cases Result\r\n\r\n```\r\n PASS x-pack/plugins/security_solution/server/lib/entity_analytics/risk_engine/routes/delete.test.ts (7.017 s)\r\n risk engine cleanup route\r\n invokes the risk engine cleanup route\r\n ✓ should call the router with the correct route and handler (71 ms)\r\n ✓ returns a 200 when cleanup is successful (64 ms)\r\n ✓ returns a 500 when cleanup is unsuccessful (57 ms)\r\n ✓ returns a 500 when cleanup is unsuccessful with multiple errors (53 ms)\r\n when task manager is unavailable\r\n ✓ returns a 400 when task manager is unavailable (55 ms)\r\n when user does not have the required privileges\r\n ✓ returns a 403 when user does not have the required privileges (88 ms)\r\n```\r\n\r\n\r\n### API Responses\r\n\r\n## When multiple errors encountered\r\n```\r\n{\r\n \"risk_engine_cleanup\": false,\r\n \"errors\": [\r\n {\r\n \"seq\": 1,\r\n \"error\": \"resource_not_found_exception\\n\\tRoot causes:\\n\\t\\tresource_not_found_exception: Transform with id [risk_score_latest_transform_default] could not be found\"\r\n },\r\n {\r\n \"seq\": 2,\r\n \"error\": \"index_not_found_exception\\n\\tRoot causes:\\n\\t\\tindex_not_found_exception: no such index [risk-score.risk-score-default]\"\r\n },\r\n {\r\n \"seq\": 3,\r\n \"error\": \"index_template_missing_exception\\n\\tRoot causes:\\n\\t\\tindex_template_missing_exception: index_template [.risk-score.risk-score-default-index-template] missing\"\r\n },\r\n {\r\n \"seq\": 4,\r\n \"error\": \"resource_not_found_exception\\n\\tRoot causes:\\n\\t\\tresource_not_found_exception: .risk-score-mappings\"\r\n }\r\n ],\r\n \"status_code\": 500\r\n}\r\n```\r\n\r\n## Success\r\n\r\n```\r\n{\r\n \"risk_engine_cleanup\": true\r\n}\r\n```\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [x]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n\r\n### OpenAPI spec\r\n\r\n\r\n\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"22f451b30df3f80974b73314c9db4c8f8a509789","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement","v9.0.0","Team: SecuritySolution","backport:prev-minor","Feature:Entity Analytics","Team:Entity Analytics"],"number":191843,"url":"https://github.com/elastic/kibana/pull/191843","mergeCommit":{"message":"[Entity Analytics] New API endpoint to cleanup the risk engine installation and data (#191843)\n\n## Summary\r\n\r\n1. Create a new public API endpoint : `DELETE\r\n/api/risk_score/engine/dangerously_delete_data`\r\n\r\n\r\n## Test cases Result\r\n\r\n```\r\n PASS x-pack/plugins/security_solution/server/lib/entity_analytics/risk_engine/routes/delete.test.ts (7.017 s)\r\n risk engine cleanup route\r\n invokes the risk engine cleanup route\r\n ✓ should call the router with the correct route and handler (71 ms)\r\n ✓ returns a 200 when cleanup is successful (64 ms)\r\n ✓ returns a 500 when cleanup is unsuccessful (57 ms)\r\n ✓ returns a 500 when cleanup is unsuccessful with multiple errors (53 ms)\r\n when task manager is unavailable\r\n ✓ returns a 400 when task manager is unavailable (55 ms)\r\n when user does not have the required privileges\r\n ✓ returns a 403 when user does not have the required privileges (88 ms)\r\n```\r\n\r\n\r\n### API Responses\r\n\r\n## When multiple errors encountered\r\n```\r\n{\r\n \"risk_engine_cleanup\": false,\r\n \"errors\": [\r\n {\r\n \"seq\": 1,\r\n \"error\": \"resource_not_found_exception\\n\\tRoot causes:\\n\\t\\tresource_not_found_exception: Transform with id [risk_score_latest_transform_default] could not be found\"\r\n },\r\n {\r\n \"seq\": 2,\r\n \"error\": \"index_not_found_exception\\n\\tRoot causes:\\n\\t\\tindex_not_found_exception: no such index [risk-score.risk-score-default]\"\r\n },\r\n {\r\n \"seq\": 3,\r\n \"error\": \"index_template_missing_exception\\n\\tRoot causes:\\n\\t\\tindex_template_missing_exception: index_template [.risk-score.risk-score-default-index-template] missing\"\r\n },\r\n {\r\n \"seq\": 4,\r\n \"error\": \"resource_not_found_exception\\n\\tRoot causes:\\n\\t\\tresource_not_found_exception: .risk-score-mappings\"\r\n }\r\n ],\r\n \"status_code\": 500\r\n}\r\n```\r\n\r\n## Success\r\n\r\n```\r\n{\r\n \"risk_engine_cleanup\": true\r\n}\r\n```\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [x]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n\r\n### OpenAPI spec\r\n\r\n\r\n\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"22f451b30df3f80974b73314c9db4c8f8a509789"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/191843","number":191843,"mergeCommit":{"message":"[Entity Analytics] New API endpoint to cleanup the risk engine installation and data (#191843)\n\n## Summary\r\n\r\n1. Create a new public API endpoint : `DELETE\r\n/api/risk_score/engine/dangerously_delete_data`\r\n\r\n\r\n## Test cases Result\r\n\r\n```\r\n PASS x-pack/plugins/security_solution/server/lib/entity_analytics/risk_engine/routes/delete.test.ts (7.017 s)\r\n risk engine cleanup route\r\n invokes the risk engine cleanup route\r\n ✓ should call the router with the correct route and handler (71 ms)\r\n ✓ returns a 200 when cleanup is successful (64 ms)\r\n ✓ returns a 500 when cleanup is unsuccessful (57 ms)\r\n ✓ returns a 500 when cleanup is unsuccessful with multiple errors (53 ms)\r\n when task manager is unavailable\r\n ✓ returns a 400 when task manager is unavailable (55 ms)\r\n when user does not have the required privileges\r\n ✓ returns a 403 when user does not have the required privileges (88 ms)\r\n```\r\n\r\n\r\n### API Responses\r\n\r\n## When multiple errors encountered\r\n```\r\n{\r\n \"risk_engine_cleanup\": false,\r\n \"errors\": [\r\n {\r\n \"seq\": 1,\r\n \"error\": \"resource_not_found_exception\\n\\tRoot causes:\\n\\t\\tresource_not_found_exception: Transform with id [risk_score_latest_transform_default] could not be found\"\r\n },\r\n {\r\n \"seq\": 2,\r\n \"error\": \"index_not_found_exception\\n\\tRoot causes:\\n\\t\\tindex_not_found_exception: no such index [risk-score.risk-score-default]\"\r\n },\r\n {\r\n \"seq\": 3,\r\n \"error\": \"index_template_missing_exception\\n\\tRoot causes:\\n\\t\\tindex_template_missing_exception: index_template [.risk-score.risk-score-default-index-template] missing\"\r\n },\r\n {\r\n \"seq\": 4,\r\n \"error\": \"resource_not_found_exception\\n\\tRoot causes:\\n\\t\\tresource_not_found_exception: .risk-score-mappings\"\r\n }\r\n ],\r\n \"status_code\": 500\r\n}\r\n```\r\n\r\n## Success\r\n\r\n```\r\n{\r\n \"risk_engine_cleanup\": true\r\n}\r\n```\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [x]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n\r\n### OpenAPI spec\r\n\r\n\r\n\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"22f451b30df3f80974b73314c9db4c8f8a509789"}}]}] BACKPORT--> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>	2024-09-26 12:19:56 -05:00
Kibana Machine	ce7477005f	[8.x] [Entity Analytics] [Entity Store] Add basic e2e tests and + tidy API route names (#193161) (#193829) ... # Backport This will backport the following commits from `main` to `8.x`: - [[Entity Analytics] [Entity Store] Add basic e2e tests and + tidy API route names (#193161)](https://github.com/elastic/kibana/pull/193161) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Mark Hopkin","email":"mark.hopkin@elastic.co"},"sourceCommit":{"committedDate":"2024-09-24T08:31:32Z","message":"[Entity Analytics] [Entity Store] Add basic e2e tests and + tidy API route names (#193161)\n\n## Summary\r\n\r\nAdds basic end to end tests for the engine CRUD routes.\r\n\r\nI also noticed there was some inconsistency in the naming of the API\r\nroutes which I have fixed.\r\n\r\n---------\r\n\r\nCo-authored-by: machadoum <pablo.nevesmachado@elastic.co>\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by: Tiago Vila Verde <tiago.vilaverde@elastic.co>","sha":"0b4f8774dd933dbfb9cc7816b2484bf5cc25752e","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team: SecuritySolution","backport:prev-minor","Feature:Entity Analytics","Team:Entity Analytics","v8.16.0"],"title":"[Entity Analytics] [Entity Store] Add basic e2e tests and + tidy API route names","number":193161,"url":"https://github.com/elastic/kibana/pull/193161","mergeCommit":{"message":"[Entity Analytics] [Entity Store] Add basic e2e tests and + tidy API route names (#193161)\n\n## Summary\r\n\r\nAdds basic end to end tests for the engine CRUD routes.\r\n\r\nI also noticed there was some inconsistency in the naming of the API\r\nroutes which I have fixed.\r\n\r\n---------\r\n\r\nCo-authored-by: machadoum <pablo.nevesmachado@elastic.co>\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by: Tiago Vila Verde <tiago.vilaverde@elastic.co>","sha":"0b4f8774dd933dbfb9cc7816b2484bf5cc25752e"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/193161","number":193161,"mergeCommit":{"message":"[Entity Analytics] [Entity Store] Add basic e2e tests and + tidy API route names (#193161)\n\n## Summary\r\n\r\nAdds basic end to end tests for the engine CRUD routes.\r\n\r\nI also noticed there was some inconsistency in the naming of the API\r\nroutes which I have fixed.\r\n\r\n---------\r\n\r\nCo-authored-by: machadoum <pablo.nevesmachado@elastic.co>\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by: Tiago Vila Verde <tiago.vilaverde@elastic.co>","sha":"0b4f8774dd933dbfb9cc7816b2484bf5cc25752e"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Mark Hopkin <mark.hopkin@elastic.co>	2024-09-24 04:58:02 -05:00
Pablo Machado	8a6da9bd78	[8.x] [SecuritySolutions] Create Entity Store 'entities/list' API (#192806) (#193562) ... # Backport This will backport the following commits from `main` to `8.x`: - [[SecuritySolutions] Create Entity Store 'entities/list' API (#192806)](https://github.com/elastic/kibana/pull/192806) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Pablo Machado","email":"pablo.nevesmachado@elastic.co"},"sourceCommit":{"committedDate":"2024-09-19T12:54:53Z","message":"[SecuritySolutions] Create Entity Store 'entities/list' API (#192806)\n\nThis PR introduces the following API routes for listing Entity Store\r\n\"entities\":\r\n\r\n<meta charset=\"utf-8\"><b style=\"font-weight:normal;\"\r\nid=\"docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb\"><div\r\ndir=\"ltr\" style=\"margin-left:-0.75pt;\" align=\"left\">\r\nList Entities | GET /api/entity_store/entities/list\r\n-- | --\r\n</div></b>\r\n\r\nThe PR includes the following:\r\n - The OpenAPI schemas for the route\r\n - The actual Kibana side endpoint\r\n - Add searchEntities function to the `EntityStoreDataClient`\r\n \r\n\r\n### How to test\r\n\r\n1. Add some host/user data\r\n* Easiest is to use\r\n[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)\r\n2. Make sure to add `entityStoreEnabled` under\r\n`xpack.securitySolution.enableExperimental` in your `kibana.dev.yml`\r\n3. In kibana dev tools or your terminal, call the `INIT` route for\r\neither `user` or `host`.\r\n4. You should now see 2 transforms in kibana. Make sure to re-trigger\r\nthem if needed so they process the documents.\r\n5. Call the new API, and it should return entities \r\n\r\n\r\n\r\nImplements https://github.com/elastic/security-team/issues/10517\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"27f5da436b70da1a3743ee99c54d8159918b40de","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["v9.0.0","release_note:feature","backport:prev-minor","Theme: entity_analytics","Feature:Entity Analytics","Team:Entity Analytics","v8.16.0"],"number":192806,"url":"https://github.com/elastic/kibana/pull/192806","mergeCommit":{"message":"[SecuritySolutions] Create Entity Store 'entities/list' API (#192806)\n\nThis PR introduces the following API routes for listing Entity Store\r\n\"entities\":\r\n\r\n<meta charset=\"utf-8\"><b style=\"font-weight:normal;\"\r\nid=\"docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb\"><div\r\ndir=\"ltr\" style=\"margin-left:-0.75pt;\" align=\"left\">\r\nList Entities | GET /api/entity_store/entities/list\r\n-- | --\r\n</div></b>\r\n\r\nThe PR includes the following:\r\n - The OpenAPI schemas for the route\r\n - The actual Kibana side endpoint\r\n - Add searchEntities function to the `EntityStoreDataClient`\r\n \r\n\r\n### How to test\r\n\r\n1. Add some host/user data\r\n* Easiest is to use\r\n[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)\r\n2. Make sure to add `entityStoreEnabled` under\r\n`xpack.securitySolution.enableExperimental` in your `kibana.dev.yml`\r\n3. In kibana dev tools or your terminal, call the `INIT` route for\r\neither `user` or `host`.\r\n4. You should now see 2 transforms in kibana. Make sure to re-trigger\r\nthem if needed so they process the documents.\r\n5. Call the new API, and it should return entities \r\n\r\n\r\n\r\nImplements https://github.com/elastic/security-team/issues/10517\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"27f5da436b70da1a3743ee99c54d8159918b40de"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192806","number":192806,"mergeCommit":{"message":"[SecuritySolutions] Create Entity Store 'entities/list' API (#192806)\n\nThis PR introduces the following API routes for listing Entity Store\r\n\"entities\":\r\n\r\n<meta charset=\"utf-8\"><b style=\"font-weight:normal;\"\r\nid=\"docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb\"><div\r\ndir=\"ltr\" style=\"margin-left:-0.75pt;\" align=\"left\">\r\nList Entities | GET /api/entity_store/entities/list\r\n-- | --\r\n</div></b>\r\n\r\nThe PR includes the following:\r\n - The OpenAPI schemas for the route\r\n - The actual Kibana side endpoint\r\n - Add searchEntities function to the `EntityStoreDataClient`\r\n \r\n\r\n### How to test\r\n\r\n1. Add some host/user data\r\n* Easiest is to use\r\n[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)\r\n2. Make sure to add `entityStoreEnabled` under\r\n`xpack.securitySolution.enableExperimental` in your `kibana.dev.yml`\r\n3. In kibana dev tools or your terminal, call the `INIT` route for\r\neither `user` or `host`.\r\n4. You should now see 2 transforms in kibana. Make sure to re-trigger\r\nthem if needed so they process the documents.\r\n5. Call the new API, and it should return entities \r\n\r\n\r\n\r\nImplements https://github.com/elastic/security-team/issues/10517\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"27f5da436b70da1a3743ee99c54d8159918b40de"}},{"branch":"8.x","label":"v8.16.0","labelRegex":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>	2024-09-23 05:07:53 -05:00
Tiago Vila Verde	3c136e9dc6	[8.x] [Security Solution][Entity Analytics] APIs for Entity Store engine (#191986) (#193567) ... # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution][Entity Analytics] APIs for Entity Store engine (#191986)](https://github.com/elastic/kibana/pull/191986) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Tiago Vila Verde","email":"tiago.vilaverde@elastic.co"},"sourceCommit":{"committedDate":"2024-09-16T15:15:10Z","message":"[Security Solution][Entity Analytics] APIs for Entity Store engine (#191986)\n\nThis PR introduces the following API routes for setting up Entity Store\r\n\"engines\":\r\n\r\n<meta charset=\"utf-8\"><b style=\"font-weight:normal;\"\r\nid=\"docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb\"><div\r\ndir=\"ltr\" style=\"margin-left:-0.75pt;\" align=\"left\">\r\nInitialise Engine | POST /api/entity_store/engines/<entity_type>/init\r\n-- | --\r\nStart Engine | POST /api/entity_store/engines/<entity_type>/start\r\nStop Engine | POST /api/entity_store/engines/<entity_type>/stop\r\nDelete Engine | DELETE /api/entity_store/engines/<entity_type>\r\nGet engine | GET  /api/entity_store/engines/<entity_type>\r\nList Engines | GET /api/entity_store/engines\r\n\r\n</div></b>\r\n\r\nThe PR includes the following:\r\n- Adding the `EntityManager` plugin (see elastic/obs-entities) as a\r\ndependency of the Security Solution\r\n - The OpenAPI schemas for the new routes\r\n - The actual Kibana side endpoints\r\n - A `Saved Object` to track the installed engines\r\n - A new `EntityStoreDataClient`\r\n - A new feature flag `entityStoreEngineRoutesEnabled` \r\n \r\n\r\n### How to test\r\n\r\n1. Add some host/user data\r\n* Easiest is to use\r\n[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)\r\n2. Make sure to add `entityStoreEngineRoutesEnabled` under\r\n`xpack.securitySolution.enableExperimental` in your `kibana.dev.yml`\r\n3. In kibana dev tools or your terminal, call the `INIT` route for\r\neither `user` or `host`.\r\n4. You should now see 2 transforms in kibana. Make sure to re-trigger\r\nthem if needed so they process the documents.\r\n5. Check that new entities have been observed by querying the new\r\nentities index via:\r\n * `GET .entities.v1.latest.ea*/_search`\r\n6. Check the other endpoints are working (`START`, `STOP`, `LIST`, etc)\r\n7. Calling `DELETE` should remove the transforms\r\n\r\n\r\n\r\nImplements https://github.com/elastic/security-team/issues/10230\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"cd964f1229b1fdc919677768dae22cf1c05fa3e2","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["backport:skip","v9.0.0","release_note:feature","Theme: entity_analytics","Feature:Entity Analytics","Team:Entity Analytics"],"number":191986,"url":"https://github.com/elastic/kibana/pull/191986","mergeCommit":{"message":"[Security Solution][Entity Analytics] APIs for Entity Store engine (#191986)\n\nThis PR introduces the following API routes for setting up Entity Store\r\n\"engines\":\r\n\r\n<meta charset=\"utf-8\"><b style=\"font-weight:normal;\"\r\nid=\"docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb\"><div\r\ndir=\"ltr\" style=\"margin-left:-0.75pt;\" align=\"left\">\r\nInitialise Engine | POST /api/entity_store/engines/<entity_type>/init\r\n-- | --\r\nStart Engine | POST /api/entity_store/engines/<entity_type>/start\r\nStop Engine | POST /api/entity_store/engines/<entity_type>/stop\r\nDelete Engine | DELETE /api/entity_store/engines/<entity_type>\r\nGet engine | GET  /api/entity_store/engines/<entity_type>\r\nList Engines | GET /api/entity_store/engines\r\n\r\n</div></b>\r\n\r\nThe PR includes the following:\r\n- Adding the `EntityManager` plugin (see elastic/obs-entities) as a\r\ndependency of the Security Solution\r\n - The OpenAPI schemas for the new routes\r\n - The actual Kibana side endpoints\r\n - A `Saved Object` to track the installed engines\r\n - A new `EntityStoreDataClient`\r\n - A new feature flag `entityStoreEngineRoutesEnabled` \r\n \r\n\r\n### How to test\r\n\r\n1. Add some host/user data\r\n* Easiest is to use\r\n[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)\r\n2. Make sure to add `entityStoreEngineRoutesEnabled` under\r\n`xpack.securitySolution.enableExperimental` in your `kibana.dev.yml`\r\n3. In kibana dev tools or your terminal, call the `INIT` route for\r\neither `user` or `host`.\r\n4. You should now see 2 transforms in kibana. Make sure to re-trigger\r\nthem if needed so they process the documents.\r\n5. Check that new entities have been observed by querying the new\r\nentities index via:\r\n * `GET .entities.v1.latest.ea*/_search`\r\n6. Check the other endpoints are working (`START`, `STOP`, `LIST`, etc)\r\n7. Calling `DELETE` should remove the transforms\r\n\r\n\r\n\r\nImplements https://github.com/elastic/security-team/issues/10230\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"cd964f1229b1fdc919677768dae22cf1c05fa3e2"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/191986","number":191986,"mergeCommit":{"message":"[Security Solution][Entity Analytics] APIs for Entity Store engine (#191986)\n\nThis PR introduces the following API routes for setting up Entity Store\r\n\"engines\":\r\n\r\n<meta charset=\"utf-8\"><b style=\"font-weight:normal;\"\r\nid=\"docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb\"><div\r\ndir=\"ltr\" style=\"margin-left:-0.75pt;\" align=\"left\">\r\nInitialise Engine | POST /api/entity_store/engines/<entity_type>/init\r\n-- | --\r\nStart Engine | POST /api/entity_store/engines/<entity_type>/start\r\nStop Engine | POST /api/entity_store/engines/<entity_type>/stop\r\nDelete Engine | DELETE /api/entity_store/engines/<entity_type>\r\nGet engine | GET  /api/entity_store/engines/<entity_type>\r\nList Engines | GET /api/entity_store/engines\r\n\r\n</div></b>\r\n\r\nThe PR includes the following:\r\n- Adding the `EntityManager` plugin (see elastic/obs-entities) as a\r\ndependency of the Security Solution\r\n - The OpenAPI schemas for the new routes\r\n - The actual Kibana side endpoints\r\n - A `Saved Object` to track the installed engines\r\n - A new `EntityStoreDataClient`\r\n - A new feature flag `entityStoreEngineRoutesEnabled` \r\n \r\n\r\n### How to test\r\n\r\n1. Add some host/user data\r\n* Easiest is to use\r\n[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)\r\n2. Make sure to add `entityStoreEngineRoutesEnabled` under\r\n`xpack.securitySolution.enableExperimental` in your `kibana.dev.yml`\r\n3. In kibana dev tools or your terminal, call the `INIT` route for\r\neither `user` or `host`.\r\n4. You should now see 2 transforms in kibana. Make sure to re-trigger\r\nthem if needed so they process the documents.\r\n5. Check that new entities have been observed by querying the new\r\nentities index via:\r\n * `GET .entities.v1.latest.ea*/_search`\r\n6. Check the other endpoints are working (`START`, `STOP`, `LIST`, etc)\r\n7. Calling `DELETE` should remove the transforms\r\n\r\n\r\n\r\nImplements https://github.com/elastic/security-team/issues/10230\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"cd964f1229b1fdc919677768dae22cf1c05fa3e2"}}]}] BACKPORT--> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>	2024-09-20 16:20:31 +01:00
Kibana Machine	607d71decc	[8.x] [Security Solution][Detection Engine] log ES requests when running rule preview (#191107) (#193425) ... # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution][Detection Engine] log ES requests when running rule preview (#191107)](https://github.com/elastic/kibana/pull/191107) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Vitalii Dmyterko","email":"92328789+vitaliidm@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-09-19T13:45:41Z","message":"[Security Solution][Detection Engine] log ES requests when running rule preview (#191107)\n\n## Summary\r\n\r\n**Status:** works only for **ES|QL and EQL** rule types\r\n\r\nWhen clicking on \"Show Elasticsearch requests, ran during rule\r\nexecutions\" preview would return logged Elasticsearch queries that can\r\nbe used to debug/explore rule execution.\r\nEach rule execution accordion has time rule execution started and its\r\nduration.\r\nUpon opening accordion: it will display ES requests with their\r\ndescription and duration.\r\n\r\n**NOTE**: Only search requests are returned, not the requests that\r\ncreate actual alerts\r\n\r\nFeature flag: **loggingRequestsEnabled**\r\n\r\nOn week Demo([internal\r\nlink](https://drive.google.com/drive/folders/1l-cDhbiMxykNH6BzIxFAnLeibmV9a4Cz))\r\n\r\n### Video demo (older UI)\r\n\r\n\r\nhttps://github.com/user-attachments/assets/26f963da-c528-447c-9efd-350b4d42b52c\r\n\r\n### Up to date UI\r\n\r\n#### UI control\r\n<img width=\"733\" alt=\"Screenshot 2024-09-11 at 12 39 07\"\r\nsrc=\"https://github.com/user-attachments/assets/c2b1304d-6f93-4e8e-92f9-a6a0b53cefc7\">\r\n\r\n#### List of executions and code blocks\r\n<img width=\"770\" alt=\"Screenshot 2024-09-11 at 12 38 23\"\r\nsrc=\"https://github.com/user-attachments/assets/48b5aa12-174c-46f5-b0bc-a141833b225b\">\r\n\r\n\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n\r\n🎉 All tests passed! -\r\n[kibana-flaky-test-suite-runner#6909](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6909)\r\n[✅] [Serverless] Security Solution Detection Engine - Cypress: 100/100\r\ntests passed.\r\n[✅] Security Solution Detection Engine - Cypress: 100/100 tests passed.\r\n\r\nFTR tests -\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6918\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"60176bcffdcbdb75b48823f4783923528797efe0","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["v9.0.0","Team: SecuritySolution","release_note:feature","backport:prev-minor","Team:Detection Engine","8.16 candidate","v8.16.0"],"title":"[Security Solution][Detection Engine] log ES requests when running rule preview","number":191107,"url":"https://github.com/elastic/kibana/pull/191107","mergeCommit":{"message":"[Security Solution][Detection Engine] log ES requests when running rule preview (#191107)\n\n## Summary\r\n\r\n**Status:** works only for **ES|QL and EQL** rule types\r\n\r\nWhen clicking on \"Show Elasticsearch requests, ran during rule\r\nexecutions\" preview would return logged Elasticsearch queries that can\r\nbe used to debug/explore rule execution.\r\nEach rule execution accordion has time rule execution started and its\r\nduration.\r\nUpon opening accordion: it will display ES requests with their\r\ndescription and duration.\r\n\r\n**NOTE**: Only search requests are returned, not the requests that\r\ncreate actual alerts\r\n\r\nFeature flag: **loggingRequestsEnabled**\r\n\r\nOn week Demo([internal\r\nlink](https://drive.google.com/drive/folders/1l-cDhbiMxykNH6BzIxFAnLeibmV9a4Cz))\r\n\r\n### Video demo (older UI)\r\n\r\n\r\nhttps://github.com/user-attachments/assets/26f963da-c528-447c-9efd-350b4d42b52c\r\n\r\n### Up to date UI\r\n\r\n#### UI control\r\n<img width=\"733\" alt=\"Screenshot 2024-09-11 at 12 39 07\"\r\nsrc=\"https://github.com/user-attachments/assets/c2b1304d-6f93-4e8e-92f9-a6a0b53cefc7\">\r\n\r\n#### List of executions and code blocks\r\n<img width=\"770\" alt=\"Screenshot 2024-09-11 at 12 38 23\"\r\nsrc=\"https://github.com/user-attachments/assets/48b5aa12-174c-46f5-b0bc-a141833b225b\">\r\n\r\n\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n\r\n🎉 All tests passed! -\r\n[kibana-flaky-test-suite-runner#6909](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6909)\r\n[✅] [Serverless] Security Solution Detection Engine - Cypress: 100/100\r\ntests passed.\r\n[✅] Security Solution Detection Engine - Cypress: 100/100 tests passed.\r\n\r\nFTR tests -\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6918\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"60176bcffdcbdb75b48823f4783923528797efe0"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/191107","number":191107,"mergeCommit":{"message":"[Security Solution][Detection Engine] log ES requests when running rule preview (#191107)\n\n## Summary\r\n\r\n**Status:** works only for **ES|QL and EQL** rule types\r\n\r\nWhen clicking on \"Show Elasticsearch requests, ran during rule\r\nexecutions\" preview would return logged Elasticsearch queries that can\r\nbe used to debug/explore rule execution.\r\nEach rule execution accordion has time rule execution started and its\r\nduration.\r\nUpon opening accordion: it will display ES requests with their\r\ndescription and duration.\r\n\r\n**NOTE**: Only search requests are returned, not the requests that\r\ncreate actual alerts\r\n\r\nFeature flag: **loggingRequestsEnabled**\r\n\r\nOn week Demo([internal\r\nlink](https://drive.google.com/drive/folders/1l-cDhbiMxykNH6BzIxFAnLeibmV9a4Cz))\r\n\r\n### Video demo (older UI)\r\n\r\n\r\nhttps://github.com/user-attachments/assets/26f963da-c528-447c-9efd-350b4d42b52c\r\n\r\n### Up to date UI\r\n\r\n#### UI control\r\n<img width=\"733\" alt=\"Screenshot 2024-09-11 at 12 39 07\"\r\nsrc=\"https://github.com/user-attachments/assets/c2b1304d-6f93-4e8e-92f9-a6a0b53cefc7\">\r\n\r\n#### List of executions and code blocks\r\n<img width=\"770\" alt=\"Screenshot 2024-09-11 at 12 38 23\"\r\nsrc=\"https://github.com/user-attachments/assets/48b5aa12-174c-46f5-b0bc-a141833b225b\">\r\n\r\n\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n\r\n🎉 All tests passed! -\r\n[kibana-flaky-test-suite-runner#6909](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6909)\r\n[✅] [Serverless] Security Solution Detection Engine - Cypress: 100/100\r\ntests passed.\r\n[✅] Security Solution Detection Engine - Cypress: 100/100 tests passed.\r\n\r\nFTR tests -\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6918\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"60176bcffdcbdb75b48823f4783923528797efe0"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Vitalii Dmyterko <92328789+vitaliidm@users.noreply.github.com>	2024-09-19 10:17:26 -05:00
Marshall Main	3cc7029197	[Security Solution] Quickstart script tooling for Detections and Response (#190634) ... ## Summary Creates CLI script tooling for building data, rules, exceptions, and lists in any (local, cloud, serverless) environment for manual testing. The initial commits here add generated clients for accessing security solution, exceptions, and lists APIs and a placeholder script where those clients are set up for use. See README for more details. Much of the code in this PR is auto-generated clients. The hand written code is intended to be primarily in `quickstart/modules/`, where we can add wrapper code to simplify the process for common test environment setup. For example, `createValueListException` takes an array of items and some metadata and automatically creates a new value list and an exception that references that value list. `/modules/data/` contains functions to generate documents of arbitrary size, and we can add more functions to create various other types of documents. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>	2024-09-06 06:41:57 -07:00