[8.17] [Security Solution] Added concurrency limits and request throttling to prebuilt rule routes (#209551) (#210773)

# Backport

This will backport the following commits from `main` to `8.17`:
- [[Security Solution] Added concurrency limits and request throttling
to prebuilt rule routes
(#209551)](https://github.com/elastic/kibana/pull/209551)

<!--- Backport version: 9.6.4 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Dmitrii
Shevchenko","email":"dmitrii.shevchenko@elastic.co"},"sourceCommit":{"committedDate":"2025-02-11T17:12:03Z","message":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule routes (#209551)\n\n**Resolves:
https://github.com/elastic/kibana/issues/208357**\n**Resolves:
https://github.com/elastic/kibana/issues/208355**\n\n## Summary \n\nTo
prevent possible OOM errors, we need to limit concurrent requests
to\nprebuilt rule routes (see attached tickets for more details).\n\n-
`installation/_perform` and `upgrade/_perform` endpoints\n- Concurrency
is limited to one parallel call. If another call is
made\nsimultaneously, the server responds with 429 Too Many Requests.\n-
On the front end, all rule install and upgrade operations are
retried\nin case of a 429 response. This ensures proper handling when a
user\nclicks multiple times an update or install rule buttons\n\n-
`prebuilt_rules/_bootstrap` endpoint\n- Install prebuilt rules and
endpoint packages sequentially instead of\nin parallel to prevent from
having them both downloaded into memory\nsimultaneously.\n- Added a
30-minute socket timeout to prevent the proxy from closing
the\nconnection while rule installation is in progress.\n- Introduced a
`throttleRequests` wrapper, ensuring the endpoint handler\nis called
only once when multiple concurrent requests are received.\n- The first
request triggers the handler, while subsequent requests wait\nfor the
first one to complete and reuse its result.\n- This prevents costly
prebuilt rule package installation from running\nin parallel.\n- Reusing
the response ensures the frontend correctly invalidates cached\nprebuilt
rule queries. Since concurrent frontend requests should receive\nthe
same installed package information, responding with 421 and using\nthe
retry logic as in cases above is not an option here because the\nsecond
request would receive a package installation skipped response\nleading
to no cache invalidation.\n\n- `installation/_review` and
`upgrade/_review` endpoints\n- Concurrency is limited to one parallel
call. If another call is made\nsimultaneously, the server responds with
429 Too Many Requests.\n- On the front end, all rule install and upgrade
operations are retried\nin case of a 429 response. This ensures proper
handling when a user\nclicks multiple times an update or install rule
buttons","sha":"c5557f33213f699acd9bb656af9166b1449d18f9","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","performance","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","backport:version","v8.18.0","v9.1.0","v8.19.0","v8.17.3"],"title":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule
routes","number":209551,"url":"https://github.com/elastic/kibana/pull/209551","mergeCommit":{"message":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule routes (#209551)\n\n**Resolves:
https://github.com/elastic/kibana/issues/208357**\n**Resolves:
https://github.com/elastic/kibana/issues/208355**\n\n## Summary \n\nTo
prevent possible OOM errors, we need to limit concurrent requests
to\nprebuilt rule routes (see attached tickets for more details).\n\n-
`installation/_perform` and `upgrade/_perform` endpoints\n- Concurrency
is limited to one parallel call. If another call is
made\nsimultaneously, the server responds with 429 Too Many Requests.\n-
On the front end, all rule install and upgrade operations are
retried\nin case of a 429 response. This ensures proper handling when a
user\nclicks multiple times an update or install rule buttons\n\n-
`prebuilt_rules/_bootstrap` endpoint\n- Install prebuilt rules and
endpoint packages sequentially instead of\nin parallel to prevent from
having them both downloaded into memory\nsimultaneously.\n- Added a
30-minute socket timeout to prevent the proxy from closing
the\nconnection while rule installation is in progress.\n- Introduced a
`throttleRequests` wrapper, ensuring the endpoint handler\nis called
only once when multiple concurrent requests are received.\n- The first
request triggers the handler, while subsequent requests wait\nfor the
first one to complete and reuse its result.\n- This prevents costly
prebuilt rule package installation from running\nin parallel.\n- Reusing
the response ensures the frontend correctly invalidates cached\nprebuilt
rule queries. Since concurrent frontend requests should receive\nthe
same installed package information, responding with 421 and using\nthe
retry logic as in cases above is not an option here because the\nsecond
request would receive a package installation skipped response\nleading
to no cache invalidation.\n\n- `installation/_review` and
`upgrade/_review` endpoints\n- Concurrency is limited to one parallel
call. If another call is made\nsimultaneously, the server responds with
429 Too Many Requests.\n- On the front end, all rule install and upgrade
operations are retried\nin case of a 429 response. This ensures proper
handling when a user\nclicks multiple times an update or install rule
buttons","sha":"c5557f33213f699acd9bb656af9166b1449d18f9"}},"sourceBranch":"main","suggestedTargetBranches":["8.17"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/210642","number":210642,"state":"MERGED","mergeCommit":{"sha":"df87081d8a574c6b0bb2d9fc026776502622dc11","message":"[9.0]
[Security Solution] Added concurrency limits and request throttling to
prebuilt rule routes (#209551) (#210642)\n\n# Backport\n\nThis will
backport the following commits from `main` to `9.0`:\n- [[Security
Solution] Added concurrency limits and request throttling\nto prebuilt
rule
routes\n(#209551)](https://github.com/elastic/kibana/pull/209551)\n\n<!---
Backport version: 9.4.3 -->\n\n### Questions ?\nPlease refer to the
[Backport
tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT
[{\"author\":{\"name\":\"Dmitrii\nShevchenko\",\"email\":\"dmitrii.shevchenko@elastic.co\"},\"sourceCommit\":{\"committedDate\":\"2025-02-11T17:12:03Z\",\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com/elastic/kibana/issues/208357**\\n**Resolves:\nhttps://github.com/elastic/kibana/issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\",\"branchLabelMapping\":{\"^v9.1.0$\":\"main\",\"^v8.19.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:fix\",\"performance\",\"v9.0.0\",\"Team:Detections\nand
Resp\",\"Team: SecuritySolution\",\"Team:Detection
Rule\nManagement\",\"Feature:Prebuilt
Detection\nRules\",\"backport:version\",\"v8.18.0\",\"v9.1.0\",\"v8.19.0\",\"v8.17.3\"],\"title\":\"[Security\nSolution]
Added concurrency limits and request throttling to
prebuilt\nrule\nroutes\",\"number\":209551,\"url\":\"https://github.com/elastic/kibana/pull/209551\",\"mergeCommit\":{\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com/elastic/kibana/issues/208357**\\n**Resolves:\nhttps://github.com/elastic/kibana/issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[\"9.0\",\"8.18\",\"8.x\",\"8.17\"],\"targetPullRequestStates\":[{\"branch\":\"9.0\",\"label\":\"v9.0.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.18\",\"label\":\"v8.18.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"main\",\"label\":\"v9.1.0\",\"branchLabelMappingKey\":\"^v9.1.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/209551\",\"number\":209551,\"mergeCommit\":{\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com/elastic/kibana/issues/208357**\\n**Resolves:\nhttps://github.com/elastic/kibana/issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\"}},{\"branch\":\"8.x\",\"label\":\"v8.19.0\",\"branchLabelMappingKey\":\"^v8.19.0$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.17\",\"label\":\"v8.17.3\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"}]}]\nBACKPORT-->\n\nCo-authored-by:
Dmitrii Shevchenko
<dmitrii.shevchenko@elastic.co>"}},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/210640","number":210640,"state":"MERGED","mergeCommit":{"sha":"2bd85b19aa84575ec0745ba2ad24d4c7718d824f","message":"[8.18]
[Security Solution] Added concurrency limits and request throttling to
prebuilt rule routes (#209551) (#210640)\n\n# Backport\n\nThis will
backport the following commits from `main` to `8.18`:\n- [[Security
Solution] Added concurrency limits and request throttling\nto prebuilt
rule
routes\n(#209551)](https://github.com/elastic/kibana/pull/209551)\n\n<!---
Backport version: 9.4.3 -->\n\n### Questions ?\nPlease refer to the
[Backport
tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT
[{\"author\":{\"name\":\"Dmitrii\nShevchenko\",\"email\":\"dmitrii.shevchenko@elastic.co\"},\"sourceCommit\":{\"committedDate\":\"2025-02-11T17:12:03Z\",\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com/elastic/kibana/issues/208357**\\n**Resolves:\nhttps://github.com/elastic/kibana/issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\",\"branchLabelMapping\":{\"^v9.1.0$\":\"main\",\"^v8.19.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:fix\",\"performance\",\"v9.0.0\",\"Team:Detections\nand
Resp\",\"Team: SecuritySolution\",\"Team:Detection
Rule\nManagement\",\"Feature:Prebuilt
Detection\nRules\",\"backport:version\",\"v8.18.0\",\"v9.1.0\",\"v8.19.0\",\"v8.17.3\"],\"title\":\"[Security\nSolution]
Added concurrency limits and request throttling to
prebuilt\nrule\nroutes\",\"number\":209551,\"url\":\"https://github.com/elastic/kibana/pull/209551\",\"mergeCommit\":{\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com/elastic/kibana/issues/208357**\\n**Resolves:\nhttps://github.com/elastic/kibana/issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[\"9.0\",\"8.18\",\"8.x\",\"8.17\"],\"targetPullRequestStates\":[{\"branch\":\"9.0\",\"label\":\"v9.0.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.18\",\"label\":\"v8.18.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"main\",\"label\":\"v9.1.0\",\"branchLabelMappingKey\":\"^v9.1.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/209551\",\"number\":209551,\"mergeCommit\":{\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com/elastic/kibana/issues/208357**\\n**Resolves:\nhttps://github.com/elastic/kibana/issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\"}},{\"branch\":\"8.x\",\"label\":\"v8.19.0\",\"branchLabelMappingKey\":\"^v8.19.0$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.17\",\"label\":\"v8.17.3\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"}]}]\nBACKPORT-->\n\nCo-authored-by:
Dmitrii Shevchenko
<dmitrii.shevchenko@elastic.co>"}},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/209551","number":209551,"mergeCommit":{"message":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule routes (#209551)\n\n**Resolves:
https://github.com/elastic/kibana/issues/208357**\n**Resolves:
https://github.com/elastic/kibana/issues/208355**\n\n## Summary \n\nTo
prevent possible OOM errors, we need to limit concurrent requests
to\nprebuilt rule routes (see attached tickets for more details).\n\n-
`installation/_perform` and `upgrade/_perform` endpoints\n- Concurrency
is limited to one parallel call. If another call is
made\nsimultaneously, the server responds with 429 Too Many Requests.\n-
On the front end, all rule install and upgrade operations are
retried\nin case of a 429 response. This ensures proper handling when a
user\nclicks multiple times an update or install rule buttons\n\n-
`prebuilt_rules/_bootstrap` endpoint\n- Install prebuilt rules and
endpoint packages sequentially instead of\nin parallel to prevent from
having them both downloaded into memory\nsimultaneously.\n- Added a
30-minute socket timeout to prevent the proxy from closing
the\nconnection while rule installation is in progress.\n- Introduced a
`throttleRequests` wrapper, ensuring the endpoint handler\nis called
only once when multiple concurrent requests are received.\n- The first
request triggers the handler, while subsequent requests wait\nfor the
first one to complete and reuse its result.\n- This prevents costly
prebuilt rule package installation from running\nin parallel.\n- Reusing
the response ensures the frontend correctly invalidates cached\nprebuilt
rule queries. Since concurrent frontend requests should receive\nthe
same installed package information, responding with 421 and using\nthe
retry logic as in cases above is not an option here because the\nsecond
request would receive a package installation skipped response\nleading
to no cache invalidation.\n\n- `installation/_review` and
`upgrade/_review` endpoints\n- Concurrency is limited to one parallel
call. If another call is made\nsimultaneously, the server responds with
429 Too Many Requests.\n- On the front end, all rule install and upgrade
operations are retried\nin case of a 429 response. This ensures proper
handling when a user\nclicks multiple times an update or install rule
buttons","sha":"c5557f33213f699acd9bb656af9166b1449d18f9"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/210641","number":210641,"state":"MERGED","mergeCommit":{"sha":"8018c82f7d92b1d4798dea3c0a677bae740d4f8a","message":"[8.x]
[Security Solution] Added concurrency limits and request throttling to
prebuilt rule routes (#209551) (#210641)\n\n# Backport\n\nThis will
backport the following commits from `main` to `8.x`:\n- [[Security
Solution] Added concurrency limits and request throttling\nto prebuilt
rule
routes\n(#209551)](https://github.com/elastic/kibana/pull/209551)\n\n<!---
Backport version: 9.4.3 -->\n\n### Questions ?\nPlease refer to the
[Backport
tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT
[{\"author\":{\"name\":\"Dmitrii\nShevchenko\",\"email\":\"dmitrii.shevchenko@elastic.co\"},\"sourceCommit\":{\"committedDate\":\"2025-02-11T17:12:03Z\",\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com/elastic/kibana/issues/208357**\\n**Resolves:\nhttps://github.com/elastic/kibana/issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\",\"branchLabelMapping\":{\"^v9.1.0$\":\"main\",\"^v8.19.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:fix\",\"performance\",\"v9.0.0\",\"Team:Detections\nand
Resp\",\"Team: SecuritySolution\",\"Team:Detection
Rule\nManagement\",\"Feature:Prebuilt
Detection\nRules\",\"backport:version\",\"v8.18.0\",\"v9.1.0\",\"v8.19.0\",\"v8.17.3\"],\"title\":\"[Security\nSolution]
Added concurrency limits and request throttling to
prebuilt\nrule\nroutes\",\"number\":209551,\"url\":\"https://github.com/elastic/kibana/pull/209551\",\"mergeCommit\":{\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com/elastic/kibana/issues/208357**\\n**Resolves:\nhttps://github.com/elastic/kibana/issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[\"9.0\",\"8.18\",\"8.x\",\"8.17\"],\"targetPullRequestStates\":[{\"branch\":\"9.0\",\"label\":\"v9.0.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.18\",\"label\":\"v8.18.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"main\",\"label\":\"v9.1.0\",\"branchLabelMappingKey\":\"^v9.1.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/209551\",\"number\":209551,\"mergeCommit\":{\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com/elastic/kibana/issues/208357**\\n**Resolves:\nhttps://github.com/elastic/kibana/issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\"}},{\"branch\":\"8.x\",\"label\":\"v8.19.0\",\"branchLabelMappingKey\":\"^v8.19.0$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.17\",\"label\":\"v8.17.3\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"}]}]\nBACKPORT-->\n\nCo-authored-by:
Dmitrii Shevchenko
<dmitrii.shevchenko@elastic.co>"}},{"branch":"8.17","label":"v8.17.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
This commit is contained in:
Dmitrii Shevchenko 2025-02-12 13:07:29 +01:00 committed by GitHub
parent 3014b85ac7
commit 6259401adf
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
21 changed files with 566 additions and 264 deletions

View file

@ -0,0 +1,20 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
const MAX_BACKOFF = 30000;
/**
* Calculates a backoff delay using an exponential growth formula, capped by a
* predefined maximum value.
*
* @param failedAttempts - The number of consecutive failed attempts.
* @returns The calculated backoff delay, in milliseconds.
*/
export const cappedExponentialBackoff = (failedAttempts: number) => {
const backoff = Math.min(1000 * 2 ** failedAttempts, MAX_BACKOFF);
return backoff;
};

View file

@ -0,0 +1,22 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { get } from 'lodash';
/*
Prebuilt rule operations like install and upgrade are rate limited to one at a
time. In most cases it is fine to wait for the other operation to finish using
the retry logic.
429 can be caused by a user clicking multiple times on the install or upgrade
rule buttons and in most cases the operations can be performed in succession
without any conflicts.
*/
export const retryOnRateLimitedError = (failureCount: number, error: unknown) => {
const statusCode = get(error, 'response.status');
return statusCode === 429;
};

View file

@ -6,18 +6,18 @@
*/
import type { UseMutationOptions } from '@tanstack/react-query';
import { useMutation } from '@tanstack/react-query';
import { BOOTSTRAP_PREBUILT_RULES_URL } from '../../../../../common/api/detection_engine';
import type { BootstrapPrebuiltRulesResponse } from '../../../../../common/api/detection_engine/prebuilt_rules/bootstrap_prebuilt_rules/bootstrap_prebuilt_rules.gen';
import { PREBUILT_RULES_PACKAGE_NAME } from '../../../../../common/detection_engine/constants';
import { bootstrapPrebuiltRules } from '../api';
import { useInvalidateFetchPrebuiltRulesInstallReviewQuery } from './prebuilt_rules/use_fetch_prebuilt_rules_install_review_query';
import { useInvalidateFetchPrebuiltRulesStatusQuery } from './prebuilt_rules/use_fetch_prebuilt_rules_status_query';
import { useInvalidateFetchPrebuiltRulesUpgradeReviewQuery } from './prebuilt_rules/use_fetch_prebuilt_rules_upgrade_review_query';
import { BOOTSTRAP_PREBUILT_RULES_URL } from '../../../../../../common/api/detection_engine';
import type { BootstrapPrebuiltRulesResponse } from '../../../../../../common/api/detection_engine/prebuilt_rules/bootstrap_prebuilt_rules/bootstrap_prebuilt_rules.gen';
import { PREBUILT_RULES_PACKAGE_NAME } from '../../../../../../common/detection_engine/constants';
import { bootstrapPrebuiltRules } from '../../api';
import { useInvalidateFetchPrebuiltRulesInstallReviewQuery } from './use_fetch_prebuilt_rules_install_review_query';
import { useInvalidateFetchPrebuiltRulesStatusQuery } from './use_fetch_prebuilt_rules_status_query';
import { useInvalidateFetchPrebuiltRulesUpgradeReviewQuery } from './use_fetch_prebuilt_rules_upgrade_review_query';
export const BOOTSTRAP_PREBUILT_RULES_KEY = ['POST', BOOTSTRAP_PREBUILT_RULES_URL];
export const useBootstrapPrebuiltRulesMutation = (
options?: UseMutationOptions<BootstrapPrebuiltRulesResponse, Error>
options?: UseMutationOptions<BootstrapPrebuiltRulesResponse>
) => {
const invalidatePrePackagedRulesStatus = useInvalidateFetchPrebuiltRulesStatusQuery();
const invalidatePrebuiltRulesInstallReview = useInvalidateFetchPrebuiltRulesInstallReviewQuery();
@ -26,7 +26,7 @@ export const useBootstrapPrebuiltRulesMutation = (
return useMutation(() => bootstrapPrebuiltRules(), {
...options,
mutationKey: BOOTSTRAP_PREBUILT_RULES_KEY,
onSettled: (...args) => {
onSuccess: (...args) => {
const response = args[0];
if (
response?.packages.find((pkg) => pkg.name === PREBUILT_RULES_PACKAGE_NAME)?.status ===
@ -40,8 +40,8 @@ export const useBootstrapPrebuiltRulesMutation = (
invalidatePrebuiltRulesUpdateReview();
}
if (options?.onSettled) {
options.onSettled(...args);
if (options?.onSuccess) {
options.onSuccess(...args);
}
},
});

View file

@ -11,6 +11,8 @@ import { reviewRuleInstall } from '../../api';
import { REVIEW_RULE_INSTALLATION_URL } from '../../../../../../common/api/detection_engine/prebuilt_rules/urls';
import type { ReviewRuleInstallationResponseBody } from '../../../../../../common/api/detection_engine/prebuilt_rules';
import { DEFAULT_QUERY_OPTIONS } from '../constants';
import { retryOnRateLimitedError } from './retry_on_rate_limited_error';
import { cappedExponentialBackoff } from './capped_exponential_backoff';
export const REVIEW_RULE_INSTALLATION_QUERY_KEY = ['POST', REVIEW_RULE_INSTALLATION_URL];
@ -26,6 +28,8 @@ export const useFetchPrebuiltRulesInstallReviewQuery = (
{
...DEFAULT_QUERY_OPTIONS,
...options,
retry: retryOnRateLimitedError,
retryDelay: cappedExponentialBackoff,
}
);
};

View file

@ -11,6 +11,8 @@ import { reviewRuleUpgrade } from '../../api';
import { REVIEW_RULE_UPGRADE_URL } from '../../../../../../common/api/detection_engine/prebuilt_rules/urls';
import type { ReviewRuleUpgradeResponseBody } from '../../../../../../common/api/detection_engine/prebuilt_rules';
import { DEFAULT_QUERY_OPTIONS } from '../constants';
import { retryOnRateLimitedError } from './retry_on_rate_limited_error';
import { cappedExponentialBackoff } from './capped_exponential_backoff';
export const REVIEW_RULE_UPGRADE_QUERY_KEY = ['POST', REVIEW_RULE_UPGRADE_URL];
@ -26,6 +28,8 @@ export const useFetchPrebuiltRulesUpgradeReviewQuery = (
{
...DEFAULT_QUERY_OPTIONS,
...options,
retry: retryOnRateLimitedError,
retryDelay: cappedExponentialBackoff,
}
);
};

View file

@ -8,13 +8,15 @@ import type { UseMutationOptions } from '@tanstack/react-query';
import { useMutation } from '@tanstack/react-query';
import type { PerformRuleInstallationResponseBody } from '../../../../../../common/api/detection_engine/prebuilt_rules';
import { PERFORM_RULE_INSTALLATION_URL } from '../../../../../../common/api/detection_engine/prebuilt_rules/urls';
import { useInvalidateFetchPrebuiltRulesStatusQuery } from './use_fetch_prebuilt_rules_status_query';
import { useInvalidateFindRulesQuery } from '../use_find_rules_query';
import { useInvalidateFetchRuleManagementFiltersQuery } from '../use_fetch_rule_management_filters_query';
import { useInvalidateFetchRulesSnoozeSettingsQuery } from '../use_fetch_rules_snooze_settings_query';
import { useInvalidateFetchPrebuiltRulesInstallReviewQuery } from './use_fetch_prebuilt_rules_install_review_query';
import { performInstallAllRules } from '../../api';
import { useInvalidateFetchCoverageOverviewQuery } from '../use_fetch_coverage_overview_query';
import { useInvalidateFetchRuleManagementFiltersQuery } from '../use_fetch_rule_management_filters_query';
import { useInvalidateFetchRulesSnoozeSettingsQuery } from '../use_fetch_rules_snooze_settings_query';
import { useInvalidateFindRulesQuery } from '../use_find_rules_query';
import { retryOnRateLimitedError } from './retry_on_rate_limited_error';
import { useInvalidateFetchPrebuiltRulesInstallReviewQuery } from './use_fetch_prebuilt_rules_install_review_query';
import { useInvalidateFetchPrebuiltRulesStatusQuery } from './use_fetch_prebuilt_rules_status_query';
import { cappedExponentialBackoff } from './capped_exponential_backoff';
export const PERFORM_ALL_RULES_INSTALLATION_KEY = [
'POST',
@ -23,7 +25,7 @@ export const PERFORM_ALL_RULES_INSTALLATION_KEY = [
];
export const usePerformAllRulesInstallMutation = (
options?: UseMutationOptions<PerformRuleInstallationResponseBody, Error>
options?: UseMutationOptions<PerformRuleInstallationResponseBody>
) => {
const invalidateFindRulesQuery = useInvalidateFindRulesQuery();
const invalidateFetchRulesSnoozeSettings = useInvalidateFetchRulesSnoozeSettingsQuery();
@ -33,7 +35,7 @@ export const usePerformAllRulesInstallMutation = (
const invalidateRuleStatus = useInvalidateFetchPrebuiltRulesStatusQuery();
const invalidateFetchCoverageOverviewQuery = useInvalidateFetchCoverageOverviewQuery();
return useMutation<PerformRuleInstallationResponseBody, Error>(() => performInstallAllRules(), {
return useMutation<PerformRuleInstallationResponseBody>(() => performInstallAllRules(), {
...options,
mutationKey: PERFORM_ALL_RULES_INSTALLATION_KEY,
onSettled: (...args) => {
@ -49,5 +51,7 @@ export const usePerformAllRulesInstallMutation = (
options.onSettled(...args);
}
},
retry: retryOnRateLimitedError,
retryDelay: cappedExponentialBackoff,
});
};

View file

@ -11,15 +11,17 @@ import type {
PerformRuleInstallationResponseBody,
} from '../../../../../../common/api/detection_engine/prebuilt_rules';
import { PERFORM_RULE_INSTALLATION_URL } from '../../../../../../common/api/detection_engine/prebuilt_rules/urls';
import { useInvalidateFetchPrebuiltRulesStatusQuery } from './use_fetch_prebuilt_rules_status_query';
import { useInvalidateFindRulesQuery } from '../use_find_rules_query';
import { useInvalidateFetchRuleManagementFiltersQuery } from '../use_fetch_rule_management_filters_query';
import { useInvalidateFetchRulesSnoozeSettingsQuery } from '../use_fetch_rules_snooze_settings_query';
import { useInvalidateFetchPrebuiltRulesInstallReviewQuery } from './use_fetch_prebuilt_rules_install_review_query';
import type { BulkAction } from '../../api';
import { performInstallSpecificRules } from '../../api';
import { useInvalidateFetchCoverageOverviewQuery } from '../use_fetch_coverage_overview_query';
import { useBulkActionMutation } from '../use_bulk_action_mutation';
import { useInvalidateFetchCoverageOverviewQuery } from '../use_fetch_coverage_overview_query';
import { useInvalidateFetchRuleManagementFiltersQuery } from '../use_fetch_rule_management_filters_query';
import { useInvalidateFetchRulesSnoozeSettingsQuery } from '../use_fetch_rules_snooze_settings_query';
import { useInvalidateFindRulesQuery } from '../use_find_rules_query';
import { retryOnRateLimitedError } from './retry_on_rate_limited_error';
import { useInvalidateFetchPrebuiltRulesInstallReviewQuery } from './use_fetch_prebuilt_rules_install_review_query';
import { useInvalidateFetchPrebuiltRulesStatusQuery } from './use_fetch_prebuilt_rules_status_query';
import { cappedExponentialBackoff } from './capped_exponential_backoff';
export const PERFORM_SPECIFIC_RULES_INSTALLATION_KEY = [
'POST',
@ -35,7 +37,7 @@ export interface UsePerformSpecificRulesInstallParams {
export const usePerformSpecificRulesInstallMutation = (
options?: UseMutationOptions<
PerformRuleInstallationResponseBody,
Error,
unknown,
UsePerformSpecificRulesInstallParams
>
) => {
@ -51,7 +53,7 @@ export const usePerformSpecificRulesInstallMutation = (
return useMutation<
PerformRuleInstallationResponseBody,
Error,
unknown,
UsePerformSpecificRulesInstallParams
>(
(rulesToInstall: UsePerformSpecificRulesInstallParams) =>
@ -81,6 +83,8 @@ export const usePerformSpecificRulesInstallMutation = (
options.onSettled(...args);
}
},
retry: retryOnRateLimitedError,
retryDelay: cappedExponentialBackoff,
}
);
};

View file

@ -12,13 +12,15 @@ import type {
UpgradeSpecificRulesRequest,
} from '../../../../../../common/api/detection_engine/prebuilt_rules';
import { PERFORM_RULE_UPGRADE_URL } from '../../../../../../common/api/detection_engine/prebuilt_rules/urls';
import { useInvalidateFetchPrebuiltRulesStatusQuery } from './use_fetch_prebuilt_rules_status_query';
import { useInvalidateFindRulesQuery } from '../use_find_rules_query';
import { performUpgradeSpecificRules } from '../../api';
import { useInvalidateFetchCoverageOverviewQuery } from '../use_fetch_coverage_overview_query';
import { useInvalidateFetchRuleManagementFiltersQuery } from '../use_fetch_rule_management_filters_query';
import { useInvalidateFetchRulesSnoozeSettingsQuery } from '../use_fetch_rules_snooze_settings_query';
import { performUpgradeSpecificRules } from '../../api';
import { useInvalidateFindRulesQuery } from '../use_find_rules_query';
import { useInvalidateFetchPrebuiltRulesStatusQuery } from './use_fetch_prebuilt_rules_status_query';
import { useInvalidateFetchPrebuiltRulesUpgradeReviewQuery } from './use_fetch_prebuilt_rules_upgrade_review_query';
import { useInvalidateFetchCoverageOverviewQuery } from '../use_fetch_coverage_overview_query';
import { retryOnRateLimitedError } from './retry_on_rate_limited_error';
import { cappedExponentialBackoff } from './capped_exponential_backoff';
export const PERFORM_SPECIFIC_RULES_UPGRADE_KEY = [
'POST',
@ -64,6 +66,8 @@ export const usePerformSpecificRulesUpgradeMutation = (
options.onSettled(...args);
}
},
retry: retryOnRateLimitedError,
retryDelay: cappedExponentialBackoff,
}
);
};

View file

@ -10,7 +10,7 @@ import { useEffect } from 'react';
import {
BOOTSTRAP_PREBUILT_RULES_KEY,
useBootstrapPrebuiltRulesMutation,
} from '../api/hooks/use_bootstrap_prebuilt_rules';
} from '../api/hooks/prebuilt_rules/use_bootstrap_prebuilt_rules';
/**
* Install or upgrade the security packages (endpoint and prebuilt rules)

View file

@ -166,6 +166,8 @@ export const AddPrebuiltRulesTableContextProvider = ({
rules: [{ rule_id: ruleId, version: rule.version }],
enable,
});
} catch {
// Error is handled by the mutation's onError callback, so no need to do anything here
} finally {
setLoadingRules((prev) => prev.filter((id) => id !== ruleId));
}
@ -182,6 +184,8 @@ export const AddPrebuiltRulesTableContextProvider = ({
setLoadingRules((prev) => [...prev, ...rulesToUpgrade.map((r) => r.rule_id)]);
try {
await installSpecificRulesRequest({ rules: rulesToUpgrade, enable });
} catch {
// Error is handled by the mutation's onError callback, so no need to do anything here
} finally {
setLoadingRules((prev) =>
prev.filter((id) => !rulesToUpgrade.some((r) => r.rule_id === id))
@ -197,6 +201,8 @@ export const AddPrebuiltRulesTableContextProvider = ({
setLoadingRules((prev) => [...prev, ...rules.map((r) => r.rule_id)]);
try {
await installAllRulesRequest();
} catch {
// Error is handled by the mutation's onError callback, so no need to do anything here
} finally {
setLoadingRules([]);
setSelectedRules([]);

View file

@ -5,16 +5,11 @@
* 2.0.
*/
import { transformError } from '@kbn/securitysolution-es-utils';
import type { IKibanaResponse } from '@kbn/core/server';
import { BOOTSTRAP_PREBUILT_RULES_URL } from '../../../../../../common/api/detection_engine/prebuilt_rules';
import type { BootstrapPrebuiltRulesResponse } from '../../../../../../common/api/detection_engine/prebuilt_rules/bootstrap_prebuilt_rules/bootstrap_prebuilt_rules.gen';
import type { SecuritySolutionPluginRouter } from '../../../../../types';
import { buildSiemResponse } from '../../../routes/utils';
import {
installEndpointPackage,
installPrebuiltRulesPackage,
} from '../install_prebuilt_rules_and_timelines/install_prebuilt_rules_package';
import { PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS } from '../../constants';
import { bootstrapPrebuiltRulesHandler } from './bootstrap_prebuilt_rules_handler';
import { throttleRequests } from '../../../../../utils/throttle_requests';
export const bootstrapPrebuiltRulesRoute = (router: SecuritySolutionPluginRouter) => {
router.versioned
@ -26,43 +21,17 @@ export const bootstrapPrebuiltRulesRoute = (router: SecuritySolutionPluginRouter
requiredPrivileges: ['securitySolution'],
},
},
options: {
timeout: {
idleSocket: PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS,
},
},
})
.addVersion(
{
version: '1',
validate: {},
},
async (context, _, response): Promise<IKibanaResponse<BootstrapPrebuiltRulesResponse>> => {
const siemResponse = buildSiemResponse(response);
try {
const ctx = await context.resolve(['securitySolution']);
const securityContext = ctx.securitySolution;
const config = securityContext.getConfig();
const results = await Promise.all([
installPrebuiltRulesPackage(config, securityContext),
installEndpointPackage(config, securityContext),
]);
const responseBody: BootstrapPrebuiltRulesResponse = {
packages: results.map((result) => ({
name: result.package.name,
version: result.package.version,
status: result.status,
})),
};
return response.ok({
body: responseBody,
});
} catch (err) {
const error = transformError(err);
return siemResponse.error({
body: error.message,
statusCode: error.statusCode,
});
}
}
throttleRequests(bootstrapPrebuiltRulesHandler)
);
};

View file

@ -0,0 +1,58 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { IKibanaResponse, KibanaRequest, KibanaResponseFactory } from '@kbn/core/server';
import { transformError } from '@kbn/securitysolution-es-utils';
import type { BootstrapPrebuiltRulesResponse } from '../../../../../../common/api/detection_engine/prebuilt_rules/bootstrap_prebuilt_rules/bootstrap_prebuilt_rules.gen';
import type { SecuritySolutionRequestHandlerContext } from '../../../../../types';
import { buildSiemResponse } from '../../../routes/utils';
import {
installEndpointPackage,
installPrebuiltRulesPackage,
} from '../install_prebuilt_rules_and_timelines/install_prebuilt_rules_package';
export const bootstrapPrebuiltRulesHandler = async (
context: SecuritySolutionRequestHandlerContext,
_: KibanaRequest,
response: KibanaResponseFactory
): Promise<IKibanaResponse<BootstrapPrebuiltRulesResponse>> => {
const siemResponse = buildSiemResponse(response);
try {
const ctx = await context.resolve(['securitySolution']);
const securityContext = ctx.securitySolution;
const config = securityContext.getConfig();
const prebuiltRulesResult = await installPrebuiltRulesPackage(config, securityContext);
const endpointResult = await installEndpointPackage(config, securityContext);
const responseBody: BootstrapPrebuiltRulesResponse = {
packages: [
{
name: prebuiltRulesResult.package.name,
version: prebuiltRulesResult.package.version,
status: prebuiltRulesResult.status,
},
{
name: endpointResult.package.name,
version: endpointResult.package.version,
status: endpointResult.status,
},
],
};
return response.ok({
body: responseBody,
});
} catch (err) {
const error = transformError(err);
return siemResponse.error({
body: error.message,
statusCode: error.statusCode,
});
}
};

View file

@ -26,8 +26,12 @@ import { createPrebuiltRules } from '../../logic/rule_objects/create_prebuilt_ru
import { createPrebuiltRuleObjectsClient } from '../../logic/rule_objects/prebuilt_rule_objects_client';
import { fetchRuleVersionsTriad } from '../../logic/rule_versions/fetch_rule_versions_triad';
import { performTimelinesInstallation } from '../../logic/perform_timelines_installation';
import { PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS } from '../../constants';
import {
PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS,
PREBUILT_RULES_OPERATION_CONCURRENCY,
} from '../../constants';
import { getRuleGroups } from '../../model/rule_groups/get_rule_groups';
import { routeLimitedConcurrencyTag } from '../../../../../utils/route_limited_concurrency_tag';
export const performRuleInstallationRoute = (router: SecuritySolutionPluginRouter) => {
router.versioned
@ -40,6 +44,7 @@ export const performRuleInstallationRoute = (router: SecuritySolutionPluginRoute
},
},
options: {
tags: [routeLimitedConcurrencyTag(PREBUILT_RULES_OPERATION_CONCURRENCY)],
timeout: {
idleSocket: PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS,
},

View file

@ -21,11 +21,15 @@ import { createPrebuiltRuleAssetsClient } from '../../logic/rule_assets/prebuilt
import { createPrebuiltRuleObjectsClient } from '../../logic/rule_objects/prebuilt_rule_objects_client';
import { upgradePrebuiltRules } from '../../logic/rule_objects/upgrade_prebuilt_rules';
import { fetchRuleVersionsTriad } from '../../logic/rule_versions/fetch_rule_versions_triad';
import { PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS } from '../../constants';
import {
PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS,
PREBUILT_RULES_OPERATION_CONCURRENCY,
} from '../../constants';
import { getUpgradeableRules } from './get_upgradeable_rules';
import { createModifiedPrebuiltRuleAssets } from './create_upgradeable_rules_payload';
import { getRuleGroups } from '../../model/rule_groups/get_rule_groups';
import type { ConfigType } from '../../../../../config';
import { routeLimitedConcurrencyTag } from '../../../../../utils/route_limited_concurrency_tag';
export const performRuleUpgradeRoute = (
router: SecuritySolutionPluginRouter,
@ -41,6 +45,7 @@ export const performRuleUpgradeRoute = (
},
},
options: {
tags: [routeLimitedConcurrencyTag(PREBUILT_RULES_OPERATION_CONCURRENCY)],
timeout: {
idleSocket: PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS,
},

View file

@ -0,0 +1,71 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { KibanaRequest, KibanaResponseFactory } from '@kbn/core/server';
import { transformError } from '@kbn/securitysolution-es-utils';
import type {
ReviewRuleInstallationResponseBody,
RuleInstallationStatsForReview,
} from '../../../../../../common/api/detection_engine/prebuilt_rules';
import type { SecuritySolutionRequestHandlerContext } from '../../../../../types';
import { buildSiemResponse } from '../../../routes/utils';
import { convertPrebuiltRuleAssetToRuleResponse } from '../../../rule_management/logic/detection_rules_client/converters/convert_prebuilt_rule_asset_to_rule_response';
import { createPrebuiltRuleAssetsClient } from '../../logic/rule_assets/prebuilt_rule_assets_client';
import { createPrebuiltRuleObjectsClient } from '../../logic/rule_objects/prebuilt_rule_objects_client';
import { fetchRuleVersionsTriad } from '../../logic/rule_versions/fetch_rule_versions_triad';
import type { PrebuiltRuleAsset } from '../../model/rule_assets/prebuilt_rule_asset';
import { getRuleGroups } from '../../model/rule_groups/get_rule_groups';
export const reviewRuleInstallationHandler = async (
context: SecuritySolutionRequestHandlerContext,
request: KibanaRequest,
response: KibanaResponseFactory
) => {
const siemResponse = buildSiemResponse(response);
try {
const ctx = await context.resolve(['core', 'alerting']);
const soClient = ctx.core.savedObjects.client;
const rulesClient = await ctx.alerting.getRulesClient();
const ruleAssetsClient = createPrebuiltRuleAssetsClient(soClient);
const ruleObjectsClient = createPrebuiltRuleObjectsClient(rulesClient);
const ruleVersionsMap = await fetchRuleVersionsTriad({
ruleAssetsClient,
ruleObjectsClient,
});
const { installableRules } = getRuleGroups(ruleVersionsMap);
const body: ReviewRuleInstallationResponseBody = {
stats: calculateRuleStats(installableRules),
rules: installableRules.map((prebuiltRuleAsset) =>
convertPrebuiltRuleAssetToRuleResponse(prebuiltRuleAsset)
),
};
return response.ok({ body });
} catch (err) {
const error = transformError(err);
return siemResponse.error({
body: error.message,
statusCode: error.statusCode,
});
}
};
const getAggregatedTags = (rules: PrebuiltRuleAsset[]): string[] => {
const set = new Set<string>(rules.flatMap((rule) => rule.tags || []));
return Array.from(set.values()).sort((a, b) => a.localeCompare(b));
};
const calculateRuleStats = (
rulesToInstall: PrebuiltRuleAsset[]
): RuleInstallationStatsForReview => {
const tagsOfRulesToInstall = getAggregatedTags(rulesToInstall);
return {
num_rules_to_install: rulesToInstall.length,
tags: tagsOfRulesToInstall,
};
};

View file

@ -5,21 +5,14 @@
* 2.0.
*/
import { transformError } from '@kbn/securitysolution-es-utils';
import { REVIEW_RULE_INSTALLATION_URL } from '../../../../../../common/api/detection_engine/prebuilt_rules';
import type {
ReviewRuleInstallationResponseBody,
RuleInstallationStatsForReview,
} from '../../../../../../common/api/detection_engine/prebuilt_rules';
import type { SecuritySolutionPluginRouter } from '../../../../../types';
import { buildSiemResponse } from '../../../routes/utils';
import { createPrebuiltRuleAssetsClient } from '../../logic/rule_assets/prebuilt_rule_assets_client';
import { createPrebuiltRuleObjectsClient } from '../../logic/rule_objects/prebuilt_rule_objects_client';
import { fetchRuleVersionsTriad } from '../../logic/rule_versions/fetch_rule_versions_triad';
import type { PrebuiltRuleAsset } from '../../model/rule_assets/prebuilt_rule_asset';
import { convertPrebuiltRuleAssetToRuleResponse } from '../../../rule_management/logic/detection_rules_client/converters/convert_prebuilt_rule_asset_to_rule_response';
import { PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS } from '../../constants';
import { getRuleGroups } from '../../model/rule_groups/get_rule_groups';
import { routeLimitedConcurrencyTag } from '../../../../../utils/route_limited_concurrency_tag';
import {
PREBUILT_RULES_OPERATION_CONCURRENCY,
PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS,
} from '../../constants';
import { reviewRuleInstallationHandler } from './review_rule_installation_handler';
export const reviewRuleInstallationRoute = (router: SecuritySolutionPluginRouter) => {
router.versioned
@ -32,6 +25,7 @@ export const reviewRuleInstallationRoute = (router: SecuritySolutionPluginRouter
},
},
options: {
tags: [routeLimitedConcurrencyTag(PREBUILT_RULES_OPERATION_CONCURRENCY)],
timeout: {
idleSocket: PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS,
},
@ -42,52 +36,6 @@ export const reviewRuleInstallationRoute = (router: SecuritySolutionPluginRouter
version: '1',
validate: {},
},
async (context, request, response) => {
const siemResponse = buildSiemResponse(response);
try {
const ctx = await context.resolve(['core', 'alerting']);
const soClient = ctx.core.savedObjects.client;
const rulesClient = ctx.alerting.getRulesClient();
const ruleAssetsClient = createPrebuiltRuleAssetsClient(soClient);
const ruleObjectsClient = createPrebuiltRuleObjectsClient(rulesClient);
const ruleVersionsMap = await fetchRuleVersionsTriad({
ruleAssetsClient,
ruleObjectsClient,
});
const { installableRules } = getRuleGroups(ruleVersionsMap);
const body: ReviewRuleInstallationResponseBody = {
stats: calculateRuleStats(installableRules),
rules: installableRules.map((prebuiltRuleAsset) =>
convertPrebuiltRuleAssetToRuleResponse(prebuiltRuleAsset)
),
};
return response.ok({ body });
} catch (err) {
const error = transformError(err);
return siemResponse.error({
body: error.message,
statusCode: error.statusCode,
});
}
}
reviewRuleInstallationHandler
);
};
const getAggregatedTags = (rules: PrebuiltRuleAsset[]): string[] => {
const set = new Set<string>(rules.flatMap((rule) => rule.tags || []));
return Array.from(set.values()).sort((a, b) => a.localeCompare(b));
};
const calculateRuleStats = (
rulesToInstall: PrebuiltRuleAsset[]
): RuleInstallationStatsForReview => {
const tagsOfRulesToInstall = getAggregatedTags(rulesToInstall);
return {
num_rules_to_install: rulesToInstall.length,
tags: tagsOfRulesToInstall,
};
};

View file

@ -0,0 +1,138 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { KibanaRequest, KibanaResponseFactory } from '@kbn/core/server';
import { transformError } from '@kbn/securitysolution-es-utils';
import { pickBy } from 'lodash';
import type { RuleResponse } from '../../../../../../common/api/detection_engine/model/rule_schema';
import type {
ReviewRuleUpgradeResponseBody,
RuleUpgradeInfoForReview,
RuleUpgradeStatsForReview,
ThreeWayDiff,
} from '../../../../../../common/api/detection_engine/prebuilt_rules';
import { ThreeWayDiffOutcome } from '../../../../../../common/api/detection_engine/prebuilt_rules';
import { invariant } from '../../../../../../common/utils/invariant';
import type { SecuritySolutionRequestHandlerContext } from '../../../../../types';
import { buildSiemResponse } from '../../../routes/utils';
import { convertPrebuiltRuleAssetToRuleResponse } from '../../../rule_management/logic/detection_rules_client/converters/convert_prebuilt_rule_asset_to_rule_response';
import type { CalculateRuleDiffResult } from '../../logic/diff/calculate_rule_diff';
import { calculateRuleDiff } from '../../logic/diff/calculate_rule_diff';
import { createPrebuiltRuleAssetsClient } from '../../logic/rule_assets/prebuilt_rule_assets_client';
import { createPrebuiltRuleObjectsClient } from '../../logic/rule_objects/prebuilt_rule_objects_client';
import { fetchRuleVersionsTriad } from '../../logic/rule_versions/fetch_rule_versions_triad';
import { getRuleGroups } from '../../model/rule_groups/get_rule_groups';
export const reviewRuleUpgradeHandler = async (
context: SecuritySolutionRequestHandlerContext,
request: KibanaRequest,
response: KibanaResponseFactory
) => {
const siemResponse = buildSiemResponse(response);
try {
const ctx = await context.resolve(['core', 'alerting']);
const soClient = ctx.core.savedObjects.client;
const rulesClient = await ctx.alerting.getRulesClient();
const ruleAssetsClient = createPrebuiltRuleAssetsClient(soClient);
const ruleObjectsClient = createPrebuiltRuleObjectsClient(rulesClient);
const ruleVersionsMap = await fetchRuleVersionsTriad({
ruleAssetsClient,
ruleObjectsClient,
});
const { upgradeableRules } = getRuleGroups(ruleVersionsMap);
const ruleDiffCalculationResults = upgradeableRules.map(({ current }) => {
const ruleVersions = ruleVersionsMap.get(current.rule_id);
invariant(ruleVersions != null, 'ruleVersions not found');
return calculateRuleDiff(ruleVersions);
});
const body: ReviewRuleUpgradeResponseBody = {
stats: calculateRuleStats(ruleDiffCalculationResults),
rules: calculateRuleInfos(ruleDiffCalculationResults),
};
return response.ok({ body });
} catch (err) {
const error = transformError(err);
return siemResponse.error({
body: error.message,
statusCode: error.statusCode,
});
}
};
const calculateRuleStats = (results: CalculateRuleDiffResult[]): RuleUpgradeStatsForReview => {
const allTags = new Set<string>();
const stats = results.reduce(
(acc, result) => {
acc.num_rules_to_upgrade_total += 1;
if (result.ruleDiff.num_fields_with_conflicts > 0) {
acc.num_rules_with_conflicts += 1;
}
if (result.ruleDiff.num_fields_with_non_solvable_conflicts > 0) {
acc.num_rules_with_non_solvable_conflicts += 1;
}
result.ruleVersions.input.current?.tags.forEach((tag) => allTags.add(tag));
return acc;
},
{
num_rules_to_upgrade_total: 0,
num_rules_with_conflicts: 0,
num_rules_with_non_solvable_conflicts: 0,
}
);
return {
...stats,
tags: Array.from(allTags),
};
};
const calculateRuleInfos = (results: CalculateRuleDiffResult[]): RuleUpgradeInfoForReview[] => {
return results.map((result) => {
const { ruleDiff, ruleVersions } = result;
const installedCurrentVersion = ruleVersions.input.current;
const targetVersion = ruleVersions.input.target;
invariant(installedCurrentVersion != null, 'installedCurrentVersion not found');
invariant(targetVersion != null, 'targetVersion not found');
const targetRule: RuleResponse = {
...convertPrebuiltRuleAssetToRuleResponse(targetVersion),
id: installedCurrentVersion.id,
revision: installedCurrentVersion.revision + 1,
created_at: installedCurrentVersion.created_at,
created_by: installedCurrentVersion.created_by,
updated_at: new Date().toISOString(),
updated_by: installedCurrentVersion.updated_by,
};
return {
id: installedCurrentVersion.id,
rule_id: installedCurrentVersion.rule_id,
revision: installedCurrentVersion.revision,
current_rule: installedCurrentVersion,
target_rule: targetRule,
diff: {
fields: pickBy<ThreeWayDiff<unknown>>(
ruleDiff.fields,
(fieldDiff) =>
fieldDiff.diff_outcome !== ThreeWayDiffOutcome.StockValueNoUpdate &&
fieldDiff.diff_outcome !== ThreeWayDiffOutcome.MissingBaseNoUpdate
),
num_fields_with_updates: ruleDiff.num_fields_with_updates,
num_fields_with_conflicts: ruleDiff.num_fields_with_conflicts,
num_fields_with_non_solvable_conflicts: ruleDiff.num_fields_with_non_solvable_conflicts,
},
};
});
};

View file

@ -5,30 +5,14 @@
* 2.0.
*/
import { transformError } from '@kbn/securitysolution-es-utils';
import { pickBy } from 'lodash';
import {
REVIEW_RULE_UPGRADE_URL,
ThreeWayDiffOutcome,
} from '../../../../../../common/api/detection_engine/prebuilt_rules';
import type {
ReviewRuleUpgradeResponseBody,
RuleUpgradeInfoForReview,
RuleUpgradeStatsForReview,
ThreeWayDiff,
} from '../../../../../../common/api/detection_engine/prebuilt_rules';
import { invariant } from '../../../../../../common/utils/invariant';
import type { RuleResponse } from '../../../../../../common/api/detection_engine/model/rule_schema';
import { REVIEW_RULE_UPGRADE_URL } from '../../../../../../common/api/detection_engine/prebuilt_rules';
import type { SecuritySolutionPluginRouter } from '../../../../../types';
import { buildSiemResponse } from '../../../routes/utils';
import type { CalculateRuleDiffResult } from '../../logic/diff/calculate_rule_diff';
import { calculateRuleDiff } from '../../logic/diff/calculate_rule_diff';
import { createPrebuiltRuleAssetsClient } from '../../logic/rule_assets/prebuilt_rule_assets_client';
import { createPrebuiltRuleObjectsClient } from '../../logic/rule_objects/prebuilt_rule_objects_client';
import { fetchRuleVersionsTriad } from '../../logic/rule_versions/fetch_rule_versions_triad';
import { convertPrebuiltRuleAssetToRuleResponse } from '../../../rule_management/logic/detection_rules_client/converters/convert_prebuilt_rule_asset_to_rule_response';
import { PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS } from '../../constants';
import { getRuleGroups } from '../../model/rule_groups/get_rule_groups';
import { routeLimitedConcurrencyTag } from '../../../../../utils/route_limited_concurrency_tag';
import {
PREBUILT_RULES_OPERATION_CONCURRENCY,
PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS,
} from '../../constants';
import { reviewRuleUpgradeHandler } from './review_rule_upgrade_handler';
export const reviewRuleUpgradeRoute = (router: SecuritySolutionPluginRouter) => {
router.versioned
@ -41,6 +25,7 @@ export const reviewRuleUpgradeRoute = (router: SecuritySolutionPluginRouter) =>
},
},
options: {
tags: [routeLimitedConcurrencyTag(PREBUILT_RULES_OPERATION_CONCURRENCY)],
timeout: {
idleSocket: PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS,
},
@ -51,112 +36,6 @@ export const reviewRuleUpgradeRoute = (router: SecuritySolutionPluginRouter) =>
version: '1',
validate: {},
},
async (context, request, response) => {
const siemResponse = buildSiemResponse(response);
try {
const ctx = await context.resolve(['core', 'alerting']);
const soClient = ctx.core.savedObjects.client;
const rulesClient = ctx.alerting.getRulesClient();
const ruleAssetsClient = createPrebuiltRuleAssetsClient(soClient);
const ruleObjectsClient = createPrebuiltRuleObjectsClient(rulesClient);
const ruleVersionsMap = await fetchRuleVersionsTriad({
ruleAssetsClient,
ruleObjectsClient,
});
const { upgradeableRules } = getRuleGroups(ruleVersionsMap);
const ruleDiffCalculationResults = upgradeableRules.map(({ current }) => {
const ruleVersions = ruleVersionsMap.get(current.rule_id);
invariant(ruleVersions != null, 'ruleVersions not found');
return calculateRuleDiff(ruleVersions);
});
const body: ReviewRuleUpgradeResponseBody = {
stats: calculateRuleStats(ruleDiffCalculationResults),
rules: calculateRuleInfos(ruleDiffCalculationResults),
};
return response.ok({ body });
} catch (err) {
const error = transformError(err);
return siemResponse.error({
body: error.message,
statusCode: error.statusCode,
});
}
}
reviewRuleUpgradeHandler
);
};
const calculateRuleStats = (results: CalculateRuleDiffResult[]): RuleUpgradeStatsForReview => {
const allTags = new Set<string>();
const stats = results.reduce(
(acc, result) => {
acc.num_rules_to_upgrade_total += 1;
if (result.ruleDiff.num_fields_with_conflicts > 0) {
acc.num_rules_with_conflicts += 1;
}
if (result.ruleDiff.num_fields_with_non_solvable_conflicts > 0) {
acc.num_rules_with_non_solvable_conflicts += 1;
}
result.ruleVersions.input.current?.tags.forEach((tag) => allTags.add(tag));
return acc;
},
{
num_rules_to_upgrade_total: 0,
num_rules_with_conflicts: 0,
num_rules_with_non_solvable_conflicts: 0,
}
);
return {
...stats,
tags: Array.from(allTags),
};
};
const calculateRuleInfos = (results: CalculateRuleDiffResult[]): RuleUpgradeInfoForReview[] => {
return results.map((result) => {
const { ruleDiff, ruleVersions } = result;
const installedCurrentVersion = ruleVersions.input.current;
const targetVersion = ruleVersions.input.target;
invariant(installedCurrentVersion != null, 'installedCurrentVersion not found');
invariant(targetVersion != null, 'targetVersion not found');
const targetRule: RuleResponse = {
...convertPrebuiltRuleAssetToRuleResponse(targetVersion),
id: installedCurrentVersion.id,
revision: installedCurrentVersion.revision + 1,
created_at: installedCurrentVersion.created_at,
created_by: installedCurrentVersion.created_by,
updated_at: new Date().toISOString(),
updated_by: installedCurrentVersion.updated_by,
};
return {
id: installedCurrentVersion.id,
rule_id: installedCurrentVersion.rule_id,
revision: installedCurrentVersion.revision,
current_rule: installedCurrentVersion,
target_rule: targetRule,
diff: {
fields: pickBy<ThreeWayDiff<unknown>>(
ruleDiff.fields,
(fieldDiff) =>
fieldDiff.diff_outcome !== ThreeWayDiffOutcome.StockValueNoUpdate &&
fieldDiff.diff_outcome !== ThreeWayDiffOutcome.MissingBaseNoUpdate
),
num_fields_with_updates: ruleDiff.num_fields_with_updates,
num_fields_with_conflicts: ruleDiff.num_fields_with_conflicts,
num_fields_with_non_solvable_conflicts: ruleDiff.num_fields_with_non_solvable_conflicts,
},
};
});
};

View file

@ -5,4 +5,8 @@
* 2.0.
*/
export const PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS = 1800000 as const; // 30 minutes
export const PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS = 1_800_000 as const; // 30 minutes
// Only one rule installation or upgrade request can be processed at a time.
// Multiple requests can lead to high memory usage and unexpected behavior.
export const PREBUILT_RULES_OPERATION_CONCURRENCY = 1;

View file

@ -0,0 +1,96 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { IKibanaResponse, KibanaRequest, KibanaResponseFactory } from '@kbn/core/server';
import type { MaybePromise } from '@kbn/utility-types';
import type {
SecuritySolutionApiRequestHandlerContext,
SecuritySolutionRequestHandlerContext,
} from '../types';
import { throttleRequests } from './throttle_requests';
describe('throttleRequests', () => {
let mockContext: SecuritySolutionRequestHandlerContext;
let mockRequest: KibanaRequest;
let mockResponse: KibanaResponseFactory;
let mockHandler: jest.Mock<MaybePromise<IKibanaResponse>>;
beforeEach(() => {
mockContext = {} as SecuritySolutionRequestHandlerContext;
mockRequest = {} as KibanaRequest;
mockResponse = {} as KibanaResponseFactory;
mockHandler = jest.fn();
});
it('should call the route handler if no request is running', async () => {
const throttledHandler = throttleRequests(mockHandler);
mockHandler.mockResolvedValueOnce({} as IKibanaResponse);
await throttledHandler(mockContext, mockRequest, mockResponse);
expect(mockHandler).toHaveBeenCalledWith(mockContext, mockRequest, mockResponse);
});
it('should not call the route handler if a request is already running', async () => {
const throttledHandler = throttleRequests(mockHandler);
mockHandler.mockResolvedValueOnce(
new Promise((resolve) => setTimeout(() => resolve({} as IKibanaResponse), 100))
);
// Call the handler concurrently
void throttledHandler(mockContext, mockRequest, mockResponse);
await throttledHandler(mockContext, mockRequest, mockResponse);
expect(mockHandler).toHaveBeenCalledTimes(1);
});
it('should call the route handler multiple times on consecutive requests', async () => {
const throttledHandler = throttleRequests(mockHandler);
mockHandler.mockResolvedValueOnce({} as IKibanaResponse);
// Call the handler sequentially
await throttledHandler(mockContext, mockRequest, mockResponse);
await throttledHandler(mockContext, mockRequest, mockResponse);
expect(mockHandler).toHaveBeenCalledTimes(2);
});
it('should handle a failed route handler', async () => {
const throttledHandler = throttleRequests(mockHandler);
const error = new Error('Handler failed');
mockHandler.mockRejectedValueOnce(error);
await expect(throttledHandler(mockContext, mockRequest, mockResponse)).rejects.toThrow(
'Handler failed'
);
// Ensure that the next call can proceed after a failure
const resolvedValue = {} as IKibanaResponse;
mockHandler.mockResolvedValueOnce(resolvedValue);
const result = await throttledHandler(mockContext, mockRequest, mockResponse);
expect(mockHandler).toHaveBeenCalledTimes(2);
expect(result).toBe(resolvedValue);
});
it('should not throttle requests across different spaces when spaceAware is true', async () => {
const mockSpaceId1 = 'space-1';
const mockSpaceId2 = 'space-2';
mockContext.securitySolution = {
getSpaceId: jest.fn().mockResolvedValueOnce(mockSpaceId1).mockResolvedValueOnce(mockSpaceId2),
} as unknown as Promise<SecuritySolutionApiRequestHandlerContext>;
const throttledHandler = throttleRequests(mockHandler, { spaceAware: true });
mockHandler.mockResolvedValue({} as IKibanaResponse);
// Call the handler concurrently with different space IDs
void throttledHandler(mockContext, mockRequest, mockResponse);
await throttledHandler(mockContext, mockRequest, mockResponse);
expect(mockHandler).toHaveBeenCalledTimes(2);
});
});

View file

@ -0,0 +1,61 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { IKibanaResponse, KibanaRequest, KibanaResponseFactory } from '@kbn/core/server';
import type { MaybePromise } from '@kbn/utility-types';
import type { SecuritySolutionRequestHandlerContext } from '../types';
type RouteHandlerParams = [
context: SecuritySolutionRequestHandlerContext,
request: KibanaRequest,
response: KibanaResponseFactory
];
interface ThrottleOptions {
spaceAware?: boolean;
}
/**
* Throttles requests to ensure that only one request is processed at a time.
* Concurrent requests will be deduplicated and will share the same response.
*
* Optionally, it can be space-aware, meaning it will throttle requests based on
* the space ID.
*
* Note: This function is not suitable for routes that accept parameters in the
* request body. It might also lead to high memory usage in case of big response
* payloads and many concurrent requests.
*
* @param routeHandler - The route handler function to be throttled.
* @param options - Throttle options.
* @param options.spaceAware - If true, throttles requests based on the space
* ID.
* @returns A throttled version of the route handler.
*/
export const throttleRequests = (
routeHandler: (...params: RouteHandlerParams) => MaybePromise<IKibanaResponse>,
{ spaceAware = false }: ThrottleOptions = {}
) => {
const runningRequests = new Map<string, MaybePromise<IKibanaResponse>>();
return async (...params: RouteHandlerParams) => {
const spaceId = spaceAware ? (await params[0].securitySolution).getSpaceId() : 'default';
let currentRequest = runningRequests.get(spaceId);
if (!currentRequest) {
// There is no running request for this space, so we can start a new one
currentRequest = routeHandler(...params);
runningRequests.set(spaceId, currentRequest);
}
try {
return await currentRequest;
} finally {
runningRequests.delete(spaceId);
}
};
};