9566f1a7b3
## Summary Original issue: https://github.com/elastic/security-team/issues/10688 This PR adds a new function to the rules client called `bulkFillGapsByRuleIds`, that takes a list of rule ids and a date range, and attempts to fill out the gaps simultaneously, 10 rules at a time (can be customizable). For all rules we verify if the user has access to write to the gaps object, then for each rule we fetch gaps and trigger the backfilling of its unfilled intervals. During the execution, we aggregate the errors and the rules that are skipped in lists and return them along with the scheduling outcomes. ### Error handling There are 2 error types that are handled throughout the scheduling of a gaps backfilling for a rule: - A user doesn't have access the rule - We fail to backfill gaps ### Skipped rules A rule can be skipped when it doesn't have unfilled gaps in the given time range, or when gaps are found, but they are currently "in progress". ### Endpoint This PR also updates the existing bulk actions endpoint by adding the action `fill_gaps`. The endpoint for now should limit the amount of rules per call to 100. The bulk actions endpoint returns a summary of the execution which is a list of counters of rules that succeeded, failed or were skipped. ## How to test? 1. Create several rules that run every 5 seconds and disable them. Leave them disabled for a couple of minutes and then re-enable them in order to create some gaps for it. You can confirm that there are gaps by clicking on the rule, then on the "Execution results" tab, and then scrolling down to "Gaps". Alternatively you can use [this tool](https://github.com/elastic/security-documents-generator) to create 1 rule (5m) with 1000 gaps: ``` yarn start rules --rules 1 -g 1000 -c -i"5m" ``` 2. Get the ids of each rule that you created and call the endpoint to backfill them. You can select a time range that is smaller than the gaps so that you can call it several times. Here is an example of a curl to do this ``` curl -X POST "http://localhost:5601/api/detection_engine/rules/_bulk_action?dry_run=false" \ -H "Accept: */*" \ -H "Content-Type: application/json" \ -H "kbn-version: 9.1.0" \ -H "kbn-build-number: 9007199254740991" \ -H "elastic-api-version: 2023-10-31" \ -H "Authorization: Basic $(echo -n 'elastic:changeme' | base64)" \ --data-raw '{ "action": "fill_gaps", "ids": ["307bdea8-28be-419f-bb25-dac0024f32af"], "fill_gaps": { "start_date": "2025-05-09T08:12:09.457Z", "end_date": "2025-05-09T09:12:09.457Z" } }' ``` 3. Then you can go to the "Gaps" tab from step 1 and verify that the gaps are being filled. Additionally, you can see that there are manual runs scheduled for the rule in order to fill the gaps. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| examples | ||
| linters | ||
| output | ||
| overlays | ||
| scripts | ||
| bundle.json | ||
| bundle.serverless.json | ||
| kibana.info.serverless.yaml | ||
| kibana.info.yaml | ||
| makefile | ||
| package-lock.json | ||
| package.json | ||
| README.md | ||
Kibana API reference documentation
Documentation about our OpenAPI bundling workflow and configuration. See Kibana's hosted stateful and serverless docs.
Workflow
The final goal of this workflow is to produce an OpenAPI bundle containing all Kibana's public APIs.
Step 0
OAS from Kibana's APIs are continuously extracted and captured in bundle.json and bundle.serverless.json as fully formed OAS documentation. See node scripts/capture_oas_snapshot --help for more info.
These bundles form the basis of our OpenAPI bundles to which we append and layer extra information before publishing.
Step 1
Append pre-existing bundles not extracted from code using kbn-openapi-bundler to produce the final resulting bundles.
To add more files into the final bundle, edit the appropriate oas_docs/scripts/merge*.js files.
Step 2
Apply any final overalys to the document that might include examples or final tweaks (see the "Scripts" section for more details).
Scripts
The oas_docs/scripts folder contains scripts that point to the source domain-specific OpenAPI bundles and specify additional parameters for producing the final output bundle. Currently, there are the following scripts:
-
merge_ess_oas.jsscript produces production an output bundle for ESS -
merge_serverless_oas.jsscript produces production an output bundle for Serverless
Output Kibana OpenAPI bundles
The oas_docs/output folder contains the final resulting Kibana OpenAPI bundles
kibana.yamlproduction ready ESS OpenAPI bundlekibana.serverless.yamlproduction ready Serverless OpenAPI bundle
Bundling commands
Besides the scripts in the oas_docs/scripts folder, there is an oas_docs/makefile to simplify the workflow. Use make help to see available commands.