**Epic:** https://github.com/elastic/kibana/issues/118324
**Tickets:** https://github.com/elastic/kibana/issues/119603, https://github.com/elastic/kibana/issues/119597, https://github.com/elastic/kibana/issues/91265, https://github.com/elastic/kibana/issues/118511
## Summary
The legacy rule execution logging implementation is replaced by a new one that introduces a new model for execution-related data, a new saved object and a new, cleaner interface and implementation.
- [x] The legacy data model is deleted (`IRuleStatusResponseAttributes`, `IRuleStatusSOAttributes`)
- [x] The legacy `siem-detection-engine-rule-status` saved object type is deleted and marked as deleted in `src/core`
- [x] A new data model is introduced (`x-pack/plugins/security_solution/common/detection_engine/schemas/common/rule_monitoring.ts`). This data model doesn't contain a mixture of successful and failed statuses, which should simplify client-side code (e.g. the code of Rule Management and Monitoring tables, as well as Rule Details page).
- [x] A new `siem-detection-engine-rule-execution-info` saved object is introduced (`x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/rule_execution_info/saved_object.ts`).
- [x] This SO has 1:1 association with the rule SO, so every rule can have 0 or 1 execution info associated with it. This SO is used in order to 1) update the last execution status and metrics and 2) fetch execution data for N rules more efficiently comparing to the legacy SO.
- [x] The logic of creating or updating this SOs is based on the "upsert" approach (planned in https://github.com/elastic/kibana/issues/118511). It does not fetch the SO by rule id before updating it anymore.
- [x] Rule execution logging logic is rewritten (see `x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log`). The previous rule execution log client is split into two objects: `IRuleExecutionLogClient` for using it from route handlers, and `IRuleExecutionLogger` for writing logs from rule executors.
- [x] `IRuleExecutionLogger` instance is scoped to the currently executing rule and space id. There's no need to pass rule id, name, type etc to `.logStatusChange()` every time.
- [x] Rule executors and related functions are updated.
- [x] API routes are updated, including the rule preview route which uses a special "spy" implementation of `IRuleExecutionLogger`. A rule returned from an API endpoint now has optional `execution_summary` field of type `RuleExecutionSummary`.
- [x] UI is updated to use the new data model of `RuleExecutionSummary`:
- [x] Rule Management and Monitoring tables
- [x] Rule Details page
- [x] A new API route is introduced for fetching rule execution events: `/internal/detection_engine/rules/{ruleId}/execution/events`. It is used for rendering the Failure History tab (last 5 failures) and is intended to be used in the coming UI of Rule Execution Log on the Details page.
- [x] Rule Details page and Failure History tab are updated to use the new data models and API routes.
- [x] I used `react-query` for fetching execution events
- [x] See `x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/use_rule_execution_events.tsx`
- [x] The lib is updated to the latest version
- [x] Tests and fixed and updated according to all the changes
- [x] Components related to rule execution statuses are all moved to `x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status`.
- [x] I left a lot of `// TODO: https://github.com/elastic/kibana/pull/121644` comments in the code which I'm planning to address and remove in a follow-up PR. Lots of clean up work is needed, but I'd like to unblock the work on Rule Execution Log UI.
## In the next episodes
- Address and remove `// TODO: https://github.com/elastic/kibana/pull/121644` comments in the code
- Make sure that SO id generation for `siem-detection-engine-rule-execution-info` is safe and future-proof. Sync with the Core team. If there are risks, we will need to choose between risks and performance (reading the SO before updating it). It would be easy to submit a fix if needed.
- Add APM integration. Use `withSecuritySpan` in methods of `rule_execution_log` citizens.
- Add comments to the code and README.
- Add test coverage.
- Etc...
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### For maintainers
- [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
* chore(NA): auto creation of the package.json for the new types pkg rule
* chore(NA): first alpha api extractor working version
* chore(NA): support kbn-analytics
* chore(NA): correctly read tsconfig files and deps from ts_config rule
* chore(NA): layed out pkg_npm_types tree artifact custom rule
* chore(NA): missing todos
* chore(NA): node modules link mapping
* chore(NA): fully working pkg_npm_types rule
* chore(NA): fix changes on new packages using elastic datemath pkgs
* docs(NA): remove todo
* docs(NA): last todo text correction
* chore(NA): removed commented lines
* fix(NA): include missing package version
* chore(NA): include license keys
* chore(NA): change mock types package into private
* chore(NA): disable validator on ts_project rule
* chore(NA): use the wrapper for ts_project
* commit using @elastic.co
* chore(NA): commit using @elastic.co
* chore(NA): split types from code on @kbn/i18n
* chore(NA): update yarn.lock file
* chore(NA): split @kbn/i18n and @kbn/i18n-react
* chore(NA): missing import fix
* chore(NA): fix jest project configs
* chore(NA): change imports on kbn i18n
* chore(NA): change imports on kbn i18n
* chore(NA): correct loader imports
* chore(NA): missnig i18nLoader export key
* chore(NA): fix type exports
* chore(NA): export type only
* chore(NA): export type only
* fix(NA): type exports
* chore(NA): missing @Kbn/i18n/react imports
* chore(NA): missing skip path for kbn-i18n-react
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* eui to v41.0.0
* update eui i18n tokens
* sass tokens
* EuiLoadingKibana
* EuiCodeEditor
* subdued
* betaBadgeProps
* EuiCodeEditor null-loader
* src secondary -> success
* [enterprise_search] Deprecate 'secondary' color prop
* [apm] Deprecate 'secondary' color prop
* [canvas] Deprecate 'secondary' color prop
* [cases] Deprecate 'secondary' color prop
* [cross_cluster_replication] Deprecate 'secondary' color prop
* [data_enhanced] Deprecate 'secondary' color prop
* [data_visualizer] Deprecate 'secondary' color prop
* [fleet] Deprecate 'secondary' color prop
* [index_management] Deprecate 'secondary' color prop
* [infra] Deprecate 'secondary' color prop
* [ingest_pipelines] Deprecate 'secondary' color prop
* [maps] Deprecate 'secondary' color prop
* [ml] Deprecate 'secondary' color prop
* [monitoring] Deprecate 'secondary' color prop
* [observability] Deprecate 'secondary' color prop
NB: conditional became `type === 'success' ? 'success' : type` after find&replace, which felt fairly redundant, so I simplified it
* [osquery] Deprecate 'secondary' color prop
* [painless_lab] Deprecate 'secondary' color prop
* [remote_clusters] Deprecate 'secondary' color prop
* [rollup] Deprecate 'secondary' color prop
* [security] Deprecate 'secondary' color prop
* [security_solution] Deprecate 'secondary' color prop
NB: several conditional became `type === 'success' ? 'success' : type` after find&replace, which felt fairly redundant, so I simplified them
* [snapshot_restore] Deprecate 'secondary' color prop
* [spaces] Deprecate 'secondary' color prop
* [transform] Deprecate 'secondary' color prop
* [triggers_actions_ui] Deprecate 'secondary' color prop
* [uptime] Deprecate 'secondary' color prop
* [watcher] Deprecate 'secondary' color prop
* [infra] replace ambiguous 'secondary' color
- GaugesSectionVis doesn't appear to use the color property but it's required by the SeriesOverrides types, so changing it just in case
* [examples] Deprecate 'secondary' color prop
* [uptime] deprecate 'subdued' prop on EuiButtonIcon
* revert EuiKeyPadMenuItem betaBadge props
* mobileOptions
* examples/ updates
* fix brace import
* fix type exports
* update expressions_explorer requiredBundles
* remove make_id mocks
* snapshot updates
* fix import 🤦
* Fix `ReferenceError: ace is not defined` Jest failures
* Remove unused brace import (?)
- Assuming here, as no code editor is actually being used in this file
* Fix failing Jest test due to EuiCodeEditor moving to es_ui_shared plugin
+ minor cleanup of `jest.mock()`s
* Fix failing Jest test due to snapshot update
* Fix failing `TypeError: Cannot read properties of undefined (reading 'euiBorderRadius')` Jest test
- since this is being mount()'d, EuiThemeProvider as a wrapper is needed to prevent the failure
* access uiSettings
* Move react-ace dependency into kbn-ui-shared-deps-npm
* Revert App Search shenanigans
- caused local unsaved changes shenanigans, somehow
* secondary -> success
Co-authored-by: Constance Chen <constance.chen.3@gmail.com>
Co-authored-by: Chandler Prall <chandler.prall@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Constance Chen <constance.chen@elastic.co>
* move pdf types to /common folder
* move pdf v2 types to /common folder
* move png v2 types to /common folder
* move png types to /common
* move csv_searchsource_immediate types to /common
* move csv_searchsource type sto /common
* move csv types to /common folder
* export job params types on server and client
* use JobParamsPDF in example app
* use JobParamsPDFV2 in Canvas
* dont export twice
* export JobId
* improve export syntax
* update jest snapshot
* fix imports
* add JobAppParamsPDFV2 type
* add JobAppParamsPDF type
* update test snapshot
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* added page to reporting example app that contains the capture tests
* first version of PNG capture for test A
* added types file to common
* added data-shared-item attr to image, also added capture menu items
* fix image CSS by providing a fixed width and height
* explicitly add layout for print, does not seem to do anything though?
* added magic numbers of image sizes
* added reporting examples test folder
* first version of capture test for generating and comparing PNGs
* added PNG service and PNG baseline fixture
* added pdf-to-img dev dependency
* refactor compare_pngs to accept a buffer
* added comment to interface
* png service -> compare images service
* export image compare service
* added test for pdf export
* clean up log
* minor fixes and added pdf print optimized test
* added pdf and pdf print fixtures
* refactor lib function name
* Update difference thresholds
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* added new data-render-error attribute, read it and store it on job object
* added data-render-error to the example app
* added jest test
* address pr feedback
- make renderErrors optional in interfaces
- create separate selectors for data render error selector/attr
- Tidy up mergeMap behaviour
* fix observable.test.ts snapshots and browser driver mock
* updated jest snapshots
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* very wip
* - Reached first iteration of reporting body value being saved with
the report for **PDF**
- Removed v2 of the reporting since it looks like we may be able
to make a backwards compatible change on existing PDF/PNG
exports
* reintroduced pdfv2 export type, see https://github.com/elastic/kibana/issues/99890\#issuecomment-851527878
* fix a whol bunch of imports
* mapped out a working version for pdf
* refactor to tuples
* added v2 pdf to export type registry
* a lot of hackery to get reports generated in v2
* added png v2, png reports with locator state
* wip: refactored for loading the saved object on the redirect app URL
* major wip: initial stages of reporting redirect app, need to add a way to generate v2 reports!
* added a way to generate a v2 pdf from the example reporting plugin
* updated reporting example app to read and accept forwarded app state
* added reporting locator and updated server-side route to not use Boom
* removed reporting locator for now, first iteration of reports being generated using the reporting redirect app
* version with PNG working
* moved png/v2 -> png_v2
* moved printable_pdf/v2 -> printable_pdf_v2
* updated share public setup and start mocks
* fix types after merging master
* locator -> locatorParams AND added a new endpoint for getting locator params to client
* fix type import
* fix types
* clean up bad imports
* forceNow required on v2 payloads
* reworked create job interface for PNG task payload and updated consumer code report example for forcenow
* put locatorparams[] back onto the reportsource interface because on baseparams it conflicts with the different export type params
* move getCustomLogo and generatePng to common for export types
* additional import fixes
* urls -> url
* chore: fix and update types and fix jest import mocks
* - refactored v2 behaviour to avoid client-side request for locator
instead this value is injected pre-page-load so that the
redirect app can use it
- refactored the interface for the getScreenshot observable
factory. specifically we now expect 'urlsOrUrlTuples' to be
passed in. tested with new and old report types.
* updated the reporting example app to use locator migration for v2 report types
* added functionality for setting forceNow
* added forceNow to job payload for v2 report types and fixed shared components for v2
* write the output of v2 reports to stream
* fix types for forceNow
* added tests for execute job
* added comments, organized imports, removed selectors from report params
* fix some type issues
* feedback: removed duplicated PDF code, cleaned screenshot observable function and other minor tweaks
* use variable (not destructured values) and remove unused import
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [build_ts_refs] improve caches, allow building a subset of projects
* cleanup project def script and update refs in type check script
* rename browser_bazel config to avoid kebab-case
* remove execInProjects() helper
* list references for tsconfig.types.json for api-extractor workload
* disable composite features of tsconfig.types.json for api-extractor
* set declaration: true to avoid weird debug error
* fix jest tests
Co-authored-by: spalger <spalger@users.noreply.github.com>
* Use Serializable from package
* Rename to align with core
* fix
* more replacements
* docssss
* fix
* Move it to @kbn/utility-types and remove core export
* buildy build
* tests
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* add version to csv params
* fix ts
* fix api tests
* use kibana version from packageInfo
* use kibana version from packageInfo
* clean up ide warnings
* utility to log and set a default params version
* fix baseparams ts
* update snapshot
* check version in enqueue job
* add temporary ts-ignore for canvas
* clarify comment
* fix hardcoded version in png_pdf_panel
* clarify the UNVERSIONED_VERSION variable with a comment
* fix canvas jest test
* fix ts in example app
* fix types
* send version param to canvas util for job params
* update jest snapshot
* Update utils.test.ts
* fix snapshot
* remove browserTimezone and version from integration boilerplate
* wip ensure version is always populated in job params inside of the service
* wip2
* wip3
* wip4
* wip5
* wip6
* update note
* update example plugin
* wip7
* improve tests
* fix dynamic job params
* better testing
* improve enqueue_job test
* more tests
* fix types
* fix types
* fix example ts
* simplify props
* fix test
* --wip-- [skip ci]
* consolidate baseparams back into one interface
* fix rison encoding of apiClient param
* clean up
* reorganize imports
* back out functional change
* fix 400 error in download csv
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Adding function hooks into rule type definition and call extract fn on rule create
* Adding hooks for extracting and injecting saved object references. Adding extractReferences to create and update workflow
* Adding type template for extracted params
* Adding type template for extracted params
* Adding type template for extracted params
* Adding type template for extracted params
* Calling injectReferences function if defined. Finishing unit tests for create and update
* Adding tests for get
* Adding tests for find
* Cleanup
* Fixing types check
* Fixing functional tests
* Fixing functional tests
* Fixing tests
* Updating README
* Throwing boom error instead of normal error
* Adding framework level prefix to extracted saved object reference names
* Fixing types
* Fixing types
* PR fixes
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Removing feature flag changes
* Adding isExportable flag to rule type definition
* Adding isExportable flag to rule type definition
* Adding isExportable flag to rule type definition
* Filtering rule on export by rule type isExportable flag
* Fixing types
* Adding docs
* Fix condition when exportCount is 0
* Unit test for fix condition when exportCount is 0
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* WIP - creating alerting authorization client factory and exposing authorization client on plugin start contract
* Updating alerting feature privilege builder to handle different alerting types
* Passing in alerting authorization type to AlertingActions class string builder
* Passing in authorization type in each function call
* Passing in exempt consumer ids. Adding authorization type to audit logger
* Changing alertType to ruleType
* Changing alertType to ruleType
* Updating unit tests
* Updating unit tests
* Passing field names into authorization query builder. Adding kql/es dsl option
* Converting to es query if requested
* Fixing functional tests
* Removing ability to specify feature privilege name in constructor
* Fixing some types and tests
* Consolidating alerting authorization kuery filter options
* Cleanup and tests
* Cleanup and tests
* Initial commit with changes needed for subfeature privilege
* Throwing error when AlertingAuthorizationClientFactory is not defined
* Renaming authorizationType to entity
* Renaming AlertsAuthorization to AlertingAuthorization
* Fixing unit tests
* Changing schema of alerting feature privilege
* Changing schema of alerting feature privilege
* Updating feature privilege iterator
* Updating feature privilege builder
* Fixing types check
* Updating privilege string terminology
* Updating privilege string terminology
* Wip
* Fixing unit tests
* Unit tests
* Updating README and removing stack subfeature privilege changes
* Fixing README
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* initial version of the screenshot mode service
* First iteration of client side of screenshot mode plugin
Also hooked it up to the chromium browser imitating the preload
functionality of electron to set up the environment before
code runs.
* First implementation of server-side logic for detecting
screenshot mode
* fix some type issues and do a small refactor
* fix size limits, docs and ts issues
* fixed types issues and made sure screenshot mode is correctly detected on the client
* Moved the screenshot mode header definition to common
Added a server-side example for screenshot mode
Export the screenshot mode header in both public and server
* move require() to screenshotMode plugin
* Update chromium_driver.ts
* cleaned up some comments, minor refactor in ReportingCore and
changed the screenshotmode detection function to check for a
specific value.
* fix export
* Expanded server-side screenshot mode contract with function that
checks a kibana request to determine whether we in screenshot
mode
* added comments to explain use of literal value rather than external reference
* updated comment
* update reporting example
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Timothy Sullivan <tsullivan@elastic.co>
Co-authored-by: Tim Sullivan <tsullivan@users.noreply.github.com>
