## Summary
Addresses https://github.com/elastic/kibana/issues/130699
This PR implements alert throttling per rule execution for query and
saved query rules. The implementation is very similar in concept to
threshold rules. We allow users to pick one or more fields to group
source documents by and use a composite aggregation to collect documents
bucketed by those fields. We create 1 alert for each bucket based on the
first document in the bucket and add metadata to the alert that
represents how to retrieve the rest of the documents in the bucket.
The metadata fields are:
- `kibana.alert.suppression.terms`: `{field: string; value: Array<string
| number>}` An array of objects, each object represents one of the terms
used to group these alerts
- `kibana.alert.suppression.start`: `Date` The timestamp of the first
document in the bucket
- `kibana.alert.suppression.end`: `Date` The timestamp of the last
document in the bucket
- `kibana.alert.suppression.docs_count`: `number` The number of
suppressed alerts
There is one new rule parameter, currently implemented at the solution
level, to enable this feature: `alertSuppression.groupBy`: `string[]`.
Similar to threshold rules, the throttled query rules keep track of
created alerts in the rule state in order to filter out duplicate
documents in subsequent rule executions. When a throttled alert is
created, we store the bucket information including field names, values,
and end date in the rule state. Subsequent rule executions convert this
state into a filter that excludes documents that have already been
covered by existing alerts. This is necessary because consecutive rule
executions will typically query overlapping time ranges.
## Screenshots
### Rule Create/Edit With License
<details>
</details>
### Rule Details With License
<details>
</details>
### Rule Create, or Rule Edit of a rule without existing suppression
configuration, Without License
<details>
</details>
### Editing a rule that has existing suppression configuration, but
without the correct license, still allows changing the configuration (to
allow removing the params)
<details>
</details>
### Rule Details Without License
<details>
</details>
### Alerts table
<details>
</details>
### Known issues
- The layers icon in the rule name for suppressed alerts does not show
up in the rule preview table
Co-authored-by: Madi Caldwell <madison.caldwell@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* move flapping to kibana.alerting in event log
* move flapping back to under kibana.alert. Add integration tests
* add default flapping state to alert logs
* [ResponseOps] Add kibana.alert.time_range field to Alert-As-Data mappings and populate it
* Fixing snapshots to match new reality
* Removing the lte (end of range) for active alerts.
* Fixing expected resutls for mapping test
* fixing tests
* updating readme
* Fixing field name in README
Co-authored-by: Faisal Kanout <faisal.kanout@elastic.co>
* [packages] add kibana.jsonc files
* auto-migrate to kibana.jsonc
* support interactive pkg id selection too
* remove old codeowners entry
* skip codeowners generation when .github/CODEOWNERS doesn't exist
* fall back to format validation if user is offline
* update question style
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* refact(NA): apply root_input_dir=src to each already created pkg
* refact(NA): update package generator
* fix(NA): correctly use rootDir
* fix(NA): use root input dir on latest introduced pkgs for jsts_transpiler macro
* chore(NA): merge with main
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [type-summarizer] reimplement for broader support
* Enable sourceMaps in all packages
* include naming collision in summarizePackage test
* fix readmes
* remove unnecessary transient dependency
* remove code that was commented out
* remove outdated todo comment
* ensure errors triggered by untyped-exports are ligible
* remove unused import
* break out snippet generation from AstIndexer
* refactor several massive files into smaller pieces and add more inline docs
* fix typos
* update jest snapshots
* add sections to readme that points people to the useful parts of the source code along with a high-level overview of how the type-summarizer works
* remove --dump flag, it doesn't work
* use decName instead of calling names.get a second time
* include `export` as invalid name
* Replace schemas derived from FieldMaps with versioned alert schema
* Import fixes and comment
* Another import fix
* Separate read and write schemas
* Separate read and write schemas for common alert fields
* fix import
* Update ALERT_RULE_PARAMETERS type
* Fix getField type
* Fix more types
* Remove unneeded index signature from PersistenceAlertServiceResult
* Fix types and tests
* Update comment describing new schema process
* Update Ancestor800 type
* Add modified PR description as initial README
* Remove duplication in CommonAlertFields definition
* Add explicit undefined value for rule in mock
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Add exceptions to threshold timeline
* Tests and error handling
* Fix unit tests
* Add alias for exceptions filter
* Fix tests
* Type fixes
Co-authored-by: Marshall Main <marshall.main@elastic.co>
* chore(NA): splits types from code on @kbn/rule-data-utils
* chore(NA): remove old style imports for this pkg
* chore(NA): eslint fix
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Remove kibana.alert.rule.risk_score and severity
* Fix tests related to risk_score and severity
* Make translation a template
* Can't use expression in template literal
* Remove commented line added by bad merge
* Fix linting
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* add kibana.alert.rule.parameters as a flattened type
* temp
* rule_data_formatter
* fix bug in search strategy with flattend field type where prefix was wrong (kibana.alert.rule.parameters was ignored)
* fix inventory rule data formatters
* remove console log
* hack that prepends kibana.alerts.rule.parameters in the nested subfields
* import ALERT_RULE_PARAMETERS from kbn rule data utils
* remove console log
* format custom metric link
* remove ALERT_PARAMS from technical field names
* fix bug in timelines plugin to use dotField instead of prependField & fix failing tests
* remove console log and unused variable
* delete kibana.alert.rule.params from the mapping
* flatten kibana.alert.rule.parameters and add some unit tests
* fix rule_data_formatter
* handle scenario of having multiple items in an array (multiple conditions setup in the rule)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Add flattend parameters object and populate it in Security Solution
* Fix severity, risk_score, bugs, tests
* Add ALERT_RULE_PARAMETERS to package
* Skip tightly coupled test
* fix more tests
* Remove unused import
* Fix threat matching API test
* Continue overriding kibana.alert.rule.risk_score and severity for now
* Add ignore_above to ALERT_RULE_PARAMETERS
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Switches RuleDetails to query alerts by ruleId instead of SO id
* Increases integrity of test outputs
* Cleans up duplicate RuleRegistry functions
* Removes support for rule.id for alerts filter and updates exceptions to use new filter
* [kbn/rule-data-utils] add submodules and require public use them
* fix lint errors
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Initial commit
* Properly handle signal history
* Fix#95258 - cardinality sort bug
* Init threshold rule
* Create working threshold rule
* Fix threshold signal generation
* Fix tests
* Update mappings
* ALERT_TYPE_ID => RULE_TYPE_ID
* Add tests
* Fix types
* Adds RAC rule type migration
* Fix threshold tests (remove outputIndex)
* Add threshold rule type to ruleTypeMappings
* Add kbn-securitysolution-rules package for sharing with alerting framework
* Fix type errors
* Fix find_rules tests
* First round of test fixes
* Fix issues from merge conflicts
* Use ruleDataClient getReader() for reading
* Fixes to 'generating_signals' tests
* Remove more refs to legacy schema
* Linting
* Quick type fix
* Bug fixes
* Add saved query rule type
* Linting
* Fix types
* Signal generation tests
* Test updates
* Update some more refs
* build_alert tests
* Cleanup
* Ref updates
* Revert "Ref updates"
This reverts commit 4d1473d6b0.
* Update status field
* Test fixes
* Another test
* Got a little too aggressive with search/replace
* let's see where we're at
* Fix
* Test fixes
* cleanup
* Fix cases API integration test config, flaky DE tests
* Move flattenWithPrefix to package / skip signal migration tests
* Fix unit tests
* Use new schema for bulk rule creation
* event: { kind } => event.kind
* Fix signal migration API tests
* Fix ml integration test
* Fix threat match integration tests
* Fix ML rule type tests and add correct producer to all rule types
* Update threat match API integration test
* Remove dupe properties
* Type fix
* Fix ML producer in functional test
* Fix generating_signals tests
* Remove usage of RuleDataClient-based execution log client
* Don't check output index version if rule registry enabled
* Fix bulk duplicate rule
* Fix duplicate rule test
* Fix readPrivileges and timestamp check logic
* Fixes for eql and exceptions tests... disable open_close_signals
* Type fixes / keyword test fixes
* Additional test fixes
* Unit test fixes + signal -> kibana.alert
* Test fixes for exceptions
* Fix read_resolve_rules test
* Various test fixes with marshallmain
* Sort search results
* Fix create_rules tests
* Disable writer cache for integration tests
* Disable writer cache for cases integration tests
* Fix types in rule_data_plugin_service
* Fix ordering in exceptions tests
* Remove rule_registry.enabled flag
* Fix signals migration tests
* Don't check signals index before creation
* Fix cypress config
* Fix type error
* create_migrations tests
* Skip flaky test
* Helpful comment
* Fixes from merge conflicts
* Pretend that signals index exists
* Fix type errors
* Skip flaky tests
* Fix threat matching test
* Clean up
* Reverting default ruleRegistry experimental flag (breaks unit tests)
* Reenable rule registry experimental feature by default
* Execute DE rule migration in 8.0
Co-authored-by: Marshall Main <marshall.main@elastic.co>
* bump to a pre-8.0 version
* export KibanaClient from /lib sub-folder
* workaround the problem of the absence of estypes
* update es client usage in pacakges
* export estypes from another path
* import errors from root
* import errors from root 2
* update transport import
* update import path for /api/types
* update import path for /api/types
* import errors from top export
* use TransportResult instead if ApiResponse
* fix errors in client_config
* fix src/core/server/saved_objects/migrationsv2/actions/integration_tests/actions.test.ts
* use KibanaClient in mock. we dont export the original Client
* fix client mocks
* fix errors on SO
* fix remaining core errors
* update estype import path
* fix errors in data plugin
* fix data_views
* fix es_ui_shared
* fix errors in interactive_setup
* fix errors in ./test folder
* add @elastic/transport to the runtime deps
* fix errors in packages
* fix erros in src/core
* fix errors in test/
* fix an error in actions plugin
* woraround and fix errors in APM plugin
* fix errors in canvas
* fix errors in event_log
* fix errors in fleet
* fix errors in ILM
* fix errors in infra
* fix errors in ingest_pipeline
* fix errors in lens
* fix errors in license_management
* fix errors in licensing
* fix errors in logstash
* fix errors in ml
* fix errors in monitoring
* fix errors in observability
* fix errors in rule_registry
* fix errors in reporting
* fix errors in rule_registry
* fix errors in security
* fix errors in security_solution
* fix errors in snapshot_restore
* fix errors in transform
* fix errors in UA
* fix errors in uptime
* fix errors in x-pack/test
* fix eslint errors
* fix new errors
* use default HTTP Connection. Undici does not support agent config options keepAlive and maxSockets
* create does not accept require_alias option
* update deps
* use transport types exported from ES client package
* fix ErrorCause | string errors
* do not use enum
* fix errors in data plugin
* update x-pack code
* fix transport
* fix apm search request
* do not crash on reporting
* fix kbn-test build
* mute reporting error to start
* fix ftr build
* another attempt
* update import path
* address or mute new errors
* REMOVE me. pin transport version temporarily.
* remove deep imports from transport package
* fix jest crash
* fix product check tests
* remove unnecessary ts-expect-error
* fix a few failed unit tests
* bump to canary 24
* remove unnecessary ts-expect-error
* remove dependency on transport
* fix types in tests
* mute errors in xpack tests
* product check doesn;t spam in logs anymore
* filterPath --> filter_path
* ignoreUnavailable --> ignore_unavailable
* ignoreUnavailable --> ignore_unavailable
* trackScores --> track_scores
* trackTotalHits --> track_total_hits
* fix es-arcives
* fix data plugin crashes
* fix watcher test utils
* rollback unnecessary changes
* fix another problem in es-archiver
* fix scroll. for whatever reason scroll fails when request scroll_id in body
* add meta: true in kbn-securitysolution-es-utils
* bump client to canary 25
* fix errors in accordance with the es client spec
* update securityscolution-es-utils
* unify scroll api in reporting and fix tests
* fix unit tests in watcher
* refactor APM to abort request with AbortController API
* fix missing es client calls in tests
* fix missing meta in detection engine FTR tests
* fix another bunch of errors in js tests
* fix wrong coercion
* remove test-grep pattern
* fix apm unit test
* rename terminateAfter to terminate_after in infra plugin
* rename terminateAfter to terminate_after in uptime plugin
* rename terminateAfter to terminate_after in apm plugin
* fix security roles FTR tests
* fix reference
* fix post_privilidges test
* fix post_privilidges
* bump client to 26
* add meta for index_management test helpers
* remove ts-expect-error caused by bad type in reason
* bump client to 27
* REMOVE me. workaround until fixed in the es client
* fix incorrect type casting
* swtich from camelCase params
* use `HttpConnection` for FTR-related clients
* bump client to 29
* Revert "REMOVE me. workaround until fixed in the es client"
This reverts commit c038850c09.
* fix new util
* revert repository changes
* do not crash if cannot store event_loop data
* fix new estypes imports
* fix more types
* fix security test types and add ts-ignore for custom ES client
* fix more estypes imports
* yet more ts violations
* line by line fixing is hard
* adapt `evaluateAlert` from infra as it's also used from FTR tests
* use convertToKibanaClient in FTR test instead of meta:true in plugin code
* migrate from deprecated API in fleet
* fix intergration tests
* fix fleet tests
* fix another fleet test
* fix more tests
* let's call it a day
* Removes custom header check on 404 responses, includes es client ProductNotSupportedError in EsUnavailableError conditional (#116029)
* Removes custom header check on 404 responses, includes es client ProductNotSupportedError in EsUnavailableError conditional
* Updates proxy response integration test
* disable APM until compatible with client v8
* skip async_search FTR test
* use kbnClient in integration tests
* bump version to 29
* bump to 30
* have configureClient return a KibanaClient instead of Client, remove resolved violations.
* bump to 31
* bump to 31
* Revert "bump to 31"
This reverts commit 5ac713e640.
* trigger stop to unusubscribe
* update generated docs
* remove obsolete test
* put "as" back
* cleanup
* skip test
* remove new type errors in apm package
* remove ErrorCause casting
* update a comment
* bump version to 32
* remove unnecessary ts-expect-error in apm code
* update comments
* update to client v33
* remove outdated type definition
* bump to 34 without params mutation
* unskip the test that should not fail anymore
* remove unnecessary ts-expect-error comments
* update to v35. body can be string
* move `sort` to body and use body friendly syntax
* fix a failing test. maps register the same SO that has been already registered by home
Co-authored-by: pgayvallet <pierre.gayvallet@gmail.com>
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
* [RAC][Observability] remove severity fields from mapping keep only ALERT_SEVERITY
* temporarily remove severity value occurences
* remove ALERT_SEVERITY_VALUE occurences, this value is not being read and shown in the Observability alerts table
* remove duplicate ALERT_SEVERITY identifier
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Holy moly.
What is happening in this PR? 🤷🏽♀️ Let's break it down:
- Added a package `@kbn/alerts` - another one?! ...yes
- This is meant to add shared hooks and components around alerts as data
- `useGetUserAlertsPermissions` - accepts the Kibana capabilities object and returns whether the user has `read` and `crud` alerts privileges
- `AlertsFeatureNoPermissions` - component displayed when user does not have alerts privileges
- UI changes for user with NO alerts privileges
- `Alerts` tab hidden in security solution side navigation
- `Alerts` tab hidden in rule details page
- UI changes for user with alerts READ ONLY privileges
- alerts checkboxes hidden in alerts table
- alerts bulk actions hidden in alerts table
### Summary
### Fields used moving forward
`kibana.alert.rule.consumer` will refer to the context in which a rule instance is created. Rules created in:
- stack --> `alerts`
- security solution --> `siem`
- apm --> `apm`
`kibana.alert.rule.producer` will refer to the plugin that registered a rule type. Rules registered in:
- stack --> `alerts`
- security solution --> `siem`
- apm --> `apm`
So an `apm.error_rate` rule created in stack will have:
- consumer: `alerts` and producer: `apm`
An `apm.error_rate` rule created in apm will have:
- consumer: `apm` and producer: `apm`
`kibana.alert.rule.rule_type_id` will refer to a rule's rule type id. Examples:
- `apm.error_rate`
- `siem.signals`
- `siem.threshold`
Also renamed the following because `rule.*` fields are meant to be ecs fields pulled from the source/event document, not refer to our rule fields.
`rule.name` --> `kibana.alert.rule.name` will refer to the rule's name.
`rule.category` --> `kibana.alert.rule.category` will refer to the rule's category.
`rule.id` --> `kibana.alert.rule.uuid` will refer to the rule's uuid.
* [build_ts_refs] improve caches, allow building a subset of projects
* cleanup project def script and update refs in type check script
* rename browser_bazel config to avoid kebab-case
* remove execInProjects() helper
* list references for tsconfig.types.json for api-extractor workload
* disable composite features of tsconfig.types.json for api-extractor
* set declaration: true to avoid weird debug error
* fix jest tests
Co-authored-by: spalger <spalger@users.noreply.github.com>
