mirror of
https://github.com/elastic/kibana.git
synced 2025-04-25 02:09:32 -04:00
16 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Ryland Herrick
|
2aa94a27f0
|
[Detection Engine] Adds Alert Suppression to ML Rules (#181926)
## Summary This PR introduces Alert Suppression for ML Detection Rules. This feature is behaviorally similar to alerting suppression for other Detection Engine Rule types, and nearly identical to the analogous features for EQL rules. There are some additional UI behaviors introduced here as well, mainly intended to cover the shortcomings discovered in https://github.com/elastic/kibana/issues/183100. Those behaviors are: 1. Populating the suppression field list with fields from the anomaly index(es). 1. Disabling the suppression UI if no selected ML jobs are running (because we cannot populate the list of fields on which they'll be suppressing). 1. Warning the user if _some_ selected ML jobs are not running (because the list of suppression fields may be incomplete). See screenshots below for more info. ### Intermediate Serverless Deployment As per the "intermediate deployment" requirements for serverless, while the schema (and declared alert SO mappings) will be extended to allow this functionality, the user-facing features are currently hidden behind a feature flag. Once this is merged and released, we can issue a "final" deployment in which the feature flag is enabled, and the feature effectively released. ## Screenshots * Overview of new UI fields <img width="1044" alt="Screenshot 2024-05-16 at 3 22 02 PM" src=" |
||
Vitalii Dmyterko
|
6e6b99c4bb
|
[Security Solution][Detection Engine] adds alert suppression to ES|QL rule type (#180927)
## Summary
- addresses https://github.com/elastic/security-team/issues/9203
- adds alert suppression for new terms rule type
- similarly to [custom investigation
fields](https://github.com/elastic/kibana/pull/177746) list of available
suppression fields:
- shows only ES|QL fields returned in query for aggregating queries
- shows ES|QL fields returned in query + index fields for
non-aggregating queries. Since resulted alerts for this type of query,
are enriched with source documents.
### Demo
1. run esql rule w/o suppression
2. run esql rule w/ suppression per rule execution. Since ES|QL query is
aggregating, no alerts suppressed on already agrregated field `host.ip`
3. run suppression on interval 20m
4. run suppression for custom ES|QL field which is the same as
`host.ip`, hence same results
5. run suppression on interval 100m
|
||
|
Vitalii Dmyterko
|
13828920e4
|
[Security Solution][Detection Engine] removes tech preview from alert suppression for query rule type (#181279)
## Summary - removes tech preview badge/note from alert suppression for query rule type ### UI <details> <summary>Rule create form</summary> ### Before <img width="1050" alt="Screenshot 2024-04-22 at 16 26 50" src=" |
||
Wafaa Nasr
|
d01a5c4fe0
|
[Detection Engine][Rule Suppression] Add Suppression to EQL Non-sequence based queries (#176422)
# Summary - Address adding suppression to EQL rules https://github.com/elastic/security-team/issues/7773 - Milestone details https://github.com/elastic/security-team/issues/8432 ## Checklist - [x] Functional changes are hidden behind a feature flag. If not hidden, the PR explains why these changes are being implemented in a long-living feature branch. - [x] Functional changes are covered with a test plan and automated tests. [Test plan](https://github.com/elastic/security-team/pull/9155) - [ ] Stability of new and changed tests is verified using the [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner) in both ESS and Serverless. By default, use 200 runs for ESS and 200 runs for Serverless. * Cypress ESS: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5686 * Cypress Serverless: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5687 * FTR ESS: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5688 * FTR Serverless: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5689 - [x] Comprehensive manual testing is done by two engineers: the PR author and one of the PR reviewers. Changes are tested in both ESS and Serverless. - [x] Mapping changes are accompanied by a technical design document. It can be a GitHub issue or an RFC explaining the changes. The design document is shared with and approved by the appropriate teams and individual stakeholders. - [ ] (OPTIONAL) OpenAPI specs changes include detailed descriptions and examples of usage and are ready to be released on https://docs.elastic.co/api-reference. NOTE: This is optional because at the moment we don't have yet any OpenAPI specs that would be fully "documented" and "GA-ready" for publishing on https://docs.elastic.co/api-reference. - [x] Functional changes are communicated to the Docs team. A ticket is opened in https://github.com/elastic/security-docs using the [`Internal documentation request (Elastic employees)`](https://github.com/elastic/security-docs/issues/new?assignees=&labels=&projects=&template=docs-request-internal.yaml&title=%5BRequest%5D+) template. The following information is included: feature flags used, target ESS version, planned timing for ESS and Serverless releases. - [x] Check if in timeline we can show the suppression count column when the user clicks on investigate on timeline for Eql suppressed Alerts (https://github.com/elastic/kibana/issues/180976) ## Related Issues * Sub-PRs - Address EQL schema changes PR https://github.com/elastic/kibana/pull/176391 - Adding Feature flag PR and updating the Frontend Part in Rule Create/Edit https://github.com/elastic/kibana/pull/176398 - Adding Backend changes and FTR tests https://github.com/elastic/kibana/pull/176597 - Fix Investigate in Timeline for the Suppressed Alerts https://github.com/elastic/kibana/pull/177839 - Add Cypress e2e tests https://github.com/elastic/kibana/pull/177870 - Disable EQL sequence suppression in the UI and fix Cypress `after` esArchive path https://github.com/elastic/kibana/pull/178531 - Docs Issue https://github.com/elastic/security-docs/issues/4977 - Test plan https://github.com/elastic/security-team/pull/9155 ## Screenshots/recordings ### Non-Sequence Suppression 1. Rule creation, Suppression based on a single value |
||
|
Vitalii Dmyterko
|
52cfdd6fc2
|
[Security Solution][Detection Engine] adds alert suppression for New Terms rule type (#178294)
## Summary - addresses https://github.com/elastic/security-team/issues/8824 - adds alert suppression for new terms rule type - fixes `getOpenAlerts` test function, which returned closed alerts as well ### UI <img width="2294" alt="Screenshot 2024-04-02 at 12 53 26" src=" |
||
|
Vitalii Dmyterko
|
6d5a485415
|
[Security Solution][Detection Engine] adds alert suppression to Indicator Match rule (#174241)
## Summary
- addresses https://github.com/elastic/security-team/issues/7773 Epic
- addresses https://github.com/elastic/security-team/issues/8360
Alert suppression for these rule types is hidden behind feature branch
In this PR implemented:
- schema changes: allowing `alert_suppression` object in Indicator match
rule type. `alert_suppression` is identical to existing one for query
rule
- UI changes
- Cypress tests
- BE implementation
- FTR tests
Enabling feature flags
- `alertSuppressionForIndicatorMatchRuleEnabled`
### Tech implementation details
Alert candidates for IM rule deduplicated first, by searching in
existing alerts matched ids.
Once retrieved, alert candidates filtered further, to determine whether
they been already suppressed.
It's done by checking each alert candidate suppression time boundaries.
If suppression ends earlier than existing alert suppression with the
same instance id, alert candidate is removed.
The rest of alert candidates are getting suppressed in memory and either
new alerts created or existing updated.
The max limit of created and suppressed alerts is set to `5 *
max_signals`, which would allow to capture additional threats, should
rule execution's alerts number reach max_signals
### UI changes
Suppression components in IM rule are identical to Custom Query's
## Summary - related [epic](https://github.com/elastic/security-team/issues/6196) - introduces new ES|QL rule type in Technical Preview Stage - historical POC architecture [document](https://docs.google.com/document/d/1hcKzNrDEIrmoWwWoqas1YZ-bd8Kk5NRjJNSUaCvSntM/edit#heading=h.gheuu8zcz481)(internal link). Some of the information there can be outdated, but might be useful for historical context of some tech decision. In future, detailed technical documentation will be added ### UI ES|QL query component introduced in rule edit/creation form Rule name override supports values returned from ES|QL query As agreed on Adv. correlation WG, we don't introduce similar possibility for risk score/severity override at this point <details> <summary>How it looks like in UI</summary> <img width="2082" alt="Screenshot 2023-09-21 at 11 52 59" src=" |
||
Devin W. Hurley
|
fec18bab42
|
[Security Solution] [Platform] Adds support for data views and runtime field mappings in rule creation, exceptions, and during execution (#130929)
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Yara Tercero <yctercero@users.noreply.github.com> |
||
Frank Hassanabad
|
aa2f5b535d
|
[Security Solution] Utilizes constants package and deletes duplicate code (#100513)
## Summary Utilizes constants package and deletes duplicate code * Renames the `securitysolution-constants` to be `securitysolution-list-constants` to be specific * Deletes duplicated code found during cleanup * Moves more tests into the packages found along the way with the duplicated code * Moves `parseScheduleDates` from `@kbn/securitysolution-io-ts-types` to `@kbn/securitysolution-io-ts-utils` ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios |
||
|
Frank Hassanabad
|
bfe08d25c5
|
[Security Solutions] Removes deprecation and more copied code between security solutions and lists plugin (#100150)
## Summary * Removes deprecations * Removes duplicated code ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios |
||
|
Ryland Herrick
|
b5ae056ac4
|
[Security Solution][Detections] ML Rules accept multiple ML Job IDs (#97073)
* Adds helper to normalize legacy ML rule field to an array This will be used on read of rules, to normalize legacy rules while avoiding an explicit migration. * Fix our detection-specific ML search function Luckily this was just a translation layer to our anomaly call, and the underlying functions already accepted an array of strings. * WIP: Run rules against multiple ML Job IDs We don't yet support creation of rules with multiple job ids, either on the API or the UI, but when we do they will work. Note: the logic was previously to generate an error if the underlying job was not running, but to still query and generate alerts. Extending that logic to multiple jobs: if any are not running, we generate an error but continue querying and generating alerts. * WIP: updating ml rule schemas to support multiple job IDs * Simplify normalization method We don't care about null or empty string values here; those were holdovers from copying the logic of normalizeThreshold and don't apply to this situation. * Move normalized types to separate file to fix circular dependency Our use of NonEmptyArray within common/schemas seemed to be causing the above; this fixes it for now. * Normalize ML job_ids param at the API layer Previous changes to the base types already covered the majority of routes; this updates the miscellaneous helpers that don't leverage those shared utilities. At the DB level, the forthcoming migration will ensure that we always have "normalized" job IDs as an array. * Count stopped ML Jobs as partial failure during ML Rule execution Since we continue to query anomalies and potentially generate alerts, a "failure" status is no longer the most accurate for this situation. * Update 7.13 alerts migration to allow multi-job ML Rules This ensures that we can assume string[] for this field during rule execution. * Display N job statuses on rule details * WIP: converts MLJobSelect to a multiselect Unfortunately, the SuperSelect does not allow multiselect so we need to convert this to a combobox. Luckily we can reuse most of the code here and remain relatively clean. Since all combobox options must be the same (fixed) height, we're somewhat more limited than before for displaying the rows. The truncation appears fine, but I need to figure out a way to display the full description as well. * Update client-side logic to handle an array of ML job_ids * Marginally more legible error message * Conditionally call our normalize helper only if we have a value This fixes a type error where TS could not infer that the return value would not be undefined despite knowing that the argument was never undefined. I tried some fancy conditional generic types, but that didn't work. This is more analogous to normalizeThresholdObject now, anyway. * Fix remaining type error * Clean up our ML executor tests with existing contract mocks * Update ML Executor tests with new logic We now record a partial failure instead of an error. * Add and update tests for new ML normalization logic * Add and update integration tests for ML Rules Ensures that dealing with legacy job formats continues to work in the API. * Fix a type error These params can no longer be strings. * Update ML cypress test to create a rule with 2 ML jobs If we can create a rule with 2 jobs, we should also be able to create a rule with 1 job. * Remove unused constant * Persist a partial failure message written by a rule executor We added the result.warning field as a way to indicate that a partial failure was written to the rule, but neglected to account for that in the main rule execution code, which caused a success status to immediately overwrite the partial failure if the rule execution did not otherwise fail/short-circuit. |
||
Madison Caldwell
|
cb053f4672
|
[Security Solution][Detections][7.12] Critical Threshold Rule Fixes (#92667)
* Threshold cardinality validation * Remove comments * Fix legacy threshold signal dupe mitigation * Add find_threshold_signals tests * remove comment * bug fixes * Fix edit form value initialization for cardinality_value * Fix test * Type and test fixes * Tests/types * Reenable threshold cypress test * Schema fixes * Types and tests, normalize threshold field util * Continue cleaning up types * Some more pre-7.12 tests * Limit cardinality_field to length 1 for now * Cardinality to array * Cardinality to array * Tests/types * cardinality can be null * Handle empty threshold field in bulk_create_threshold_signals * Remove cardinality_field, cardinality_value |
||
Brandon Kobel
|
4584a8b570
|
Elastic License 2.0 (#90099)
* Updating everything except the license headers themselves * Applying ESLint rules * Manually replacing the stragglers |
||
Davis Plumlee
|
6a173ba19d
|
[Security Solution][Detections] Adds sequence callout in the exceptions modals for eql rule types (#79007) | ||
|
Frank Hassanabad
|
496152eea7
|
[Security Solutions][Detection Engine] Adds threat matching API and rule type (#77395)
## Summary This is the backend, first iteration of threat matching API and rule type. You see elements using the backend API on the front end but cannot use the UI to add or edit a threshold rule with this PR. Screen shots of it running in the UI elements that do work: <img width="1862" alt="Screen Shot 2020-09-16 at 10 34 26 AM" src="https://user-images.githubusercontent.com/1151048/93366465-6e2b9c00-f808-11ea-923b-78e8d0fdfbaa.png"> <img width="1863" alt="Screen Shot 2020-09-16 at 10 34 48 AM" src="https://user-images.githubusercontent.com/1151048/93366476-71268c80-f808-11ea-8247-d2091ff1599a.png"> **Usage** Since this is only backend API work and does not have the front end add/edit at the moment, you can use the existing UI's (for the most part) to validate the work here through CURL scripts below: Go to the folder: ```ts /kibana/x-pack/plugins/security_solution/server/lib/detection_engine/scripts ``` And post a small ECS threat mapping to the index called `mock-threat-list`: ```ts ./create_threat_mapping.sh ``` Then to post a small number of threats that represent simple port numbers you can run: ```ts ./create_threat_data.sh ``` However, feel free to also manually create them directly in your dev tools like so: ```ts # Posts a threat list item called some-name with an IP but change these out for valid data in your system PUT mock-threat-list-1/_doc/9999 { "@timestamp": "2020-09-09T20:30:45.725Z", "host": { "name": "some-name", "ip": "127.0.0.1" } } ``` ```ts # Posts a destination port number to watch PUT mock-threat-list-1/_doc/10000 { "@timestamp": "2020-09-08T20:30:45.725Z", "destination": { "port": "443" } } ``` ```ts # Posts a source port number to watch PUT mock-threat-list-1/_doc/10001 { "@timestamp": "2020-09-08T20:30:45.725Z", "source": { "port": "443" } } ``` Then you can post a threat match rule: ```ts ./post_rule.sh ./rules/queries/query_with_threat_mapping.json ``` <details> <summary>Click here to see Response</summary> ```ts { "actions": [], "author": [], "created_at": "2020-09-16T04:25:58.041Z", "created_by": "yo", "description": "Query with a threat mapping", "enabled": true, "exceptions_list": [], "false_positives": [], "from": "now-6m", "id": "f4226ab0-6f88-49c3-8f09-84cf5946ee7a", "immutable": false, "interval": "5m", "language": "kuery", "max_signals": 100, "name": "Query with a threat mapping", "output_index": ".siem-signals-hassanabad3-default", "query": "*:*", "references": [], "risk_score": 1, "risk_score_mapping": [], "rule_id": "threat-mapping", "severity": "high", "severity_mapping": [], "tags": [ "tag_1", "tag_2" ], "threat": [], "threat_index": "mock-threat-list-1", "threat_mapping": [ { "entries": [ { "field": "host.name", "type": "mapping", "value": "host.name" }, { "field": "host.ip", "type": "mapping", "value": "host.ip" } ] }, { "entries": [ { "field": "destination.ip", "type": "mapping", "value": "destination.ip" }, { "field": "destination.port", "type": "mapping", "value": "destination.port" } ] }, { "entries": [ { "field": "source.port", "type": "mapping", "value": "source.port" } ] }, { "entries": [ { "field": "source.ip", "type": "mapping", "value": "source.ip" } ] } ], "threat_query": "*:*", "throttle": "no_actions", "to": "now", "type": "threat_match", "updated_at": "2020-09-16T04:25:58.051Z", "updated_by": "yo", "version": 1 } ``` </details> **Structure** You can see the rule structure in the file: ```ts x-pack/plugins/security_solution/server/lib/detection_engine/scripts/rules/queries/query_with_threat_mapping.json ``` <details> <summary>Click here to see JSON</summary> ```ts { "name": "Query with a threat mapping", "description": "Query with a threat mapping", "rule_id": "threat-mapping", "risk_score": 1, "severity": "high", "type": "threat_match", "query": "*:*", "tags": ["tag_1", "tag_2"], "threat_index": "mock-threat-list", "threat_query": "*:*", "threat_mapping": [ { "entries": [ { "field": "host.name", "type": "mapping", "value": "host.name" }, { "field": "host.ip", "type": "mapping", "value": "host.ip" } ] }, { "entries": [ { "field": "destination.ip", "type": "mapping", "value": "destination.ip" }, { "field": "destination.port", "type": "mapping", "value": "destination.port" } ] }, { "entries": [ { "field": "source.port", "type": "mapping", "value": "source.port" } ] }, { "entries": [ { "field": "source.ip", "type": "mapping", "value": "source.ip" } ] } ] } ``` </details> Structural elements that are new: New type enum called "threat_match" ```ts "type": "threat_match", ``` New `threat_index` string which can be set to a single threat index (This might change to an array in the near future before release): ```ts "threat_index": "mock-threat-list" ``` New `threat_query` string which can be set any valid query to filter the threat list before executing the rule. This can be undefined, if you are only pushing in filters from the API. ```ts "threat_query": "*:*", ``` New `threat_filters` array which can be set to any valid filter like `filters`. This can be `undefined` if you are only using the query from the API. ```ts threat_filter": [] ``` New `threat_mapping` array which can be set to a valid mapping between the threat list and the ECS list. This structure has an inner array called `entries` which represent a 2 level tree of 1st level OR elements followed by 2nd level AND elements. For example, if you want to find all threat matches where ECS documents will match against some ${threatList} index where it would be like so: <details> <summary>Click here to see array from the boolean</summary> ```ts "threat_mapping": [ { "entries": [ { "field": "host.name", "type": "mapping", "value": "host.name" }, { "field": "host.ip", "type": "mapping", "value": "host.ip" } ] }, { "entries": [ { "field": "destination.ip", "type": "mapping", "value": "destination.ip" }, { "field": "destination.port", "type": "mapping", "value": "destination.port" } ] }, { "entries": [ { "field": "source.port", "type": "mapping", "value": "source.port" } ] }, { "entries": [ { "field": "source.ip", "type": "mapping", "value": "source.ip" } ] } ] ``` </details> What that array represents in pseudo boolean logic is: <details> <summary>Click here to see pseduo logic</summary> ```ts (host.name: ${threatList.host.name} AND host.ip: ${threatList.host.name}) OR (destination.ip: ${threatList.destination.ip} AND destination.port: ${threatList.destination.port}) OR (source.port ${threatList.source.port}) OR (source.ip ${threatList.source.ip}) ``` </details> ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios |
||
Yara Tercero
|
3c9fa99d68
|
[Security Solution][Detection Engine] - Update exceptions logic (#71512)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Yara Tercero <yara.tercero@elastic.co> |
