## Summary
Fixes: https://github.com/elastic/kibana/issues/114535
**What this linter rule does:**
* Sets the [@typescript-eslint/no-non-null-assertion](https://github.com/typescript-eslint/typescript-eslint/blob/master/packages/eslint-plugin/docs/rules/no-non-null-assertion.md) linter rule to become an error if seen.
If you try to use the `!` operator you get an error and nice helper message that tries to encourage better practices such as this one:
<img width="1635" alt="Screen Shot 2021-10-07 at 11 26 14 AM" src="https://user-images.githubusercontent.com/1151048/136474207-f38d3461-0af9-4cdc-885b-632cb49d8a24.png">
**Why are we deciding to set this linter rule?**
* Recommended from Kibana [styleguide](https://github.com/elastic/kibana/blob/master/STYLEGUIDE.mdx#avoid-non-null-assertions) for ~2 years now and still recommended.
* A lot of TypeScript has evolved and has operators such as `?` which can replace the `!` in most cases. Other cases can use a throw explicitly or other ways to manage this.
* Some types can change instead of using this operator and we should just change the types.
* TypeScript flows have improved and when we upgrade the linter will cause errors where we can remove the `!` operator which is 👍 better than leaving them when they're not needed anymore.
* Newer programmers and team members sometimes mistake it for the `?` when it is not the same thing.
* We have had past bugs and bugs recently because of these.
* It's easier to use the linter to find bugs than to rely on manual tests.
**How did Frank fix all the 403 areas in which the linter goes off?**
* Anywhere I could remove the `!` operator without side effects or type script errors I just removed the `!` operator.
* Anywhere in test code where I could replace the code with a `?` or a `throw` I went through that route.
* Within the code areas (non test code) where I saw what looks like a simple bug that I could fix using a `?? []` or `?` operator or `String(value)` or fixing a simple type I would choose that route first. These areas I marked in the code review with the words `NOTE:` for anyone to look at.
* Within all other areas of the code and tests where anything looked more involved I just disabled the linter for that particular line. When in doubt I chose this route.
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Co-authored-by: Frank Hassanabad <frank.hassanabad@elastic.co>
## Summary
Utilizes constants package and deletes duplicate code
* Renames the `securitysolution-constants` to be `securitysolution-list-constants` to be specific
* Deletes duplicated code found during cleanup
* Moves more tests into the packages found along the way with the duplicated code
* Moves `parseScheduleDates` from `@kbn/securitysolution-io-ts-types` to `@kbn/securitysolution-io-ts-utils`
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Co-authored-by: Frank Hassanabad <frank.hassanabad@elastic.co>
## Summary
* Creates a `securitysolution-list-utils` packaged and moves the first set of utilities into there
* Fixes a slight bug with `kbn-securitysolution-io-ts-list-types` where the wrong name was used
* Moves _all_ of the lists schemas and types into the package `kbn-securitysolution-io-ts-list-types`
* Removes copied code found in a few places
## Tech debt
* Some spots I have to use an `any` in the package as Kibana kbn packages don't have the types I need
* Some spots I copy constants until we can straighten out those pieces.
* I keep copied mock files until we figure out how to share mocks from these packages without adding weight or we create dedicated mock packages for all of this.
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Co-authored-by: Frank Hassanabad <frank.hassanabad@elastic.co>
## Summary
* Removes deprecations
* Removes duplicated code
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Co-authored-by: Frank Hassanabad <frank.hassanabad@elastic.co>
* Adds helper to normalize legacy ML rule field to an array
This will be used on read of rules, to normalize legacy rules while
avoiding an explicit migration.
* Fix our detection-specific ML search function
Luckily this was just a translation layer to our anomaly call, and the
underlying functions already accepted an array of strings.
* WIP: Run rules against multiple ML Job IDs
We don't yet support creation of rules with multiple job ids, either on
the API or the UI, but when we do they will work.
Note: the logic was previously to generate an error if the underlying
job was not running, but to still query and generate alerts. Extending
that logic to multiple jobs: if any are not running, we generate an
error but continue querying and generating alerts.
* WIP: updating ml rule schemas to support multiple job IDs
* Simplify normalization method
We don't care about null or empty string values here; those were
holdovers from copying the logic of normalizeThreshold and don't apply
to this situation.
* Move normalized types to separate file to fix circular dependency
Our use of NonEmptyArray within common/schemas seemed to be causing the
above; this fixes it for now.
* Normalize ML job_ids param at the API layer
Previous changes to the base types already covered the majority of
routes; this updates the miscellaneous helpers that don't leverage those
shared utilities.
At the DB level, the forthcoming migration will ensure that we always
have "normalized" job IDs as an array.
* Count stopped ML Jobs as partial failure during ML Rule execution
Since we continue to query anomalies and potentially generate alerts, a
"failure" status is no longer the most accurate for this situation.
* Update 7.13 alerts migration to allow multi-job ML Rules
This ensures that we can assume string[] for this field during rule
execution.
* Display N job statuses on rule details
* WIP: converts MLJobSelect to a multiselect
Unfortunately, the SuperSelect does not allow multiselect so we need to
convert this to a combobox. Luckily we can reuse most of the code here
and remain relatively clean.
Since all combobox options must be the same (fixed) height, we're
somewhat more limited than before for displaying the rows. The
truncation appears fine, but I need to figure out a way to display the
full description as well.
* Update client-side logic to handle an array of ML job_ids
* Marginally more legible error message
* Conditionally call our normalize helper only if we have a value
This fixes a type error where TS could not infer that the return value
would not be undefined despite knowing that the argument was never
undefined. I tried some fancy conditional generic types, but that didn't
work.
This is more analogous to normalizeThresholdObject now, anyway.
* Fix remaining type error
* Clean up our ML executor tests with existing contract mocks
* Update ML Executor tests with new logic
We now record a partial failure instead of an error.
* Add and update tests for new ML normalization logic
* Add and update integration tests for ML Rules
Ensures that dealing with legacy job formats continues to work in the
API.
* Fix a type error
These params can no longer be strings.
* Update ML cypress test to create a rule with 2 ML jobs
If we can create a rule with 2 jobs, we should also be able to create a
rule with 1 job.
* Remove unused constant
* Persist a partial failure message written by a rule executor
We added the result.warning field as a way to indicate that a partial
failure was written to the rule, but neglected to account for that in the
main rule execution code, which caused a success status to immediately
overwrite the partial failure if the rule execution did not otherwise
fail/short-circuit.
Co-authored-by: Ryland Herrick <ryalnd@gmail.com>
* Replace validation function in signal executor
* Remove more RuleTypeParams usage
* Add security solution rules migration to alerting plugin
* Handle and test null value in threshold.field
* Remove runtime normalization of threshold field
* Remove signalParamsSchema
Co-authored-by: Davis Plumlee <davis.plumlee@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com>
Co-authored-by: Davis Plumlee <davis.plumlee@elastic.co>
* fix validation on response of find status route when rule has a partial failure status
* replaces warning in rule status service with partial failure to maintain backwards compatibility from an API standpoint, also displays 'warning' on UI if a rule's status is partial failure
* display partial failure as 'warning' on all rules table and update e2e test to check for partial failure not warning
* add util function, show 'warning' on monitoring table, fix e2e tests
Co-authored-by: Devin W. Hurley <devin.hurley@elastic.co>
* Threshold cardinality validation
* Remove comments
* Fix legacy threshold signal dupe mitigation
* Add find_threshold_signals tests
* remove comment
* bug fixes
* Fix edit form value initialization for cardinality_value
* Fix test
* Type and test fixes
* Tests/types
* Reenable threshold cypress test
* Schema fixes
* Types and tests, normalize threshold field util
* Continue cleaning up types
* Some more pre-7.12 tests
* Limit cardinality_field to length 1 for now
* Cardinality to array
* Cardinality to array
* Tests/types
* cardinality can be null
* Handle empty threshold field in bulk_create_threshold_signals
* Remove cardinality_field, cardinality_value
* Add new field for overriding threat indicator path
There is no UI for this currently, nor is it used during rule execution.
* Adds form field for indicator path parameter
Also adds missing plumbing that was preventing the new field from being
persisted to the alert/returned in the response.
* Wire up our indicator path config to enrichment
* Add unit test for enriching from a custom indicator path
We always persist to `threat.indicator.*` on the signal, but this allows
users to specify where the enrichment fields can be found on the matched
indicator document.
* Wire up the missing piece of our indicator path config
We were not passing this from the rule itself into the threat matching
logic, and so were merely getting the default value.
An integration test will fix this. Incoming!
* Move indicator path defaulting outside of helper functions
This happens closer to where we pass data from the rule to our helpers,
and will prevent errors/bugs due to defaulting logic down the road.
It makes tests a little more verbose, but that's okay.
* Fix remaining type errors around new rule field
* Make threat indicator path a conditional field
Always sending along this field, but only allowing it for threat match
rules was implicitly breaking the workflow of otther rule types. By
making the field conditional on the rule type, this field only impacts
threat match rules.
This also fixes some types and tests accordingly.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* WIP: Adding new route for EQL Validation
This is mostly boilerplate with some rough parameter definitions; the
actual implementation of the validation is going to live in our
validateEql function.
A few tests are failing as the mocks haven't yet been implemented, I
need to see the shape of the responses first.
* Cherry-pick Marshall's EQL types
* Implements actual EQL validation
* Performs an EQL search
* filters out non-parsing errors, and returns what remains in the
response
* Adds mocks for empty EQL responses (we don't yet have a need for
mocked data, but we will when we unit-test validateEql)
* Adds validation calls to the EQL form input
* Adds EQL Validation response schema,mocks,tests
* Adds frontend function to call our validation endpoint
* Adds hook, useEqlValidation, to call the above function and return
state
* Adds labels/help text for EQL Query bar
* EqlQueryBar consumes useEqlValidation and marks the field as invalid,
but does not yet report errors.
* Do not call the validation API if query is not present
This causes a broader error that results in a 400 response; we can (and
do) handle the case of a blank query in the form itself.
* Remove EQL Help Text
It doesn't add any information for the user, and it currently looks bad
when combined with validation errors.
* Flesh out and use our popover for displaying validation errors
* Fixes issue where old errors were persisted after the user had made
modifications
* Include verification_exception errors as validation errors
These include errors related to index fields and mappings.
* Generalize our validation helpers
We're concerned with validation errors; the source of those errors is an
implementation detail of these functions.
* Move error popover and EQL reference link to footer
This more closely resembles the new Eui Markdown editor, which places
errors and doc links in a footer.
* Fix jest tests following additional prop
* Add icon for EQL Rule card
* Fixes existing EqlQueryBar tests
These were broken by our use of useAppToasts and the EUI theme.
* Add unit tests around error rendering on EQL Query Bar
* Add tests for ErrorPopover
* Remove unused schema type
Decode doesn't do any additional processing, so we can use t.TypeOf here
(the default for buildRouteValidation).
* Remove duplicated header
* Use ignore parameter to prevent EQL validations from logging errors
Without `ignore: [400]` the ES client will log errors and then throw
them. We can catch the error, but the log is undesirable.
This updates the query to use the ignore parameter, along with updating
the validation logic to work with the updated response.
Adds some mocks and tests around these responses and helpers, since
these will exist independent of the validation implementation.
* Include mapping_exceptions during EQL query validation
These include errors for inaccessible indexes, which should be useful to
the rule writer in writing their EQL query.
* Display toast messages for non-validation messages
* fix type errors
This type was renamed.
* Do not request data in our validation request
By not having the cluster retrieve/send any data, this should saves us
a few CPU cycles.
* Move EQL validation to an async form validator
Rather than invoking a custom validation hook (useEqlValidation) at custom times (onBlur) in our EqlQueryBar
component, we can instead move this functionality to a form validation
function and have it be invoked automatically by our form when values
change. However, because we still need to handle the validation messages
slightly differently (place them in a popover as opposed to an
EuiFormRow), we also need custom error retrieval in the form of
getValidationResults.
After much pain, it was determined that the default behavior of
_.debounce does not work with async validator functions, as a debounced
call will not "wait" for the eventual invocation but will instead return
the most recently resolved value. This leads to stale validation
results and terrible UX, so I wrote a custom function (debounceAsync)
that behaves like we want/need; see tests for details.
* Invalidate our query field when index patterns change
Since EQL rules actually validate against the relevant indexes, changing
said indexes should invalidate/revalidate the query.
With the form lib, this is beautifully simple :)
* Set a min-height on our EQL textarea
* Remove unused prop from EqlQueryBar
Index corresponds to the value from the index field; now that our EQL
validation is performed by the form we have no need for it here.
* Update EQL overview link to point to elasticsearch docs
Adds an entry in our doclinks service, and uses that.
* Remove unused prop from stale tests
* Update docLinks documentation with new EQL link
* Fix bug where saved query rules had no type selected on Edit
* Wait for kibana requests to complete before moving between rule tabs
With our new async validation, a user can quickly navigate away from the
Definition tab before the validation has completed, resulting in the
form being invalidated. Any subsequent user actions cause the form to
correct itself, but until I can find a better solution here this really
just gives the validation time to complete and sidesteps the issue.
## Summary
This is the backend, first iteration of threat matching API and rule type. You see elements using the backend API on the front end but cannot use the UI to add or edit a threshold rule with this PR.
Screen shots of it running in the UI elements that do work:
<img width="1862" alt="Screen Shot 2020-09-16 at 10 34 26 AM" src="https://user-images.githubusercontent.com/1151048/93366465-6e2b9c00-f808-11ea-923b-78e8d0fdfbaa.png">
<img width="1863" alt="Screen Shot 2020-09-16 at 10 34 48 AM" src="https://user-images.githubusercontent.com/1151048/93366476-71268c80-f808-11ea-8247-d2091ff1599a.png">
**Usage**
Since this is only backend API work and does not have the front end add/edit at the moment, you can use the existing UI's (for the most part) to validate the work here through CURL scripts below:
Go to the folder:
```ts
/kibana/x-pack/plugins/security_solution/server/lib/detection_engine/scripts
```
And post a small ECS threat mapping to the index called `mock-threat-list`:
```ts
./create_threat_mapping.sh
```
Then to post a small number of threats that represent simple port numbers you can run:
```ts
./create_threat_data.sh
```
However, feel free to also manually create them directly in your dev tools like so:
```ts
# Posts a threat list item called some-name with an IP but change these out for valid data in your system
PUT mock-threat-list-1/_doc/9999
{
"@timestamp": "2020-09-09T20:30:45.725Z",
"host": {
"name": "some-name",
"ip": "127.0.0.1"
}
}
```
```ts
# Posts a destination port number to watch
PUT mock-threat-list-1/_doc/10000
{
"@timestamp": "2020-09-08T20:30:45.725Z",
"destination": {
"port": "443"
}
}
```
```ts
# Posts a source port number to watch
PUT mock-threat-list-1/_doc/10001
{
"@timestamp": "2020-09-08T20:30:45.725Z",
"source": {
"port": "443"
}
}
```
Then you can post a threat match rule:
```ts
./post_rule.sh ./rules/queries/query_with_threat_mapping.json
```
<details>
<summary>Click here to see Response</summary>
```ts
{
"actions": [],
"author": [],
"created_at": "2020-09-16T04:25:58.041Z",
"created_by": "yo",
"description": "Query with a threat mapping",
"enabled": true,
"exceptions_list": [],
"false_positives": [],
"from": "now-6m",
"id": "f4226ab0-6f88-49c3-8f09-84cf5946ee7a",
"immutable": false,
"interval": "5m",
"language": "kuery",
"max_signals": 100,
"name": "Query with a threat mapping",
"output_index": ".siem-signals-hassanabad3-default",
"query": "*:*",
"references": [],
"risk_score": 1,
"risk_score_mapping": [],
"rule_id": "threat-mapping",
"severity": "high",
"severity_mapping": [],
"tags": [
"tag_1",
"tag_2"
],
"threat": [],
"threat_index": "mock-threat-list-1",
"threat_mapping": [
{
"entries": [
{
"field": "host.name",
"type": "mapping",
"value": "host.name"
},
{
"field": "host.ip",
"type": "mapping",
"value": "host.ip"
}
]
},
{
"entries": [
{
"field": "destination.ip",
"type": "mapping",
"value": "destination.ip"
},
{
"field": "destination.port",
"type": "mapping",
"value": "destination.port"
}
]
},
{
"entries": [
{
"field": "source.port",
"type": "mapping",
"value": "source.port"
}
]
},
{
"entries": [
{
"field": "source.ip",
"type": "mapping",
"value": "source.ip"
}
]
}
],
"threat_query": "*:*",
"throttle": "no_actions",
"to": "now",
"type": "threat_match",
"updated_at": "2020-09-16T04:25:58.051Z",
"updated_by": "yo",
"version": 1
}
```
</details>
**Structure**
You can see the rule structure in the file:
```ts
x-pack/plugins/security_solution/server/lib/detection_engine/scripts/rules/queries/query_with_threat_mapping.json
```
<details>
<summary>Click here to see JSON</summary>
```ts
{
"name": "Query with a threat mapping",
"description": "Query with a threat mapping",
"rule_id": "threat-mapping",
"risk_score": 1,
"severity": "high",
"type": "threat_match",
"query": "*:*",
"tags": ["tag_1", "tag_2"],
"threat_index": "mock-threat-list",
"threat_query": "*:*",
"threat_mapping": [
{
"entries": [
{
"field": "host.name",
"type": "mapping",
"value": "host.name"
},
{
"field": "host.ip",
"type": "mapping",
"value": "host.ip"
}
]
},
{
"entries": [
{
"field": "destination.ip",
"type": "mapping",
"value": "destination.ip"
},
{
"field": "destination.port",
"type": "mapping",
"value": "destination.port"
}
]
},
{
"entries": [
{
"field": "source.port",
"type": "mapping",
"value": "source.port"
}
]
},
{
"entries": [
{
"field": "source.ip",
"type": "mapping",
"value": "source.ip"
}
]
}
]
}
```
</details>
Structural elements that are new:
New type enum called "threat_match"
```ts
"type": "threat_match",
```
New `threat_index` string which can be set to a single threat index (This might change to an array in the near future before release):
```ts
"threat_index": "mock-threat-list"
```
New `threat_query` string which can be set any valid query to filter the threat list before executing the rule. This can be undefined, if you are only pushing in filters from the API.
```ts
"threat_query": "*:*",
```
New `threat_filters` array which can be set to any valid filter like `filters`. This can be `undefined` if you are only using the query from the API.
```ts
threat_filter": []
```
New `threat_mapping` array which can be set to a valid mapping between the threat list and the ECS list. This structure has an inner array called `entries` which represent a 2 level tree of 1st level OR elements followed by 2nd level AND elements.
For example, if you want to find all threat matches where ECS documents will match against some ${threatList} index where it would be like so:
<details>
<summary>Click here to see array from the boolean</summary>
```ts
"threat_mapping": [
{
"entries": [
{
"field": "host.name",
"type": "mapping",
"value": "host.name"
},
{
"field": "host.ip",
"type": "mapping",
"value": "host.ip"
}
]
},
{
"entries": [
{
"field": "destination.ip",
"type": "mapping",
"value": "destination.ip"
},
{
"field": "destination.port",
"type": "mapping",
"value": "destination.port"
}
]
},
{
"entries": [
{
"field": "source.port",
"type": "mapping",
"value": "source.port"
}
]
},
{
"entries": [
{
"field": "source.ip",
"type": "mapping",
"value": "source.ip"
}
]
}
]
```
</details>
What that array represents in pseudo boolean logic is:
<details>
<summary>Click here to see pseduo logic</summary>
```ts
(host.name: ${threatList.host.name} AND host.ip: ${threatList.host.name}) OR
(destination.ip: ${threatList.destination.ip} AND destination.port: ${threatList.destination.port}) OR
(source.port ${threatList.source.port}) OR
(source.ip ${threatList.source.ip})
```
</details>
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* Use existing predicate helper to avoid hardcoded strings
* Render our field components with React.createElement
Without this, we get some bad behaviors:
* Cannot use React.memo'd components
* Cannot switch between UseField components (causes a "change in the
order of hooks" error from React)
* WIP: EQL Rules can be created
WIP because: they're probably not treated well in the UI, and they're certainly not
going to execute properly, and there are no tests.
* Add unit tests for changes to schema + helpers
* Add unit tests for new EQL query input component
It's mostly just a glorified textarea for now.
* Add integration test for EQL Rule creation
* Does not assert the query language, as that is not displayed on Rule
Details
* Does not exercise rule execution
* Use predicate helper
* Throw an error if an EQL Rule is executed
This is to prevent undefined behavior until EQL execution is
implemented.
* Fix failing tests
I changed the default value for the form field mock from an array to a
string; this fixes the few tests that were relying on it being an array.
* Audit our rule statements/switches
I made a pass through our treatment of RuleType to verify that EQL rules
would be treated appropriately. Since the default/fallthrough case is
typically the Query rule, and since this rule has the same
attributes/behavior as the new EQL rule, not much had to change here.
I converted a few if statements to exhaustive switches where possible,
and used predicate helpers in places where it was not.
* Add tests around use of custom components with UseField
There was an issue previously where memoized components would not work;
these are primarily regression tests covering that use case.
* Fix typo
* Add keys to UseField to ensure unmount
When swapping between the Custom Query and EQL rule types, we want to
ensure that the corresponding input component coming from UseField fully
unmounts and remounts with the new component.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Removes RuleTypeSchema in favor of Type for TypeScript. Does break out one function called `parseScheduleDates` into its own file to remove a circular ref issue.
* updates validation on 'from' param to prevent malformed datemath strings from being accepted
* fix imports
* copy paste is not my friend
* missed type check somehow
* forgot to mock common utils
* updates bodies for request validation tests
* Move isThresholdRule predicate into our common folder
This is very similar to isMlRule, which is already used extensively and
lives at this level.
* Disable endpoint association checkbox for ML and Threshold rules
The fullWidth and isDisabled props were not used; what we want is
disabled.
* Fix react warning about nesting buttons
This removes the AdvancedSettingsAccordion in favor of a plain
EuiAccordion with buttonContent, as that seems to be all that's needed
here.
* Disable Exceptions tab on Details for ML or Threshold rules
These rule types do not currently support exceptions.
* Fix type error
Unused import
