mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-04-25 15:47:23 -04:00
237 lines
7.7 KiB
Text
237 lines
7.7 KiB
Text
[role="xpack"]
|
||
[[ml-configuring-detector-custom-rules]]
|
||
= Customizing detectors with custom rules
|
||
|
||
<<ml-ad-rules,Custom rules>> – or _job rules_ as {kib} refers to them – enable you
|
||
to change the behavior of anomaly detectors based on domain-specific knowledge.
|
||
|
||
Custom rules describe _when_ a detector should take a certain _action_ instead
|
||
of following its default behavior. To specify the _when_ a rule uses
|
||
a `scope` and `conditions`. You can think of `scope` as the categorical
|
||
specification of a rule, while `conditions` are the numerical part.
|
||
A rule can have a scope, one or more conditions, or a combination of
|
||
scope and conditions. For the full list of specification details, see the
|
||
{ref}/ml-put-job.html#put-customrules[`custom_rules` object] in the create
|
||
{anomaly-jobs} API.
|
||
|
||
[[ml-custom-rules-scope]]
|
||
== Specifying custom rule scope
|
||
|
||
Let us assume we are configuring an {anomaly-job} in order to detect DNS data
|
||
exfiltration. Our data contain fields "subdomain" and "highest_registered_domain".
|
||
We can use a detector that looks like
|
||
`high_info_content(subdomain) over highest_registered_domain`. If we run such a
|
||
job, it is possible that we discover a lot of anomalies on frequently used
|
||
domains that we have reasons to trust. As security analysts, we are not
|
||
interested in such anomalies. Ideally, we could instruct the detector to skip
|
||
results for domains that we consider safe. Using a rule with a scope allows us
|
||
to achieve this.
|
||
|
||
First, we need to create a list of our safe domains. Those lists are called
|
||
_filters_ in {ml}. Filters can be shared across {anomaly-jobs}.
|
||
|
||
You can create a filter in **Anomaly Detection > Settings > Filter Lists** in
|
||
{kib} or by using the {ref}/ml-put-filter.html[put filter API]:
|
||
|
||
[source,console]
|
||
----------------------------------
|
||
PUT _ml/filters/safe_domains
|
||
{
|
||
"description": "Our list of safe domains",
|
||
"items": ["safe.com", "trusted.com"]
|
||
}
|
||
----------------------------------
|
||
// TEST[skip:needs-licence]
|
||
|
||
Now, we can create our {anomaly-job} specifying a scope that uses the
|
||
`safe_domains` filter for the `highest_registered_domain` field:
|
||
|
||
[source,console]
|
||
----------------------------------
|
||
PUT _ml/anomaly_detectors/dns_exfiltration_with_rule
|
||
{
|
||
"analysis_config" : {
|
||
"bucket_span":"5m",
|
||
"detectors" :[{
|
||
"function":"high_info_content",
|
||
"field_name": "subdomain",
|
||
"over_field_name": "highest_registered_domain",
|
||
"custom_rules": [{
|
||
"actions": ["skip_result"],
|
||
"scope": {
|
||
"highest_registered_domain": {
|
||
"filter_id": "safe_domains",
|
||
"filter_type": "include"
|
||
}
|
||
}
|
||
}]
|
||
}]
|
||
},
|
||
"data_description" : {
|
||
"time_field":"timestamp"
|
||
}
|
||
}
|
||
----------------------------------
|
||
// TEST[skip:needs-licence]
|
||
|
||
As time advances and we see more data and more results, we might encounter new
|
||
domains that we want to add in the filter. We can do that in the
|
||
**Anomaly Detection > Settings > Filter Lists** in {kib} or by using the
|
||
{ref}/ml-update-filter.html[update filter API]:
|
||
|
||
[source,console]
|
||
----------------------------------
|
||
POST _ml/filters/safe_domains/_update
|
||
{
|
||
"add_items": ["another-safe.com"]
|
||
}
|
||
----------------------------------
|
||
// TEST[skip:setup:ml_filter_safe_domains]
|
||
|
||
Note that we can use any of the `partition_field_name`, `over_field_name`, or
|
||
`by_field_name` fields in the `scope`.
|
||
|
||
In the following example we scope multiple fields:
|
||
|
||
[source,console]
|
||
----------------------------------
|
||
PUT _ml/anomaly_detectors/scoping_multiple_fields
|
||
{
|
||
"analysis_config" : {
|
||
"bucket_span":"5m",
|
||
"detectors" :[{
|
||
"function":"count",
|
||
"partition_field_name": "my_partition",
|
||
"over_field_name": "my_over",
|
||
"by_field_name": "my_by",
|
||
"custom_rules": [{
|
||
"actions": ["skip_result"],
|
||
"scope": {
|
||
"my_partition": {
|
||
"filter_id": "filter_1"
|
||
},
|
||
"my_over": {
|
||
"filter_id": "filter_2"
|
||
},
|
||
"my_by": {
|
||
"filter_id": "filter_3"
|
||
}
|
||
}
|
||
}]
|
||
}]
|
||
},
|
||
"data_description" : {
|
||
"time_field":"timestamp"
|
||
}
|
||
}
|
||
----------------------------------
|
||
// TEST[skip:needs-licence]
|
||
|
||
Such a detector skips results when the values of all three scoped fields are
|
||
included in the referenced filters.
|
||
|
||
[[ml-custom-rules-conditions]]
|
||
== Specifying custom rule conditions
|
||
|
||
Imagine a detector that looks for anomalies in CPU utilization. Given a machine
|
||
that is idle for long enough, small movement in CPU could result in anomalous
|
||
results where the `actual` value is quite small, for example, 0.02. Given our
|
||
knowledge about how CPU utilization behaves we might determine that anomalies
|
||
with such small actual values are not interesting for investigation.
|
||
|
||
Let us now configure an {anomaly-job} with a rule that skips results where CPU
|
||
utilization is less than 0.20.
|
||
|
||
[source,console]
|
||
----------------------------------
|
||
PUT _ml/anomaly_detectors/cpu_with_rule
|
||
{
|
||
"analysis_config" : {
|
||
"bucket_span":"5m",
|
||
"detectors" :[{
|
||
"function":"high_mean",
|
||
"field_name": "cpu_utilization",
|
||
"custom_rules": [{
|
||
"actions": ["skip_result"],
|
||
"conditions": [
|
||
{
|
||
"applies_to": "actual",
|
||
"operator": "lt",
|
||
"value": 0.20
|
||
}
|
||
]
|
||
}]
|
||
}]
|
||
},
|
||
"data_description" : {
|
||
"time_field":"timestamp"
|
||
}
|
||
}
|
||
----------------------------------
|
||
// TEST[skip:needs-licence]
|
||
|
||
When there are multiple conditions they are combined with a logical `AND`. This
|
||
is useful when we want the rule to apply to a range. We create a rule with two
|
||
conditions, one for each end of the desired range.
|
||
|
||
Here is an example where a count detector skips results when the count is
|
||
greater than 30 and less than 50:
|
||
|
||
[source,console]
|
||
----------------------------------
|
||
PUT _ml/anomaly_detectors/rule_with_range
|
||
{
|
||
"analysis_config" : {
|
||
"bucket_span":"5m",
|
||
"detectors" :[{
|
||
"function":"count",
|
||
"custom_rules": [{
|
||
"actions": ["skip_result"],
|
||
"conditions": [
|
||
{
|
||
"applies_to": "actual",
|
||
"operator": "gt",
|
||
"value": 30
|
||
},
|
||
{
|
||
"applies_to": "actual",
|
||
"operator": "lt",
|
||
"value": 50
|
||
}
|
||
]
|
||
}]
|
||
}]
|
||
},
|
||
"data_description" : {
|
||
"time_field":"timestamp"
|
||
}
|
||
}
|
||
----------------------------------
|
||
// TEST[skip:needs-licence]
|
||
|
||
[[ml-custom-rules-lifecycle]]
|
||
== Custom rules in the lifecycle of a job
|
||
|
||
Custom rules only affect results created after the rules were applied. Let us
|
||
imagine that we have configured an {anomaly-job} and it has been running for
|
||
some time. After observing its results, we decide that we can employ rules to
|
||
get rid of some uninteresting results. We can use the
|
||
{ref}/ml-update-job.html[update {anomaly-job} API] to do so. However, the rule
|
||
we added will only be in effect for any results created from the moment we
|
||
added the rule onwards. Past results remain unaffected.
|
||
|
||
[[ml-custom-rules-filtering]]
|
||
== Using custom rules vs. filtering data
|
||
|
||
It might appear like using rules is just another way of filtering the data that
|
||
feeds into an {anomaly-job}. For example, a rule that skips results when the
|
||
partition field value is in a filter sounds equivalent to having a query that
|
||
filters out such documents. However, there is a fundamental difference. When the
|
||
data is filtered before reaching a job, it is as if they never existed for the
|
||
job. With rules, the data still reaches the job and affects its behavior
|
||
(depending on the rule actions).
|
||
|
||
For example, a rule with the `skip_result` action means all data is still
|
||
modeled. On the other hand, a rule with the `skip_model_update` action means
|
||
results are still created even though the model is not updated by data matched
|
||
by a rule.
|