mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-06-28 17:34:17 -04:00
6dfd92c46f
* [DOCS] Focus retrieving selected fields on fields parameter * Incorporating changes from reviews * Adding clarifications from review feedback * Slight wording revisions. * Clarify language around format parameter and move text out of callout.
894 lines
25 KiB
Text
894 lines
25 KiB
Text
[role="xpack"]
|
|
[testenv="basic"]
|
|
[[eql]]
|
|
= EQL search
|
|
++++
|
|
<titleabbrev>EQL</titleabbrev>
|
|
++++
|
|
|
|
Event Query Language (EQL) is a query language for event-based time series
|
|
data, such as logs, metrics, and traces.
|
|
|
|
[discrete]
|
|
[[eql-advantages]]
|
|
=== Advantages of EQL
|
|
|
|
* *EQL lets you express relationships between events.* +
|
|
Many query languages allow you to match single events. EQL lets you match a
|
|
sequence of events across different event categories and time spans.
|
|
|
|
* *EQL has a low learning curve.* +
|
|
<<eql-syntax,EQL syntax>> looks like other common query languages, such as SQL.
|
|
EQL lets you write and read queries intuitively, which makes for quick,
|
|
iterative searching.
|
|
|
|
* *EQL is designed for security use cases.* +
|
|
While you can use it for any event-based data, we created EQL for threat
|
|
hunting. EQL not only supports indicator of compromise (IOC) searches but can
|
|
describe activity that goes beyond IOCs.
|
|
|
|
[discrete]
|
|
[[eql-required-fields]]
|
|
=== Required fields
|
|
|
|
To run an EQL search, the searched data stream or index must contain a
|
|
_timestamp_ and _event category_ field. By default, EQL uses the `@timestamp`
|
|
and `event.category` fields from the {ecs-ref}[Elastic Common Schema
|
|
(ECS)]. To use a different timestamp or event category field, see
|
|
<<specify-a-timestamp-or-event-category-field>>.
|
|
|
|
TIP: While no schema is required to use EQL, we recommend using the
|
|
{ecs-ref}[ECS]. EQL searches are designed to work with core ECS fields by
|
|
default.
|
|
|
|
[discrete]
|
|
[[run-an-eql-search]]
|
|
=== Run an EQL search
|
|
|
|
Use the <<eql-search-api,EQL search API>> to run a <<eql-basic-syntax,basic EQL
|
|
query>>.
|
|
|
|
////
|
|
[source,console]
|
|
----
|
|
DELETE /_data_stream/*
|
|
DELETE /_index_template/*
|
|
----
|
|
// TEARDOWN
|
|
////
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-data-stream/_eql/search
|
|
{
|
|
"query": """
|
|
process where process.name == "regsvr32.exe"
|
|
"""
|
|
}
|
|
----
|
|
// TEST[setup:sec_logs]
|
|
|
|
By default, basic EQL queries return the 10 most recent matching events in the
|
|
`hits.events` property. These hits are sorted by timestamp, converted to
|
|
milliseconds since the {wikipedia}/Unix_time[Unix epoch], in ascending order.
|
|
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"is_partial": false,
|
|
"is_running": false,
|
|
"took": 60,
|
|
"timed_out": false,
|
|
"hits": {
|
|
"total": {
|
|
"value": 2,
|
|
"relation": "eq"
|
|
},
|
|
"events": [
|
|
{
|
|
"_index": ".ds-my-data-stream-2099.12.07-000001",
|
|
"_id": "OQmfCaduce8zoHT93o4H",
|
|
"_source": {
|
|
"@timestamp": "2099-12-07T11:07:09.000Z",
|
|
"event": {
|
|
"category": "process",
|
|
"id": "aR3NWVOs",
|
|
"sequence": 4
|
|
},
|
|
"process": {
|
|
"pid": 2012,
|
|
"name": "regsvr32.exe",
|
|
"command_line": "regsvr32.exe /s /u /i:https://...RegSvr32.sct scrobj.dll",
|
|
"executable": "C:\\Windows\\System32\\regsvr32.exe"
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"_index": ".ds-my-data-stream-2099.12.07-000001",
|
|
"_id": "xLkCaj4EujzdNSxfYLbO",
|
|
"_source": {
|
|
"@timestamp": "2099-12-07T11:07:10.000Z",
|
|
"event": {
|
|
"category": "process",
|
|
"id": "GTSmSqgz0U",
|
|
"sequence": 6,
|
|
"type": "termination"
|
|
},
|
|
"process": {
|
|
"pid": 2012,
|
|
"name": "regsvr32.exe",
|
|
"executable": "C:\\Windows\\System32\\regsvr32.exe"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
----
|
|
// TESTRESPONSE[s/"took": 60/"took": $body.took/]
|
|
// TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.events.0._index/]
|
|
// TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.events.0._id/]
|
|
// TESTRESPONSE[s/"_id": "xLkCaj4EujzdNSxfYLbO"/"_id": $body.hits.events.1._id/]
|
|
|
|
Use the `size` parameter to get a smaller or larger set of hits:
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-data-stream/_eql/search
|
|
{
|
|
"query": """
|
|
process where process.name == "regsvr32.exe"
|
|
""",
|
|
"size": 50
|
|
}
|
|
----
|
|
// TEST[setup:sec_logs]
|
|
|
|
[discrete]
|
|
[[eql-search-sequence]]
|
|
=== Search for a sequence of events
|
|
|
|
Use EQL's <<eql-sequences,sequence syntax>> to search for a series of
|
|
ordered events. List the event items in ascending chronological order,
|
|
with the most recent event listed last:
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-data-stream/_eql/search
|
|
{
|
|
"query": """
|
|
sequence
|
|
[ process where process.name == "regsvr32.exe" ]
|
|
[ file where stringContains(file.name, "scrobj.dll") ]
|
|
"""
|
|
}
|
|
----
|
|
// TEST[setup:sec_logs]
|
|
|
|
The response's `hits.sequences` property contains the 10 most recent matching
|
|
sequences.
|
|
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"is_partial": false,
|
|
"is_running": false,
|
|
"took": 60,
|
|
"timed_out": false,
|
|
"hits": {
|
|
"total": {
|
|
"value": 1,
|
|
"relation": "eq"
|
|
},
|
|
"sequences": [
|
|
{
|
|
"events": [
|
|
{
|
|
"_index": ".ds-my-data-stream-2099.12.07-000001",
|
|
"_id": "OQmfCaduce8zoHT93o4H",
|
|
"_source": {
|
|
"@timestamp": "2099-12-07T11:07:09.000Z",
|
|
"event": {
|
|
"category": "process",
|
|
"id": "aR3NWVOs",
|
|
"sequence": 4
|
|
},
|
|
"process": {
|
|
"pid": 2012,
|
|
"name": "regsvr32.exe",
|
|
"command_line": "regsvr32.exe /s /u /i:https://...RegSvr32.sct scrobj.dll",
|
|
"executable": "C:\\Windows\\System32\\regsvr32.exe"
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"_index": ".ds-my-data-stream-2099.12.07-000001",
|
|
"_id": "yDwnGIJouOYGBzP0ZE9n",
|
|
"_source": {
|
|
"@timestamp": "2099-12-07T11:07:10.000Z",
|
|
"event": {
|
|
"category": "file",
|
|
"id": "tZ1NWVOs",
|
|
"sequence": 5
|
|
},
|
|
"process": {
|
|
"pid": 2012,
|
|
"name": "regsvr32.exe",
|
|
"executable": "C:\\Windows\\System32\\regsvr32.exe"
|
|
},
|
|
"file": {
|
|
"path": "C:\\Windows\\System32\\scrobj.dll",
|
|
"name": "scrobj.dll"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
]
|
|
}
|
|
}
|
|
----
|
|
// TESTRESPONSE[s/"took": 60/"took": $body.took/]
|
|
// TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.sequences.0.events.0._index/]
|
|
// TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.sequences.0.events.0._id/]
|
|
// TESTRESPONSE[s/"_id": "yDwnGIJouOYGBzP0ZE9n"/"_id": $body.hits.sequences.0.events.1._id/]
|
|
|
|
Use the <<eql-with-maxspan-keywords,`with maxspan` keywords>> to constrain
|
|
matching sequences to a timespan:
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-data-stream/_eql/search
|
|
{
|
|
"query": """
|
|
sequence with maxspan=1h
|
|
[ process where process.name == "regsvr32.exe" ]
|
|
[ file where stringContains(file.name, "scrobj.dll") ]
|
|
"""
|
|
}
|
|
----
|
|
// TEST[setup:sec_logs]
|
|
|
|
Use the <<eql-by-keyword,`by` keyword>> to match events that share the
|
|
same field values:
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-data-stream/_eql/search
|
|
{
|
|
"query": """
|
|
sequence with maxspan=1h
|
|
[ process where process.name == "regsvr32.exe" ] by process.pid
|
|
[ file where stringContains(file.name, "scrobj.dll") ] by process.pid
|
|
"""
|
|
}
|
|
----
|
|
// TEST[setup:sec_logs]
|
|
|
|
If a field value should be shared across all events, use the `sequence by`
|
|
keyword. The following query is equivalent to the previous one.
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-data-stream/_eql/search
|
|
{
|
|
"query": """
|
|
sequence by process.pid with maxspan=1h
|
|
[ process where process.name == "regsvr32.exe" ]
|
|
[ file where stringContains(file.name, "scrobj.dll") ]
|
|
"""
|
|
}
|
|
----
|
|
// TEST[setup:sec_logs]
|
|
|
|
The `hits.sequences.join_keys` property contains the shared field values.
|
|
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"is_partial": false,
|
|
"is_running": false,
|
|
"took": 60,
|
|
"timed_out": false,
|
|
"hits": {
|
|
"total": {
|
|
"value": 1,
|
|
"relation": "eq"
|
|
},
|
|
"sequences": [
|
|
{
|
|
"join_keys": [
|
|
2012
|
|
],
|
|
"events": [
|
|
{
|
|
"_index": ".ds-my-data-stream-2099.12.07-000001",
|
|
"_id": "OQmfCaduce8zoHT93o4H",
|
|
"_source": {
|
|
"@timestamp": "2099-12-07T11:07:09.000Z",
|
|
"event": {
|
|
"category": "process",
|
|
"id": "aR3NWVOs",
|
|
"sequence": 4
|
|
},
|
|
"process": {
|
|
"pid": 2012,
|
|
"name": "regsvr32.exe",
|
|
"command_line": "regsvr32.exe /s /u /i:https://...RegSvr32.sct scrobj.dll",
|
|
"executable": "C:\\Windows\\System32\\regsvr32.exe"
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"_index": ".ds-my-data-stream-2099.12.07-000001",
|
|
"_id": "yDwnGIJouOYGBzP0ZE9n",
|
|
"_source": {
|
|
"@timestamp": "2099-12-07T11:07:10.000Z",
|
|
"event": {
|
|
"category": "file",
|
|
"id": "tZ1NWVOs",
|
|
"sequence": 5
|
|
},
|
|
"process": {
|
|
"pid": 2012,
|
|
"name": "regsvr32.exe",
|
|
"executable": "C:\\Windows\\System32\\regsvr32.exe"
|
|
},
|
|
"file": {
|
|
"path": "C:\\Windows\\System32\\scrobj.dll",
|
|
"name": "scrobj.dll"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
]
|
|
}
|
|
}
|
|
----
|
|
// TESTRESPONSE[s/"took": 60/"took": $body.took/]
|
|
// TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.sequences.0.events.0._index/]
|
|
// TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.sequences.0.events.0._id/]
|
|
// TESTRESPONSE[s/"_id": "yDwnGIJouOYGBzP0ZE9n"/"_id": $body.hits.sequences.0.events.1._id/]
|
|
|
|
Use the <<eql-until-keyword,`until` keyword>> to specify an expiration
|
|
event for sequences. Matching sequences must end before this event.
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-data-stream/_eql/search
|
|
{
|
|
"query": """
|
|
sequence by process.pid with maxspan=1h
|
|
[ process where process.name == "regsvr32.exe" ]
|
|
[ file where stringContains(file.name, "scrobj.dll") ]
|
|
until [ process where event.type == "termination" ]
|
|
"""
|
|
}
|
|
----
|
|
// TEST[setup:sec_logs]
|
|
|
|
[discrete]
|
|
[[retrieve-selected-fields]]
|
|
=== Retrieve selected fields
|
|
|
|
By default, each hit in the search response includes the document `_source`,
|
|
which is the entire JSON object that was provided when indexing the document.
|
|
|
|
You can use the <<common-options-response-filtering,`filter_path`>> query
|
|
parameter to filter the API response. For example, the following search returns
|
|
only the timestamp and PID from the `_source` of each matching event.
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-data-stream/_eql/search?filter_path=hits.events._source.@timestamp,hits.events._source.process.pid
|
|
{
|
|
"query": """
|
|
process where process.name == "regsvr32.exe"
|
|
"""
|
|
}
|
|
----
|
|
// TEST[setup:sec_logs]
|
|
|
|
The API returns the following response.
|
|
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"hits": {
|
|
"events": [
|
|
{
|
|
"_source": {
|
|
"@timestamp": "2099-12-07T11:07:09.000Z",
|
|
"process": {
|
|
"pid": 2012
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"_source": {
|
|
"@timestamp": "2099-12-07T11:07:10.000Z",
|
|
"process": {
|
|
"pid": 2012
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
----
|
|
|
|
You can also use the `fields` parameter to retrieve and format specific fields
|
|
in the response. This field is identical to the search API's
|
|
<<search-fields,`fields` parameter>>.
|
|
|
|
|
|
include::{es-repo-dir}/search/search-your-data/retrieve-selected-fields.asciidoc[tag=fields-param-desc]
|
|
|
|
The following EQL search uses the `fields` parameter to retrieve values for the
|
|
`event.type` field, all fields starting with `process.`, and the `@timestamp`
|
|
field. The request also uses the `filter_path` query parameter to exclude the
|
|
`_source` of each hit.
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-data-stream/_eql/search?filter_path=-hits.events._source
|
|
{
|
|
"query": """
|
|
process where process.name == "regsvr32.exe"
|
|
""",
|
|
"fields": [
|
|
"event.type",
|
|
"process.*", <1>
|
|
{
|
|
"field": "@timestamp",
|
|
"format": "epoch_millis" <2>
|
|
}
|
|
]
|
|
}
|
|
----
|
|
// TEST[setup:sec_logs]
|
|
|
|
<1> Both full field names and wildcard patterns are accepted.
|
|
<2> Use the `format` parameter to apply a custom format for the field's values.
|
|
<<date,`date`>> and <<date_nanos, `date_nanos`>> fields accept a
|
|
<<mapping-date-format,date format>>. <<spatial_datatypes, Spatial fields>>
|
|
accept either `geojson` for http://www.geojson.org[GeoJSON] (the default)
|
|
or `wkt` for
|
|
{wikipedia}/Well-known_text_representation_of_geometry[Well Known Text].
|
|
Other field types do not support the `format` parameter.
|
|
|
|
The values are returned as a flat list in the `fields` section of each hit:
|
|
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"is_partial": false,
|
|
"is_running": false,
|
|
"took": 60,
|
|
"timed_out": false,
|
|
"hits": {
|
|
"total": {
|
|
"value": 2,
|
|
"relation": "eq"
|
|
},
|
|
"events": [
|
|
{
|
|
"_index": ".ds-my-data-stream-2099.12.07-000001",
|
|
"_id": "OQmfCaduce8zoHT93o4H",
|
|
"fields": {
|
|
"process.name": [
|
|
"regsvr32.exe"
|
|
],
|
|
"process.name.keyword": [
|
|
"regsvr32.exe"
|
|
],
|
|
"@timestamp": [
|
|
"4100324829000"
|
|
],
|
|
"process.command_line": [
|
|
"regsvr32.exe /s /u /i:https://...RegSvr32.sct scrobj.dll"
|
|
],
|
|
"process.command_line.keyword": [
|
|
"regsvr32.exe /s /u /i:https://...RegSvr32.sct scrobj.dll"
|
|
],
|
|
"process.executable.keyword": [
|
|
"C:\\Windows\\System32\\regsvr32.exe"
|
|
],
|
|
"process.pid": [
|
|
2012
|
|
],
|
|
"process.executable": [
|
|
"C:\\Windows\\System32\\regsvr32.exe"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"_index": ".ds-my-data-stream-2099.12.07-000001",
|
|
"_id": "xLkCaj4EujzdNSxfYLbO",
|
|
"fields": {
|
|
"process.name": [
|
|
"regsvr32.exe"
|
|
],
|
|
"process.name.keyword": [
|
|
"regsvr32.exe"
|
|
],
|
|
"@timestamp": [
|
|
"4100324830000"
|
|
],
|
|
"event.type": [
|
|
"termination"
|
|
],
|
|
"process.executable.keyword": [
|
|
"C:\\Windows\\System32\\regsvr32.exe"
|
|
],
|
|
"process.pid": [
|
|
2012
|
|
],
|
|
"process.executable": [
|
|
"C:\\Windows\\System32\\regsvr32.exe"
|
|
]
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
----
|
|
// TESTRESPONSE[s/"took": 60/"took": $body.took/]
|
|
// TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.events.0._index/]
|
|
// TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.events.0._id/]
|
|
// TESTRESPONSE[s/"_id": "xLkCaj4EujzdNSxfYLbO"/"_id": $body.hits.events.1._id/]
|
|
|
|
[discrete]
|
|
[[eql-use-runtime-fields]]
|
|
=== Use runtime fields
|
|
|
|
Use the `runtime_mappings` parameter to extract and create <<runtime,runtime
|
|
fields>> during a search. Use the `fields` parameter to include runtime fields
|
|
in the response.
|
|
|
|
The following search creates a `day_of_week` runtime field from the `@timestamp`
|
|
and returns it in the response.
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-data-stream/_eql/search?filter_path=-hits.events._source
|
|
{
|
|
"runtime_mappings": {
|
|
"day_of_week": {
|
|
"type": "keyword",
|
|
"script": "emit(doc['@timestamp'].value.dayOfWeekEnum.toString())"
|
|
}
|
|
},
|
|
"query": """
|
|
process where process.name == "regsvr32.exe"
|
|
""",
|
|
"fields": [
|
|
"@timestamp",
|
|
"day_of_week"
|
|
]
|
|
}
|
|
----
|
|
// TEST[setup:sec_logs]
|
|
|
|
The API returns:
|
|
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"is_partial": false,
|
|
"is_running": false,
|
|
"took": 60,
|
|
"timed_out": false,
|
|
"hits": {
|
|
"total": {
|
|
"value": 2,
|
|
"relation": "eq"
|
|
},
|
|
"events": [
|
|
{
|
|
"_index": ".ds-my-data-stream-2099.12.07-000001",
|
|
"_id": "OQmfCaduce8zoHT93o4H",
|
|
"fields": {
|
|
"@timestamp": [
|
|
"2099-12-07T11:07:09.000Z"
|
|
],
|
|
"day_of_week": [
|
|
"MONDAY"
|
|
]
|
|
}
|
|
},
|
|
...
|
|
]
|
|
}
|
|
}
|
|
----
|
|
// TESTRESPONSE[s/"took": 60/"took": $body.took/]
|
|
// TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.events.0._index/]
|
|
// TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.events.0._id/]
|
|
// TESTRESPONSE[s/\.\.\./$body.hits.events.1/]
|
|
|
|
[discrete]
|
|
[[specify-a-timestamp-or-event-category-field]]
|
|
=== Specify a timestamp or event category field
|
|
|
|
The EQL search API uses the `@timestamp` and `event.category` fields from the
|
|
{ecs-ref}[ECS] by default. To specify different fields, use the
|
|
`timestamp_field` and `event_category_field` parameters:
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-data-stream/_eql/search
|
|
{
|
|
"timestamp_field": "file.accessed",
|
|
"event_category_field": "file.type",
|
|
"query": """
|
|
file where (file.size > 1 and file.type == "file")
|
|
"""
|
|
}
|
|
----
|
|
// TEST[setup:sec_logs]
|
|
|
|
The event category field must be mapped as a <<keyword,`keyword`>> family field
|
|
type. The timestamp field should be mapped as a <<date,`date`>> field type.
|
|
<<date_nanos,`date_nanos`>> timestamp fields are not supported. You cannot use a
|
|
<<nested,`nested`>> field or the sub-fields of a `nested` field as the timestamp
|
|
or event category field.
|
|
|
|
[discrete]
|
|
[[eql-search-specify-a-sort-tiebreaker]]
|
|
=== Specify a sort tiebreaker
|
|
|
|
By default, the EQL search API returns matching hits by timestamp. If two or
|
|
more events share the same timestamp, {es} uses a tiebreaker field value to sort
|
|
the events in ascending order. {es} orders events with no
|
|
tiebreaker value after events with a value.
|
|
|
|
If you don't specify a tiebreaker field or the events also share the same
|
|
tiebreaker value, {es} considers the events concurrent. Concurrent events cannot
|
|
be part of the same sequence and may not be returned in a consistent sort order.
|
|
|
|
To specify a tiebreaker field, use the `tiebreaker_field` parameter. If you use
|
|
the {ecs-ref}[ECS], we recommend using `event.sequence` as the tiebreaker field.
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-data-stream/_eql/search
|
|
{
|
|
"tiebreaker_field": "event.sequence",
|
|
"query": """
|
|
process where process.name == "cmd.exe" and stringContains(process.executable, "System32")
|
|
"""
|
|
}
|
|
----
|
|
// TEST[setup:sec_logs]
|
|
|
|
[discrete]
|
|
[[eql-search-filter-query-dsl]]
|
|
=== Filter using Query DSL
|
|
|
|
The `filter` parameter uses <<query-dsl,Query DSL>> to limit the documents on
|
|
which an EQL query runs.
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-data-stream/_eql/search
|
|
{
|
|
"filter": {
|
|
"range": {
|
|
"@timestamp": {
|
|
"gte": "now-1d/d",
|
|
"lt": "now/d"
|
|
}
|
|
}
|
|
},
|
|
"query": """
|
|
file where (file.type == "file" and file.name == "cmd.exe")
|
|
"""
|
|
}
|
|
----
|
|
// TEST[setup:sec_logs]
|
|
|
|
[discrete]
|
|
[[eql-search-async]]
|
|
=== Run an async EQL search
|
|
|
|
By default, EQL search requests are synchronous and wait for complete results
|
|
before returning a response. However, complete results can take longer for
|
|
searches across large data sets, <<data-tiers,cold>> or <<data-tiers,frozen>>
|
|
data, or <<modules-cross-cluster-search,multiple clusters>>.
|
|
|
|
To avoid long waits, run an async EQL search. Set `wait_for_completion_timeout`
|
|
to a duration you'd like to wait for synchronous results.
|
|
+
|
|
[source,console]
|
|
----
|
|
GET /my-data-stream/_eql/search
|
|
{
|
|
"wait_for_completion_timeout": "2s",
|
|
"query": """
|
|
process where process.name == "cmd.exe"
|
|
"""
|
|
}
|
|
----
|
|
// TEST[setup:sec_logs]
|
|
|
|
If the request doesn't finish within the timeout period, the search becomes async
|
|
and returns a response that includes:
|
|
|
|
* A search ID
|
|
* An `is_partial` value of `true`, indicating the search results are
|
|
incomplete
|
|
* An `is_running` value of `true`, indicating the search is ongoing
|
|
|
|
The async search continues to run in the background without blocking other
|
|
requests.
|
|
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
|
|
"is_partial": true,
|
|
"is_running": true,
|
|
"took": 2000,
|
|
"timed_out": false,
|
|
"hits": ...
|
|
}
|
|
----
|
|
// TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/]
|
|
// TESTRESPONSE[s/"is_partial": true/"is_partial": $body.is_partial/]
|
|
// TESTRESPONSE[s/"is_running": true/"is_running": $body.is_running/]
|
|
// TESTRESPONSE[s/"took": 2000/"took": $body.took/]
|
|
// TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]
|
|
|
|
To check the progress of an async search, use the <<get-async-eql-search-api,get
|
|
async EQL search API>> with the search ID. Specify how long you'd like for
|
|
complete results in the `wait_for_completion_timeout` parameter.
|
|
[source,console]
|
|
----
|
|
GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?wait_for_completion_timeout=2s
|
|
----
|
|
// TEST[skip: no access to search ID]
|
|
|
|
If the response's `is_running` value is `false`, the async search has finished.
|
|
If the `is_partial` value is `false`, the returned search results are
|
|
complete.
|
|
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
|
|
"is_partial": false,
|
|
"is_running": false,
|
|
"took": 2000,
|
|
"timed_out": false,
|
|
"hits": ...
|
|
}
|
|
----
|
|
// TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/]
|
|
// TESTRESPONSE[s/"took": 2000/"took": $body.took/]
|
|
// TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]
|
|
|
|
Another more lightweight way to check the progress of an async search is to use
|
|
the <<get-async-eql-status-api,get async EQL status API>> with the search ID.
|
|
|
|
[source,console]
|
|
----
|
|
GET /_eql/search/status/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=
|
|
----
|
|
// TEST[skip: no access to search ID]
|
|
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
|
|
"is_running": false,
|
|
"is_partial": false,
|
|
"expiration_time_in_millis": 1611690295000,
|
|
"completion_status": 200
|
|
}
|
|
----
|
|
// TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/]
|
|
// TESTRESPONSE[s/"expiration_time_in_millis": 1611690295000/"expiration_time_in_millis": $body.expiration_time_in_millis/]
|
|
|
|
[discrete]
|
|
[[eql-search-store-async-eql-search]]
|
|
=== Change the search retention period
|
|
|
|
By default, the EQL search API stores async searches for five days. After this
|
|
period, any searches and their results are deleted. Use the `keep_alive`
|
|
parameter to change this retention period:
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-data-stream/_eql/search
|
|
{
|
|
"keep_alive": "2d",
|
|
"wait_for_completion_timeout": "2s",
|
|
"query": """
|
|
process where process.name == "cmd.exe"
|
|
"""
|
|
}
|
|
----
|
|
// TEST[setup:sec_logs]
|
|
|
|
You can use the <<get-async-eql-search-api,get async EQL search API>>'s
|
|
`keep_alive` parameter to later change the retention period. The new retention
|
|
period starts after the get request runs.
|
|
|
|
[source,console]
|
|
----
|
|
GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d
|
|
----
|
|
// TEST[skip: no access to search ID]
|
|
|
|
Use the <<delete-async-eql-search-api,delete async EQL search API>> to
|
|
manually delete an async EQL search before the `keep_alive` period ends. If the
|
|
search is still ongoing, {es} cancels the search request.
|
|
|
|
[source,console]
|
|
----
|
|
DELETE /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d
|
|
----
|
|
// TEST[skip: no access to search ID]
|
|
|
|
[discrete]
|
|
[[eql-search-store-sync-eql-search]]
|
|
=== Store synchronous EQL searches
|
|
|
|
By default, the EQL search API only stores async searches. To save a synchronous
|
|
search, set `keep_on_completion` to `true`:
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-data-stream/_eql/search
|
|
{
|
|
"keep_on_completion": true,
|
|
"wait_for_completion_timeout": "2s",
|
|
"query": """
|
|
process where process.name == "cmd.exe"
|
|
"""
|
|
}
|
|
----
|
|
// TEST[setup:sec_logs]
|
|
|
|
The response includes a search ID. `is_partial` and `is_running` are `false`,
|
|
indicating the EQL search was synchronous and returned complete results.
|
|
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"id": "FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=",
|
|
"is_partial": false,
|
|
"is_running": false,
|
|
"took": 52,
|
|
"timed_out": false,
|
|
"hits": ...
|
|
}
|
|
----
|
|
// TESTRESPONSE[s/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=/$body.id/]
|
|
// TESTRESPONSE[s/"took": 52/"took": $body.took/]
|
|
// TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]
|
|
|
|
Use the <<get-async-eql-search-api,get async EQL search API>> to get the
|
|
same results later:
|
|
|
|
[source,console]
|
|
----
|
|
GET /_eql/search/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=
|
|
----
|
|
// TEST[skip: no access to search ID]
|
|
|
|
Saved synchronous searches are still subject to the `keep_alive` parameter's
|
|
retention period. When this period ends, the search and its results are deleted.
|
|
|
|
You can also check only the status of the saved synchronous search without
|
|
results by using <<get-async-eql-status-api,get async EQL status API>>.
|
|
|
|
You can also manually delete saved synchronous searches using the
|
|
<<delete-async-eql-search-api,delete async EQL search API>>.
|
|
|
|
include::syntax.asciidoc[]
|
|
include::functions.asciidoc[]
|
|
include::pipes.asciidoc[]
|
|
include::detect-threats-with-eql.asciidoc[]
|