mirror of
https://github.com/elastic/kibana.git
synced 2025-04-22 00:45:43 -04:00
# Backport This will backport the following commits from `main` to `8.2`: - [[docs] [kql] Update KQL docs to include info about errors with multiple field types (#158035)](https://github.com/elastic/kibana/pull/158035) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Lukas Olson","email":"lukas@elastic.co"},"sourceCommit":{"committedDate":"2023-05-23T16:29:47Z","message":"[docs] [kql] Update KQL docs to include info about errors with multiple field types (#158035)\n\n## Summary\r\n\r\nWe received feedback that the given example would result in errors,\r\nsince it targets both\r\n[string](https://www.elastic.co/guide/en/ecs/current/ecs-http.html#field-http-response-body-content)\r\nand\r\n[numeric](https://www.elastic.co/guide/en/ecs/current/ecs-http.html#field-http-response-bytes)\r\nfields, so we've updated the example with something that won't result in\r\nan error, and updated the docs with a note that explains when/why these\r\nerrors may occur.\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [ ] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] Any UI touched in this PR is usable by keyboard only (learn more\r\nabout [keyboard accessibility](https://webaim.org/techniques/keyboard/))\r\n- [ ] Any UI touched in this PR does not create any new axe failures\r\n(run axe in browser:\r\n[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),\r\n[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))\r\n- [ ] If a plugin configuration key changed, check if it needs to be\r\nallowlisted in the cloud and added to the [docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n- [ ] This renders correctly on smaller devices using a responsive\r\nlayout. (You can test this [in your\r\nbrowser](https://www.browserstack.com/guide/responsive-testing-on-local-server))\r\n- [ ] This was checked for [cross-browser\r\ncompatibility](https://www.elastic.co/support/matrix#matrix_browsers)\r\n\r\n\r\n### Risk Matrix\r\n\r\nDelete this section if it is not applicable to this PR.\r\n\r\nBefore closing this PR, invite QA, stakeholders, and other developers to\r\nidentify risks that should be tested prior to the change/feature\r\nrelease.\r\n\r\nWhen forming the risk matrix, consider some of the following examples\r\nand how they may potentially impact the change:\r\n\r\n| Risk | Probability | Severity | Mitigation/Notes |\r\n\r\n|---------------------------|-------------|----------|-------------------------|\r\n| Multiple Spaces—unexpected behavior in non-default Kibana Space.\r\n| Low | High | Integration tests will verify that all features are still\r\nsupported in non-default Kibana Space and when user switches between\r\nspaces. |\r\n| Multiple nodes—Elasticsearch polling might have race conditions\r\nwhen multiple Kibana nodes are polling for the same tasks. | High | Low\r\n| Tasks are idempotent, so executing them multiple times will not result\r\nin logical error, but will degrade performance. To test for this case we\r\nadd plenty of unit tests around this logic and document manual testing\r\nprocedure. |\r\n| Code should gracefully handle cases when feature X or plugin Y are\r\ndisabled. | Medium | High | Unit tests will verify that any feature flag\r\nor plugin combination still results in our service operational. |\r\n| [See more potential risk\r\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>","sha":"be35641e171a7b65b2d719f3e1276c183b0a8024","branchLabelMapping":{"^v8.9.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Docs","release_note:skip","v8.0.2","v8.1.4","v8.2.4","v8.3.4","v8.4.4","v8.5.4","v8.6.3","v8.7.2","v8.9.0","v7.17.11","v8.8.1"],"number":158035,"url":"https://github.com/elastic/kibana/pull/158035","mergeCommit":{"message":"[docs] [kql] Update KQL docs to include info about errors with multiple field types (#158035)\n\n## Summary\r\n\r\nWe received feedback that the given example would result in errors,\r\nsince it targets both\r\n[string](https://www.elastic.co/guide/en/ecs/current/ecs-http.html#field-http-response-body-content)\r\nand\r\n[numeric](https://www.elastic.co/guide/en/ecs/current/ecs-http.html#field-http-response-bytes)\r\nfields, so we've updated the example with something that won't result in\r\nan error, and updated the docs with a note that explains when/why these\r\nerrors may occur.\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [ ] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] Any UI touched in this PR is usable by keyboard only (learn more\r\nabout [keyboard accessibility](https://webaim.org/techniques/keyboard/))\r\n- [ ] Any UI touched in this PR does not create any new axe failures\r\n(run axe in browser:\r\n[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),\r\n[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))\r\n- [ ] If a plugin configuration key changed, check if it needs to be\r\nallowlisted in the cloud and added to the [docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n- [ ] This renders correctly on smaller devices using a responsive\r\nlayout. (You can test this [in your\r\nbrowser](https://www.browserstack.com/guide/responsive-testing-on-local-server))\r\n- [ ] This was checked for [cross-browser\r\ncompatibility](https://www.elastic.co/support/matrix#matrix_browsers)\r\n\r\n\r\n### Risk Matrix\r\n\r\nDelete this section if it is not applicable to this PR.\r\n\r\nBefore closing this PR, invite QA, stakeholders, and other developers to\r\nidentify risks that should be tested prior to the change/feature\r\nrelease.\r\n\r\nWhen forming the risk matrix, consider some of the following examples\r\nand how they may potentially impact the change:\r\n\r\n| Risk | Probability | Severity | Mitigation/Notes |\r\n\r\n|---------------------------|-------------|----------|-------------------------|\r\n| Multiple Spaces—unexpected behavior in non-default Kibana Space.\r\n| Low | High | Integration tests will verify that all features are still\r\nsupported in non-default Kibana Space and when user switches between\r\nspaces. |\r\n| Multiple nodes—Elasticsearch polling might have race conditions\r\nwhen multiple Kibana nodes are polling for the same tasks. | High | Low\r\n| Tasks are idempotent, so executing them multiple times will not result\r\nin logical error, but will degrade performance. To test for this case we\r\nadd plenty of unit tests around this logic and document manual testing\r\nprocedure. |\r\n| Code should gracefully handle cases when feature X or plugin Y are\r\ndisabled. | Medium | High | Unit tests will verify that any feature flag\r\nor plugin combination still results in our service operational. |\r\n| [See more potential risk\r\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>","sha":"be35641e171a7b65b2d719f3e1276c183b0a8024"}},"sourceBranch":"main","suggestedTargetBranches":["8.0","8.1","8.2","8.3","8.4","8.5","8.6","8.7","7.17","8.8"],"targetPullRequestStates":[{"branch":"8.0","label":"v8.0.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.1","label":"v8.1.4","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.2","label":"v8.2.4","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.3","label":"v8.3.4","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.4","label":"v8.4.4","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.5","label":"v8.5.4","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.6","label":"v8.6.3","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.7","label":"v8.7.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.9.0","labelRegex":"^v8.9.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/158035","number":158035,"mergeCommit":{"message":"[docs] [kql] Update KQL docs to include info about errors with multiple field types (#158035)\n\n## Summary\r\n\r\nWe received feedback that the given example would result in errors,\r\nsince it targets both\r\n[string](https://www.elastic.co/guide/en/ecs/current/ecs-http.html#field-http-response-body-content)\r\nand\r\n[numeric](https://www.elastic.co/guide/en/ecs/current/ecs-http.html#field-http-response-bytes)\r\nfields, so we've updated the example with something that won't result in\r\nan error, and updated the docs with a note that explains when/why these\r\nerrors may occur.\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [ ] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] Any UI touched in this PR is usable by keyboard only (learn more\r\nabout [keyboard accessibility](https://webaim.org/techniques/keyboard/))\r\n- [ ] Any UI touched in this PR does not create any new axe failures\r\n(run axe in browser:\r\n[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),\r\n[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))\r\n- [ ] If a plugin configuration key changed, check if it needs to be\r\nallowlisted in the cloud and added to the [docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n- [ ] This renders correctly on smaller devices using a responsive\r\nlayout. (You can test this [in your\r\nbrowser](https://www.browserstack.com/guide/responsive-testing-on-local-server))\r\n- [ ] This was checked for [cross-browser\r\ncompatibility](https://www.elastic.co/support/matrix#matrix_browsers)\r\n\r\n\r\n### Risk Matrix\r\n\r\nDelete this section if it is not applicable to this PR.\r\n\r\nBefore closing this PR, invite QA, stakeholders, and other developers to\r\nidentify risks that should be tested prior to the change/feature\r\nrelease.\r\n\r\nWhen forming the risk matrix, consider some of the following examples\r\nand how they may potentially impact the change:\r\n\r\n| Risk | Probability | Severity | Mitigation/Notes |\r\n\r\n|---------------------------|-------------|----------|-------------------------|\r\n| Multiple Spaces—unexpected behavior in non-default Kibana Space.\r\n| Low | High | Integration tests will verify that all features are still\r\nsupported in non-default Kibana Space and when user switches between\r\nspaces. |\r\n| Multiple nodes—Elasticsearch polling might have race conditions\r\nwhen multiple Kibana nodes are polling for the same tasks. | High | Low\r\n| Tasks are idempotent, so executing them multiple times will not result\r\nin logical error, but will degrade performance. To test for this case we\r\nadd plenty of unit tests around this logic and document manual testing\r\nprocedure. |\r\n| Code should gracefully handle cases when feature X or plugin Y are\r\ndisabled. | Medium | High | Unit tests will verify that any feature flag\r\nor plugin combination still results in our service operational. |\r\n| [See more potential risk\r\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>","sha":"be35641e171a7b65b2d719f3e1276c183b0a8024"}},{"branch":"7.17","label":"v7.17.11","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.8","label":"v8.8.1","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Lukas Olson <lukas@elastic.co>
252 lines
7.8 KiB
Text
252 lines
7.8 KiB
Text
[[kuery-query]]
|
||
=== {kib} Query Language
|
||
|
||
The {kib} Query Language (KQL) is a simple text-based query language for filtering data.
|
||
|
||
* KQL only filters data, and has no role in aggregating, transforming, or sorting data.
|
||
* KQL is not to be confused with the <<lucene-query,Lucene query language>>, which has a different feature set.
|
||
|
||
Use KQL to filter documents where a value for a field exists, matches a given value, or is within a given range.
|
||
|
||
[discrete]
|
||
=== Filter for documents where a field exists
|
||
|
||
To filter documents for which an indexed value exists for a given field, use the `*` operator.
|
||
For example, to filter for documents where the `http.request.method` field exists, use the following syntax:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
http.request.method: *
|
||
-------------------
|
||
|
||
This checks for any indexed value, including an empty string.
|
||
|
||
[discrete]
|
||
=== Filter for documents that match a value
|
||
|
||
Use KQL to filter for documents that match a specific number, text, date, or boolean value.
|
||
For example, to filter for documents where the `http.request.method` is GET, use the following query:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
http.request.method: GET
|
||
-------------------
|
||
|
||
The field parameter is optional. If not provided, all fields are searched for the given value.
|
||
For example, to search all fields for “Hello”, use the following:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
Hello
|
||
-------------------
|
||
|
||
When querying keyword, numeric, date, or boolean fields, the value must be an exact match,
|
||
including punctuation and case. However, when querying text fields, {es} analyzes the
|
||
value provided according to the {ref}/analysis.html[field’s mapping settings].
|
||
For example, to search for documents where `http.request.body.content` (a `text` field)
|
||
contains the text “null pointer”:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
http.request.body.content: null pointer
|
||
-------------------
|
||
|
||
Because this is a `text` field, the order of these search terms does not matter, and
|
||
even documents containing “pointer null” are returned. To search `text` fields where the
|
||
terms are in the order provided, surround the value in quotation marks, as follows:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
http.request.body.content: "null pointer"
|
||
-------------------
|
||
|
||
Certain characters must be escaped by a backslash (unless surrounded by quotes).
|
||
For example, to search for documents where `http.request.referrer` is https://example.com,
|
||
use either of the following queries:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
http.request.referrer: "https://example.com"
|
||
http.request.referrer: https\://example.com
|
||
-------------------
|
||
|
||
You must escape following characters:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
\():<>"*
|
||
-------------------
|
||
|
||
[discrete]
|
||
=== Filter for documents within a range
|
||
|
||
To search documents that contain terms within a provided range, use KQL’s range syntax.
|
||
For example, to search for all documents for which `http.response.bytes` is less than 10000,
|
||
use the following syntax:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
http.response.bytes < 10000
|
||
-------------------
|
||
|
||
To search for an inclusive range, combine multiple range queries.
|
||
For example, to search for documents where `http.response.bytes` is greater than 10000
|
||
but less than or equal to 20000, use the following syntax:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
http.response.bytes > 10000 and http.response.bytes <= 20000
|
||
-------------------
|
||
|
||
You can also use range syntax for string values, IP addresses, and timestamps.
|
||
For example, to search for documents earlier than two weeks ago, use the following syntax:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
@timestamp < now-2w
|
||
-------------------
|
||
|
||
For more examples on acceptable date formats, refer to {ref}/common-options.html#date-math[Date Math].
|
||
|
||
[discrete]
|
||
=== Filter for documents using wildcards
|
||
|
||
To search for documents matching a pattern, use the wildcard syntax.
|
||
For example, to find documents where `http.response.status_code` begins with a 4, use the following syntax:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
http.response.status_code: 4*
|
||
-------------------
|
||
|
||
By default, leading wildcards are not allowed for performance reasons.
|
||
You can modify this with the <<query-allowleadingwildcards,`query:allowLeadingWildcards`>> advanced setting.
|
||
|
||
NOTE: Only `*` is currently supported. This matches zero or more characters.
|
||
|
||
[discrete]
|
||
=== Negating a query
|
||
|
||
To negate or exclude a set of documents, use the `not` keyword (not case-sensitive).
|
||
For example, to filter documents where the `http.request.method` is *not* GET, use the following query:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
NOT http.request.method: GET
|
||
-------------------
|
||
|
||
[discrete]
|
||
=== Combining multiple queries
|
||
|
||
To combine multiple queries, use the `and`/`or` keywords (not case-sensitive).
|
||
For example, to find documents where the `http.request.method` is GET *or* the `http.response.status_code` is 400,
|
||
use the following query:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
http.request.method: GET OR http.response.status_code: 400
|
||
-------------------
|
||
|
||
Similarly, to find documents where the `http.request.method` is GET *and* the
|
||
`http.response.status_code` is 400, use this query:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
http.request.method: GET AND http.response.status_code: 400
|
||
-------------------
|
||
|
||
To specify precedence when combining multiple queries, use parentheses.
|
||
For example, to find documents where the `http.request.method` is GET *and*
|
||
the `http.response.status_code` is 200, *or* the `http.request.method` is POST *and*
|
||
`http.response.status_code` is 400, use the following:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
(http.request.method: GET AND http.response.status_code: 200) OR
|
||
(http.request.method: POST AND http.response.status_code: 400)
|
||
-------------------
|
||
|
||
You can also use parentheses for shorthand syntax when querying multiple values for the same field.
|
||
For example, to find documents where the `http.request.method` is GET, POST, *or* DELETE, use the following:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
http.request.method: (GET OR POST OR DELETE)
|
||
-------------------
|
||
|
||
[discrete]
|
||
=== Matching multiple fields
|
||
|
||
Wildcards can also be used to query multiple fields. For example, to search for
|
||
documents where any sub-field of `datastream` contains “logs”, use the following:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
datastream.*: logs
|
||
-------------------
|
||
|
||
NOTE: When using wildcards to query multiple fields, errors might occur if the fields are of
|
||
different types. For example, if `datastream.*` matches both numeric and string fields, the
|
||
above query will result in an error because numeric fields cannot be queried for string values.
|
||
|
||
[discrete]
|
||
=== Querying nested fields
|
||
|
||
Querying {ref}/nested.html[nested fields] requires a special syntax. Consider the
|
||
following document, where `user` is a nested field:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
{
|
||
"user" : [
|
||
{
|
||
"first" : "John",
|
||
"last" : "Smith"
|
||
},
|
||
{
|
||
"first" : "Alice",
|
||
"last" : "White"
|
||
}
|
||
]
|
||
}
|
||
-------------------
|
||
|
||
To find documents where a single value inside the `user` array contains a first name of
|
||
“Alice” and last name of “White”, use the following:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
user:{ first: "Alice" and last: "White" }
|
||
-------------------
|
||
|
||
Because nested fields can be inside other nested fields,
|
||
you must specify the full path of the nested field you want to query.
|
||
For example, consider the following document where `user` and `names` are both nested fields:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
{
|
||
"user": [
|
||
{
|
||
"names": [
|
||
{
|
||
"first": "John",
|
||
"last": "Smith"
|
||
},
|
||
{
|
||
"first": "Alice",
|
||
"last": "White"
|
||
}
|
||
]
|
||
}
|
||
]
|
||
}
|
||
-------------------
|
||
|
||
To find documents where a single value inside the `user.names` array contains a first name of “Alice” *and*
|
||
last name of “White”, use the following:
|
||
|
||
[source,yaml]
|
||
-------------------
|
||
user.names:{ first: "Alice" and last: "White" }
|
||
-------------------
|