kibana/docs/concepts/kuery.asciidoc
Kibana Machine 553c6f63e8
[8.2] [docs] [kql] Update KQL docs to include info about errors with multiple field types (#158035) (#158288)
# Backport

This will backport the following commits from `main` to `8.2`:
- [[docs] [kql] Update KQL docs to include info about errors with
multiple field types
(#158035)](https://github.com/elastic/kibana/pull/158035)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Lukas
Olson","email":"lukas@elastic.co"},"sourceCommit":{"committedDate":"2023-05-23T16:29:47Z","message":"[docs]
[kql] Update KQL docs to include info about errors with multiple field
types (#158035)\n\n## Summary\r\n\r\nWe received feedback that the given
example would result in errors,\r\nsince it targets
both\r\n[string](https://www.elastic.co/guide/en/ecs/current/ecs-http.html#field-http-response-body-content)\r\nand\r\n[numeric](https://www.elastic.co/guide/en/ecs/current/ecs-http.html#field-http-response-bytes)\r\nfields,
so we've updated the example with something that won't result in\r\nan
error, and updated the docs with a note that explains when/why
these\r\nerrors may occur.\r\n\r\n### Checklist\r\n\r\nDelete any items
that are not applicable to this PR.\r\n\r\n- [ ] Any text added follows
[EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [ ] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] Any UI
touched in this PR is usable by keyboard only (learn more\r\nabout
[keyboard accessibility](https://webaim.org/techniques/keyboard/))\r\n-
[ ] Any UI touched in this PR does not create any new axe
failures\r\n(run axe in
browser:\r\n[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),\r\n[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))\r\n-
[ ] If a plugin configuration key changed, check if it needs to
be\r\nallowlisted in the cloud and added to the
[docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n-
[ ] This renders correctly on smaller devices using a
responsive\r\nlayout. (You can test this [in
your\r\nbrowser](https://www.browserstack.com/guide/responsive-testing-on-local-server))\r\n-
[ ] This was checked for
[cross-browser\r\ncompatibility](https://www.elastic.co/support/matrix#matrix_browsers)\r\n\r\n\r\n###
Risk Matrix\r\n\r\nDelete this section if it is not applicable to this
PR.\r\n\r\nBefore closing this PR, invite QA, stakeholders, and other
developers to\r\nidentify risks that should be tested prior to the
change/feature\r\nrelease.\r\n\r\nWhen forming the risk matrix, consider
some of the following examples\r\nand how they may potentially impact
the change:\r\n\r\n| Risk | Probability | Severity | Mitigation/Notes
|\r\n\r\n|---------------------------|-------------|----------|-------------------------|\r\n|
Multiple Spaces&mdash;unexpected behavior in non-default Kibana
Space.\r\n| Low | High | Integration tests will verify that all features
are still\r\nsupported in non-default Kibana Space and when user
switches between\r\nspaces. |\r\n| Multiple nodes&mdash;Elasticsearch
polling might have race conditions\r\nwhen multiple Kibana nodes are
polling for the same tasks. | High | Low\r\n| Tasks are idempotent, so
executing them multiple times will not result\r\nin logical error, but
will degrade performance. To test for this case we\r\nadd plenty of unit
tests around this logic and document manual testing\r\nprocedure. |\r\n|
Code should gracefully handle cases when feature X or plugin Y
are\r\ndisabled. | Medium | High | Unit tests will verify that any
feature flag\r\nor plugin combination still results in our service
operational. |\r\n| [See more potential
risk\r\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
|\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for
breaking API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by:
gchaps
<33642766+gchaps@users.noreply.github.com>","sha":"be35641e171a7b65b2d719f3e1276c183b0a8024","branchLabelMapping":{"^v8.9.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Docs","release_note:skip","v8.0.2","v8.1.4","v8.2.4","v8.3.4","v8.4.4","v8.5.4","v8.6.3","v8.7.2","v8.9.0","v7.17.11","v8.8.1"],"number":158035,"url":"https://github.com/elastic/kibana/pull/158035","mergeCommit":{"message":"[docs]
[kql] Update KQL docs to include info about errors with multiple field
types (#158035)\n\n## Summary\r\n\r\nWe received feedback that the given
example would result in errors,\r\nsince it targets
both\r\n[string](https://www.elastic.co/guide/en/ecs/current/ecs-http.html#field-http-response-body-content)\r\nand\r\n[numeric](https://www.elastic.co/guide/en/ecs/current/ecs-http.html#field-http-response-bytes)\r\nfields,
so we've updated the example with something that won't result in\r\nan
error, and updated the docs with a note that explains when/why
these\r\nerrors may occur.\r\n\r\n### Checklist\r\n\r\nDelete any items
that are not applicable to this PR.\r\n\r\n- [ ] Any text added follows
[EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [ ] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] Any UI
touched in this PR is usable by keyboard only (learn more\r\nabout
[keyboard accessibility](https://webaim.org/techniques/keyboard/))\r\n-
[ ] Any UI touched in this PR does not create any new axe
failures\r\n(run axe in
browser:\r\n[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),\r\n[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))\r\n-
[ ] If a plugin configuration key changed, check if it needs to
be\r\nallowlisted in the cloud and added to the
[docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n-
[ ] This renders correctly on smaller devices using a
responsive\r\nlayout. (You can test this [in
your\r\nbrowser](https://www.browserstack.com/guide/responsive-testing-on-local-server))\r\n-
[ ] This was checked for
[cross-browser\r\ncompatibility](https://www.elastic.co/support/matrix#matrix_browsers)\r\n\r\n\r\n###
Risk Matrix\r\n\r\nDelete this section if it is not applicable to this
PR.\r\n\r\nBefore closing this PR, invite QA, stakeholders, and other
developers to\r\nidentify risks that should be tested prior to the
change/feature\r\nrelease.\r\n\r\nWhen forming the risk matrix, consider
some of the following examples\r\nand how they may potentially impact
the change:\r\n\r\n| Risk | Probability | Severity | Mitigation/Notes
|\r\n\r\n|---------------------------|-------------|----------|-------------------------|\r\n|
Multiple Spaces&mdash;unexpected behavior in non-default Kibana
Space.\r\n| Low | High | Integration tests will verify that all features
are still\r\nsupported in non-default Kibana Space and when user
switches between\r\nspaces. |\r\n| Multiple nodes&mdash;Elasticsearch
polling might have race conditions\r\nwhen multiple Kibana nodes are
polling for the same tasks. | High | Low\r\n| Tasks are idempotent, so
executing them multiple times will not result\r\nin logical error, but
will degrade performance. To test for this case we\r\nadd plenty of unit
tests around this logic and document manual testing\r\nprocedure. |\r\n|
Code should gracefully handle cases when feature X or plugin Y
are\r\ndisabled. | Medium | High | Unit tests will verify that any
feature flag\r\nor plugin combination still results in our service
operational. |\r\n| [See more potential
risk\r\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
|\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for
breaking API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by:
gchaps
<33642766+gchaps@users.noreply.github.com>","sha":"be35641e171a7b65b2d719f3e1276c183b0a8024"}},"sourceBranch":"main","suggestedTargetBranches":["8.0","8.1","8.2","8.3","8.4","8.5","8.6","8.7","7.17","8.8"],"targetPullRequestStates":[{"branch":"8.0","label":"v8.0.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.1","label":"v8.1.4","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.2","label":"v8.2.4","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.3","label":"v8.3.4","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.4","label":"v8.4.4","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.5","label":"v8.5.4","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.6","label":"v8.6.3","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.7","label":"v8.7.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.9.0","labelRegex":"^v8.9.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/158035","number":158035,"mergeCommit":{"message":"[docs]
[kql] Update KQL docs to include info about errors with multiple field
types (#158035)\n\n## Summary\r\n\r\nWe received feedback that the given
example would result in errors,\r\nsince it targets
both\r\n[string](https://www.elastic.co/guide/en/ecs/current/ecs-http.html#field-http-response-body-content)\r\nand\r\n[numeric](https://www.elastic.co/guide/en/ecs/current/ecs-http.html#field-http-response-bytes)\r\nfields,
so we've updated the example with something that won't result in\r\nan
error, and updated the docs with a note that explains when/why
these\r\nerrors may occur.\r\n\r\n### Checklist\r\n\r\nDelete any items
that are not applicable to this PR.\r\n\r\n- [ ] Any text added follows
[EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [ ] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] Any UI
touched in this PR is usable by keyboard only (learn more\r\nabout
[keyboard accessibility](https://webaim.org/techniques/keyboard/))\r\n-
[ ] Any UI touched in this PR does not create any new axe
failures\r\n(run axe in
browser:\r\n[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),\r\n[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))\r\n-
[ ] If a plugin configuration key changed, check if it needs to
be\r\nallowlisted in the cloud and added to the
[docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n-
[ ] This renders correctly on smaller devices using a
responsive\r\nlayout. (You can test this [in
your\r\nbrowser](https://www.browserstack.com/guide/responsive-testing-on-local-server))\r\n-
[ ] This was checked for
[cross-browser\r\ncompatibility](https://www.elastic.co/support/matrix#matrix_browsers)\r\n\r\n\r\n###
Risk Matrix\r\n\r\nDelete this section if it is not applicable to this
PR.\r\n\r\nBefore closing this PR, invite QA, stakeholders, and other
developers to\r\nidentify risks that should be tested prior to the
change/feature\r\nrelease.\r\n\r\nWhen forming the risk matrix, consider
some of the following examples\r\nand how they may potentially impact
the change:\r\n\r\n| Risk | Probability | Severity | Mitigation/Notes
|\r\n\r\n|---------------------------|-------------|----------|-------------------------|\r\n|
Multiple Spaces&mdash;unexpected behavior in non-default Kibana
Space.\r\n| Low | High | Integration tests will verify that all features
are still\r\nsupported in non-default Kibana Space and when user
switches between\r\nspaces. |\r\n| Multiple nodes&mdash;Elasticsearch
polling might have race conditions\r\nwhen multiple Kibana nodes are
polling for the same tasks. | High | Low\r\n| Tasks are idempotent, so
executing them multiple times will not result\r\nin logical error, but
will degrade performance. To test for this case we\r\nadd plenty of unit
tests around this logic and document manual testing\r\nprocedure. |\r\n|
Code should gracefully handle cases when feature X or plugin Y
are\r\ndisabled. | Medium | High | Unit tests will verify that any
feature flag\r\nor plugin combination still results in our service
operational. |\r\n| [See more potential
risk\r\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
|\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for
breaking API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by:
gchaps
<33642766+gchaps@users.noreply.github.com>","sha":"be35641e171a7b65b2d719f3e1276c183b0a8024"}},{"branch":"7.17","label":"v7.17.11","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.8","label":"v8.8.1","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Lukas Olson <lukas@elastic.co>
2023-05-23 12:54:17 -04:00

252 lines
7.8 KiB
Text
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

[[kuery-query]]
=== {kib} Query Language
The {kib} Query Language (KQL) is a simple text-based query language for filtering data.
* KQL only filters data, and has no role in aggregating, transforming, or sorting data.
* KQL is not to be confused with the <<lucene-query,Lucene query language>>, which has a different feature set.
Use KQL to filter documents where a value for a field exists, matches a given value, or is within a given range.
[discrete]
=== Filter for documents where a field exists
To filter documents for which an indexed value exists for a given field, use the `*` operator.
For example, to filter for documents where the `http.request.method` field exists, use the following syntax:
[source,yaml]
-------------------
http.request.method: *
-------------------
This checks for any indexed value, including an empty string.
[discrete]
=== Filter for documents that match a value
Use KQL to filter for documents that match a specific number, text, date, or boolean value.
For example, to filter for documents where the `http.request.method` is GET, use the following query:
[source,yaml]
-------------------
http.request.method: GET
-------------------
The field parameter is optional. If not provided, all fields are searched for the given value.
For example, to search all fields for “Hello”, use the following:
[source,yaml]
-------------------
Hello
-------------------
When querying keyword, numeric, date, or boolean fields, the value must be an exact match,
including punctuation and case. However, when querying text fields, {es} analyzes the
value provided according to the {ref}/analysis.html[fields mapping settings].
For example, to search for documents where `http.request.body.content` (a `text` field)
contains the text “null pointer”:
[source,yaml]
-------------------
http.request.body.content: null pointer
-------------------
Because this is a `text` field, the order of these search terms does not matter, and
even documents containing “pointer null” are returned. To search `text` fields where the
terms are in the order provided, surround the value in quotation marks, as follows:
[source,yaml]
-------------------
http.request.body.content: "null pointer"
-------------------
Certain characters must be escaped by a backslash (unless surrounded by quotes).
For example, to search for documents where `http.request.referrer` is https://example.com,
use either of the following queries:
[source,yaml]
-------------------
http.request.referrer: "https://example.com"
http.request.referrer: https\://example.com
-------------------
You must escape following characters:
[source,yaml]
-------------------
\():<>"*
-------------------
[discrete]
=== Filter for documents within a range
To search documents that contain terms within a provided range, use KQLs range syntax.
For example, to search for all documents for which `http.response.bytes` is less than 10000,
use the following syntax:
[source,yaml]
-------------------
http.response.bytes < 10000
-------------------
To search for an inclusive range, combine multiple range queries.
For example, to search for documents where `http.response.bytes` is greater than 10000
but less than or equal to 20000, use the following syntax:
[source,yaml]
-------------------
http.response.bytes > 10000 and http.response.bytes <= 20000
-------------------
You can also use range syntax for string values, IP addresses, and timestamps.
For example, to search for documents earlier than two weeks ago, use the following syntax:
[source,yaml]
-------------------
@timestamp < now-2w
-------------------
For more examples on acceptable date formats, refer to {ref}/common-options.html#date-math[Date Math].
[discrete]
=== Filter for documents using wildcards
To search for documents matching a pattern, use the wildcard syntax.
For example, to find documents where `http.response.status_code` begins with a 4, use the following syntax:
[source,yaml]
-------------------
http.response.status_code: 4*
-------------------
By default, leading wildcards are not allowed for performance reasons.
You can modify this with the <<query-allowleadingwildcards,`query:allowLeadingWildcards`>> advanced setting.
NOTE: Only `*` is currently supported. This matches zero or more characters.
[discrete]
=== Negating a query
To negate or exclude a set of documents, use the `not` keyword (not case-sensitive).
For example, to filter documents where the `http.request.method` is *not* GET, use the following query:
[source,yaml]
-------------------
NOT http.request.method: GET
-------------------
[discrete]
=== Combining multiple queries
To combine multiple queries, use the `and`/`or` keywords (not case-sensitive).
For example, to find documents where the `http.request.method` is GET *or* the `http.response.status_code` is 400,
use the following query:
[source,yaml]
-------------------
http.request.method: GET OR http.response.status_code: 400
-------------------
Similarly, to find documents where the `http.request.method` is GET *and* the
`http.response.status_code` is 400, use this query:
[source,yaml]
-------------------
http.request.method: GET AND http.response.status_code: 400
-------------------
To specify precedence when combining multiple queries, use parentheses.
For example, to find documents where the `http.request.method` is GET *and*
the `http.response.status_code` is 200, *or* the `http.request.method` is POST *and*
`http.response.status_code` is 400, use the following:
[source,yaml]
-------------------
(http.request.method: GET AND http.response.status_code: 200) OR
(http.request.method: POST AND http.response.status_code: 400)
-------------------
You can also use parentheses for shorthand syntax when querying multiple values for the same field.
For example, to find documents where the `http.request.method` is GET, POST, *or* DELETE, use the following:
[source,yaml]
-------------------
http.request.method: (GET OR POST OR DELETE)
-------------------
[discrete]
=== Matching multiple fields
Wildcards can also be used to query multiple fields. For example, to search for
documents where any sub-field of `datastream` contains “logs”, use the following:
[source,yaml]
-------------------
datastream.*: logs
-------------------
NOTE: When using wildcards to query multiple fields, errors might occur if the fields are of
different types. For example, if `datastream.*` matches both numeric and string fields, the
above query will result in an error because numeric fields cannot be queried for string values.
[discrete]
=== Querying nested fields
Querying {ref}/nested.html[nested fields] requires a special syntax. Consider the
following document, where `user` is a nested field:
[source,yaml]
-------------------
{
"user" : [
{
"first" : "John",
"last" : "Smith"
},
{
"first" : "Alice",
"last" : "White"
}
]
}
-------------------
To find documents where a single value inside the `user` array contains a first name of
“Alice” and last name of “White”, use the following:
[source,yaml]
-------------------
user:{ first: "Alice" and last: "White" }
-------------------
Because nested fields can be inside other nested fields,
you must specify the full path of the nested field you want to query.
For example, consider the following document where `user` and `names` are both nested fields:
[source,yaml]
-------------------
{
"user": [
{
"names": [
{
"first": "John",
"last": "Smith"
},
{
"first": "Alice",
"last": "White"
}
]
}
]
}
-------------------
To find documents where a single value inside the `user.names` array contains a first name of “Alice” *and*
last name of “White”, use the following:
[source,yaml]
-------------------
user.names:{ first: "Alice" and last: "White" }
-------------------