mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-06-28 09:28:55 -04:00
e12e83fa37
May we add a section to [this page](https://www.elastic.co/guide/en/elasticsearch/reference/master/troubleshooting-searches.html#troubleshooting-check-field-values) to query for the latest record on an index (pattern)? This will be helpful to decide between Kibana Discover filter and Elasticsearch ingest lag problems. --------- Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
339 lines
8.8 KiB
Text
339 lines
8.8 KiB
Text
|
||
[[troubleshooting-searches]]
|
||
== Troubleshooting searches
|
||
|
||
When you query your data, Elasticsearch may return an error, no search results,
|
||
or results in an unexpected order. This guide describes how to troubleshoot
|
||
searches.
|
||
|
||
[discrete]
|
||
[[troubleshooting-searches-exists]]
|
||
=== Ensure the data stream, index, or alias exists
|
||
|
||
Elasticsearch returns an `index_not_found_exception` when the data stream, index
|
||
or alias you try to query does not exist. This can happen when you misspell the
|
||
name or when the data has been indexed to a different data stream or index.
|
||
|
||
Use the <<indices-exists,exists API>> to check whether a data stream, index, or
|
||
alias exists:
|
||
|
||
[source,console]
|
||
----
|
||
HEAD my-data-stream
|
||
----
|
||
|
||
Use the <<data-stream-stats-api,data stream stats API>> to list all data
|
||
streams:
|
||
|
||
[source,console]
|
||
----
|
||
GET /_data_stream/_stats?human=true
|
||
----
|
||
|
||
Use the <<indices-get-index,get index API>> to list all indices and their
|
||
aliases:
|
||
|
||
[source,console]
|
||
----
|
||
GET _all?filter_path=*.aliases
|
||
----
|
||
|
||
Instead of an error, it is possible to retrieve partial search results if some
|
||
of the indices you're querying are unavailable. Set `ignore_unavailable` to
|
||
`true`:
|
||
|
||
[source,console]
|
||
----
|
||
GET /my-alias/_search?ignore_unavailable=true
|
||
----
|
||
|
||
[discrete]
|
||
[[troubleshooting-searches-data]]
|
||
=== Ensure the data stream or index contains data
|
||
|
||
When a search request returns no hits, the data stream or index may contain no
|
||
data. This can happen when there is a data ingestion issue. For example, the
|
||
data may have been indexed to a data stream or index with another name.
|
||
|
||
Use the <<search-count,count API>> to retrieve the number of documents in a data
|
||
stream or index. Check that `count` in the response is not 0.
|
||
|
||
////
|
||
[source,console]
|
||
----
|
||
PUT my-index-000001
|
||
{
|
||
"mappings": {
|
||
"properties": {
|
||
"@timestamp": {
|
||
"type": "date"
|
||
},
|
||
"my-field": {
|
||
"type": "keyword"
|
||
},
|
||
"my-num-field": {
|
||
"type": "integer"
|
||
}
|
||
}
|
||
}
|
||
}
|
||
----
|
||
////
|
||
|
||
[source,console]
|
||
----
|
||
GET /my-index-000001/_count
|
||
----
|
||
//TEST[continued]
|
||
|
||
NOTE: When getting no search results in {kib}, check that you have selected the
|
||
correct data view and a valid time range. Also, ensure the data view has been
|
||
configured with the correct time field.
|
||
|
||
[discrete]
|
||
[[troubleshooting-searches-field-exists-caps]]
|
||
=== Check that the field exists and its capabilities
|
||
|
||
Querying a field that does not exist will not return any results. Use the
|
||
<<search-field-caps,field capabilities API>> to check whether a field exists:
|
||
|
||
[source,console]
|
||
----
|
||
GET /my-index-000001/_field_caps?fields=my-field
|
||
----
|
||
//TEST[continued]
|
||
|
||
If the field does not exist, check the data ingestion process. The field may
|
||
have a different name.
|
||
|
||
If the field exists, the request will return the field's type and whether it is
|
||
searchable and aggregatable.
|
||
|
||
[source,console-response]
|
||
----
|
||
{
|
||
"indices": [
|
||
"my-index-000001"
|
||
],
|
||
"fields": {
|
||
"my-field": {
|
||
"keyword": {
|
||
"type": "keyword", <1>
|
||
"metadata_field": false,
|
||
"searchable": true, <2>
|
||
"aggregatable": true <3>
|
||
}
|
||
}
|
||
}
|
||
}
|
||
----
|
||
|
||
<1> The field is of type `keyword` in this index.
|
||
<2> The field is searchable in this index.
|
||
<3> The field is aggregatable in this index.
|
||
|
||
[discrete]
|
||
[[troubleshooting-searches-mappings]]
|
||
=== Check the field's mappings
|
||
|
||
A field's capabilities are determined by its <<mapping,mapping>>. To retrieve
|
||
the mapping, use the <<indices-get-mapping,get mapping API>>:
|
||
|
||
[source,console]
|
||
----
|
||
GET /my-index-000001/_mappings
|
||
----
|
||
//TEST[continued]
|
||
|
||
If you query a `text` field, pay attention to the analyzer that may have been
|
||
configured. You can use the <<indices-analyze,analyze API>> to check how a
|
||
field's analyzer processes values and query terms:
|
||
|
||
[source,console]
|
||
----
|
||
GET /my-index-000001/_analyze
|
||
{
|
||
"field" : "my-field",
|
||
"text" : "this is a test"
|
||
}
|
||
----
|
||
//TEST[continued]
|
||
|
||
To change the mapping of an existing field, refer to
|
||
<<updating-field-mappings,Changing the mapping of a field>>.
|
||
|
||
[discrete]
|
||
[[troubleshooting-check-field-values]]
|
||
=== Check the field's values
|
||
|
||
Use the <<query-dsl-exists-query,`exists` query>> to check whether there are
|
||
documents that return a value for a field. Check that `count` in the response is
|
||
not 0.
|
||
|
||
[source,console]
|
||
----
|
||
GET /my-index-000001/_count
|
||
{
|
||
"query": {
|
||
"exists": {
|
||
"field": "my-field"
|
||
}
|
||
}
|
||
}
|
||
----
|
||
//TEST[continued]
|
||
|
||
If the field is aggregatable, you can use <<search-aggregations,aggregations>>
|
||
to check the field's values. For `keyword` fields, you can use a
|
||
<<search-aggregations-bucket-terms-aggregation,terms aggregation>> to retrieve
|
||
the field's most common values:
|
||
|
||
[source,console]
|
||
----
|
||
GET /my-index-000001/_search?filter_path=aggregations
|
||
{
|
||
"size": 0,
|
||
"aggs": {
|
||
"top_values": {
|
||
"terms": {
|
||
"field": "my-field",
|
||
"size": 10
|
||
}
|
||
}
|
||
}
|
||
}
|
||
----
|
||
//TEST[continued]
|
||
|
||
For numeric fields, you can use the
|
||
<<search-aggregations-metrics-stats-aggregation,stats aggregation>> to get an
|
||
idea of the field's value distribution:
|
||
|
||
[source,console]
|
||
----
|
||
GET my-index-000001/_search?filter_path=aggregations
|
||
{
|
||
"aggs": {
|
||
"my-num-field-stats": {
|
||
"stats": {
|
||
"field": "my-num-field"
|
||
}
|
||
}
|
||
}
|
||
}
|
||
----
|
||
//TEST[continued]
|
||
|
||
If the field does not return any values, check the data ingestion process. The
|
||
field may have a different name.
|
||
|
||
[discrete]
|
||
[[troubleshooting-searches-latest-data]]
|
||
=== Check the latest value
|
||
|
||
For time-series data, confirm there is non-filtered data within the attempted
|
||
time range. For example, if you are trying to query the latest data for the
|
||
`@timestamp` field, run the following to see if the max `@timestamp` falls
|
||
within the attempted range:
|
||
|
||
[source,console]
|
||
----
|
||
GET my-index-000001/_search?sort=@timestamp:desc&size=1
|
||
----
|
||
//TEST[continued]
|
||
|
||
[discrete]
|
||
[[troubleshooting-searches-validate-explain-profile]]
|
||
=== Validate, explain, and profile queries
|
||
|
||
When a query returns unexpected results, Elasticsearch offers several tools to
|
||
investigate why.
|
||
|
||
The <<search-validate,validate API>> enables you to validate a query. Use the
|
||
`rewrite` parameter to return the Lucene query an Elasticsearch query is
|
||
rewritten into:
|
||
|
||
[source,console]
|
||
--------------------------------------------------
|
||
GET /my-index-000001/_validate/query?rewrite=true
|
||
{
|
||
"query": {
|
||
"match": {
|
||
"user.id": {
|
||
"query": "kimchy",
|
||
"fuzziness": "auto"
|
||
}
|
||
}
|
||
}
|
||
}
|
||
--------------------------------------------------
|
||
//TEST[continued]
|
||
|
||
Use the <<search-explain,explain API>> to find out why a specific document
|
||
matches or doesn’t match a query:
|
||
|
||
[source,console]
|
||
--------------------------------------------------
|
||
GET /my-index-000001/_explain/0
|
||
{
|
||
"query" : {
|
||
"match" : { "message" : "elasticsearch" }
|
||
}
|
||
}
|
||
--------------------------------------------------
|
||
// TEST[setup:messages]
|
||
|
||
The <<search-profile,profile API>> provides detailed timing information about a
|
||
search request. For a visual representation of the results, use the
|
||
{kibana-ref}/xpack-profiler.html[Search Profiler] in {kib}.
|
||
|
||
NOTE: To troubleshoot queries in {kib}, select **Inspect** in the toolbar. Next,
|
||
select **Request**. You can now copy the query {kib} sent to {es} for
|
||
further analysis in Console.
|
||
|
||
[discrete]
|
||
[[troubleshooting-searches-settings]]
|
||
=== Check index settings
|
||
|
||
<<index-modules-settings,Index settings>> can influence search results. For
|
||
example, the `index.query.default_field` setting, which determines the field
|
||
that is queried when a query specifies no explicit field. Use the
|
||
<<indices-get-settings,get index settings API>> to retrieve the settings for an
|
||
index:
|
||
|
||
[source,console]
|
||
----
|
||
GET /my-index-000001/_settings
|
||
----
|
||
//TEST[continued]
|
||
|
||
You can update dynamic index settings with the <<indices-update-settings,update
|
||
index settings API>>. <<change-dynamic-index-setting-for-a-data-stream,Changing
|
||
dynamic index settings for a data stream>> requires changing the index template
|
||
used by the data stream.
|
||
|
||
For static settings, you need to create a new index with the correct settings.
|
||
Next, you can reindex the data into that index. For data streams, refer to
|
||
<<change-static-index-setting-for-a-data-stream,Change a static index setting
|
||
for a data stream>>.
|
||
|
||
[discrete]
|
||
[[troubleshooting-slow-searches]]
|
||
=== Find slow queries
|
||
|
||
<<index-modules-slowlog,Slow logs>> can help pinpoint slow performing search
|
||
requests. Enabling <<auditing-settings,audit logging>> on top can help determine
|
||
query source. Add the following settings to the `elasticsearch.yml` configuration file
|
||
to trace queries. The resulting logging is verbose, so disable these settings when not
|
||
troubleshooting.
|
||
|
||
[source,yaml]
|
||
----
|
||
xpack.security.audit.enabled: true
|
||
xpack.security.audit.logfile.events.include: _all
|
||
xpack.security.audit.logfile.events.emit_request_body: true
|
||
----
|
||
|
||
Refer to
|
||
https://www.elastic.co/blog/advanced-tuning-finding-and-fixing-slow-elasticsearch-queries[Advanced
|
||
tuning: finding and fixing slow Elasticsearch queries] for more information.
|