kibana/x-pack/plugins/maps
Kibana Machine 7d33d5a8e6
[8.11] [fix] Using max_result_window to set up MVT size request leads to not showing all results (#171344) (#171440)
# Backport

This will backport the following commits from `main` to `8.11`:
- [[fix] Using max_result_window to set up MVT size request leads to not
showing all results
(#171344)](https://github.com/elastic/kibana/pull/171344)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Nathan
Reese","email":"reese.nathan@elastic.co"},"sourceCommit":{"committedDate":"2023-11-16T19:11:13Z","message":"[fix]
Using max_result_window to set up MVT size request leads to not showing
all results (#171344)\n\nCloses
https://github.com/elastic/kibana/issues/170272\r\n\r\nThe cause of the
problem is that Maps is not
setting\r\n[_mvt](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-vector-tile-api.html)\r\n`size`
parameter.\r\n\r\nLets illustrate the problem with an example.\r\n*
index.max_result_window is set to 20,000.\r\n* There are 15,000 hits
matching the query for the tile, but because\r\n`size` is not provided,
the tile returns the default size value of\r\n10,000 hits.\r\n* Tile
request sets `track_total_hits` to 20,001. Tile meta
returns\r\n`\"hits.total.relation\": \"eq\"` and `\"hits.total.value\":
15000`\r\nproperties, indicating that all possible hits are returned in
the tile.\r\n* Maps UI does not display \"incomplete results\" even
though 5000 hits\r\nare not contained in the tile.\r\n* This happens
because `hits.total` properties from tile meta is used to\r\nindicate
\"incomplete results\" and the returned value indicates that
all\r\nresults are contained in the tile. We know this is not true
because the\r\ntile actually only returned 10,000 hits and not all
15,000 matching\r\nhits. Thus, without setting `size`, the tile is not
returning all of the\r\nhits indicated by `track_total_hits`.\r\n\r\n###
test instructions\r\n1. install web logs sample data set\r\n2. In dev
tools, run the following\r\n ```\r\n POST
kibana_sample_data_logs/_doc/\r\n {\r\n \"geo\": {\r\n \"coordinates\":
\"59,-106.5\"\r\n },\r\n \"@timestamp\": \"2023-08-01\"\r\n }\r\n\r\n
PUT /kibana_sample_data_logs/_settings\r\n {\r\n \"index\" : {\r\n
\"max_result_window\" : 20000\r\n }\r\n }\r\n ```\r\n3. Create new
map\r\n4. Set time range to `1 year ago to 1 year from now` to display
all web\r\nlogs data\r\n5. add documents layer. Verify point circled in
image is displayed.\r\nWithout changes, this point would not be returned
in the tile because\r\nthe tile would only return the first 10000
matches.\r\n<img width=\"500\" alt=\"Screenshot 2023-11-15 at 11 31 25
AM\"\r\nsrc=\"2249d571-24c4-41ad-975e-d44ee6d246ce\">","sha":"95a8d6a18f4f6e93ede4efdaaca880fc94378fb1","branchLabelMapping":{"^v8.12.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Presentation","Feature:Maps","v8.12.0","v8.11.2"],"number":171344,"url":"https://github.com/elastic/kibana/pull/171344","mergeCommit":{"message":"[fix]
Using max_result_window to set up MVT size request leads to not showing
all results (#171344)\n\nCloses
https://github.com/elastic/kibana/issues/170272\r\n\r\nThe cause of the
problem is that Maps is not
setting\r\n[_mvt](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-vector-tile-api.html)\r\n`size`
parameter.\r\n\r\nLets illustrate the problem with an example.\r\n*
index.max_result_window is set to 20,000.\r\n* There are 15,000 hits
matching the query for the tile, but because\r\n`size` is not provided,
the tile returns the default size value of\r\n10,000 hits.\r\n* Tile
request sets `track_total_hits` to 20,001. Tile meta
returns\r\n`\"hits.total.relation\": \"eq\"` and `\"hits.total.value\":
15000`\r\nproperties, indicating that all possible hits are returned in
the tile.\r\n* Maps UI does not display \"incomplete results\" even
though 5000 hits\r\nare not contained in the tile.\r\n* This happens
because `hits.total` properties from tile meta is used to\r\nindicate
\"incomplete results\" and the returned value indicates that
all\r\nresults are contained in the tile. We know this is not true
because the\r\ntile actually only returned 10,000 hits and not all
15,000 matching\r\nhits. Thus, without setting `size`, the tile is not
returning all of the\r\nhits indicated by `track_total_hits`.\r\n\r\n###
test instructions\r\n1. install web logs sample data set\r\n2. In dev
tools, run the following\r\n ```\r\n POST
kibana_sample_data_logs/_doc/\r\n {\r\n \"geo\": {\r\n \"coordinates\":
\"59,-106.5\"\r\n },\r\n \"@timestamp\": \"2023-08-01\"\r\n }\r\n\r\n
PUT /kibana_sample_data_logs/_settings\r\n {\r\n \"index\" : {\r\n
\"max_result_window\" : 20000\r\n }\r\n }\r\n ```\r\n3. Create new
map\r\n4. Set time range to `1 year ago to 1 year from now` to display
all web\r\nlogs data\r\n5. add documents layer. Verify point circled in
image is displayed.\r\nWithout changes, this point would not be returned
in the tile because\r\nthe tile would only return the first 10000
matches.\r\n<img width=\"500\" alt=\"Screenshot 2023-11-15 at 11 31 25
AM\"\r\nsrc=\"2249d571-24c4-41ad-975e-d44ee6d246ce\">","sha":"95a8d6a18f4f6e93ede4efdaaca880fc94378fb1"}},"sourceBranch":"main","suggestedTargetBranches":["8.11"],"targetPullRequestStates":[{"branch":"main","label":"v8.12.0","labelRegex":"^v8.12.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/171344","number":171344,"mergeCommit":{"message":"[fix]
Using max_result_window to set up MVT size request leads to not showing
all results (#171344)\n\nCloses
https://github.com/elastic/kibana/issues/170272\r\n\r\nThe cause of the
problem is that Maps is not
setting\r\n[_mvt](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-vector-tile-api.html)\r\n`size`
parameter.\r\n\r\nLets illustrate the problem with an example.\r\n*
index.max_result_window is set to 20,000.\r\n* There are 15,000 hits
matching the query for the tile, but because\r\n`size` is not provided,
the tile returns the default size value of\r\n10,000 hits.\r\n* Tile
request sets `track_total_hits` to 20,001. Tile meta
returns\r\n`\"hits.total.relation\": \"eq\"` and `\"hits.total.value\":
15000`\r\nproperties, indicating that all possible hits are returned in
the tile.\r\n* Maps UI does not display \"incomplete results\" even
though 5000 hits\r\nare not contained in the tile.\r\n* This happens
because `hits.total` properties from tile meta is used to\r\nindicate
\"incomplete results\" and the returned value indicates that
all\r\nresults are contained in the tile. We know this is not true
because the\r\ntile actually only returned 10,000 hits and not all
15,000 matching\r\nhits. Thus, without setting `size`, the tile is not
returning all of the\r\nhits indicated by `track_total_hits`.\r\n\r\n###
test instructions\r\n1. install web logs sample data set\r\n2. In dev
tools, run the following\r\n ```\r\n POST
kibana_sample_data_logs/_doc/\r\n {\r\n \"geo\": {\r\n \"coordinates\":
\"59,-106.5\"\r\n },\r\n \"@timestamp\": \"2023-08-01\"\r\n }\r\n\r\n
PUT /kibana_sample_data_logs/_settings\r\n {\r\n \"index\" : {\r\n
\"max_result_window\" : 20000\r\n }\r\n }\r\n ```\r\n3. Create new
map\r\n4. Set time range to `1 year ago to 1 year from now` to display
all web\r\nlogs data\r\n5. add documents layer. Verify point circled in
image is displayed.\r\nWithout changes, this point would not be returned
in the tile because\r\nthe tile would only return the first 10000
matches.\r\n<img width=\"500\" alt=\"Screenshot 2023-11-15 at 11 31 25
AM\"\r\nsrc=\"2249d571-24c4-41ad-975e-d44ee6d246ce\">","sha":"95a8d6a18f4f6e93ede4efdaaca880fc94378fb1"}},{"branch":"8.11","label":"v8.11.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Nathan Reese <reese.nathan@elastic.co>
2023-11-16 13:20:14 -07:00
..
common [8.11] [fix] Using max_result_window to set up MVT size request leads to not showing all results (#171344) (#171440) 2023-11-16 13:20:14 -07:00
public [8.11] [fix] Using max_result_window to set up MVT size request leads to not showing all results (#171344) (#171440) 2023-11-16 13:20:14 -07:00
server [CM] Soften response validation (#166919) 2023-09-28 16:33:04 +02:00
config.ts [Breaking] Remove deprecated enabled settings from plugins. (#113495) 2021-10-17 16:54:30 +01:00
jest.config.js [jest] update config files to get coverage per plugin (#111299) 2021-09-09 08:14:56 +02:00
kibana.jsonc [Serverless] Partially fix lens/maps/visualize breadcrumbs missing title (#163476) 2023-08-11 10:33:50 +02:00
README.md [ftr] automatically determine config run order (#130983) 2022-05-04 17:05:58 -05:00
tsconfig.json [CM] Soften response validation (#166919) 2023-09-28 16:33:04 +02:00

Maps

Visualize geo data from Elasticsearch or 3rd party geo-services.

Testing

Run all tests from the x-pack root directory

  • Unit tests: yarn test:jest x-pack/plugins/maps --watch
  • Functional tests:
    • Run node scripts/functional_tests_server
    • Run node ../scripts/functional_test_runner.js --config ./test/functional/apps/maps/group1/config.ts
    • Run node ../scripts/functional_test_runner.js --config ./test/functional/apps/maps/group2/config.ts
    • Run node ../scripts/functional_test_runner.js --config ./test/functional/apps/maps/group3/config.ts
    • Run node ../scripts/functional_test_runner.js --config ./test/functional/apps/maps/group4/config.ts