kibana/x-pack/plugins
Michael Olorunnisola 543bf80673
[8.17] [Performance][Security Solution][1/4] - Field Browser Performance (#212469) (#213025)
# Backport

This will backport the following commits from `main` to `8.17`:
- [[Performance][Security Solution][1/4] - Field Browser Performance
(#212469)](https://github.com/elastic/kibana/pull/212469)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Michael
Olorunnisola","email":"michael.olorunnisola@elastic.co"},"sourceCommit":{"committedDate":"2025-03-04T01:22:25Z","message":"[Performance][Security
Solution][1/4] - Field Browser Performance (#212469)\n\n## Summary\nPart
1 of https://github.com/elastic/kibana/pull/212173\n\n### Testing\nFor
setup see testing section
here:\nhttps://github.com/elastic/kibana/pull/212173#issue-2870522020\n\n**Areas
to test:**\n- Alert Table (Alerts, Rule Detail, Rule Preview pages)\n-
Security solution field browser component\n- Flyout table tab.\n\n###
Background\n\nWhen investigating the performance of the security
solution application,\none of the issues that was observed was locking
of the page and field\nbrowser component when the total number of fields
returned were\nsignificantly high.\n\nThis led to cell values not
rendering in the alert table, and the field\nbrowser in all the security
solution pages causing the page to crash.\nThe relevant images can be
seen at the bottom of this description\n\nIn short: The
`push(...fields)` is non-performant at scale, and at a\nsignificant
enough scale (Testing was done with 500k mapped fields),\nfails to run
due to excessive arguments provided to the `push` method.\nIn this PR
improvements are made in the `browserFields` transformations\nthat are
done for the field browser component, expandable flyout table\ntab, and
alert/rule tables via `CellValue` component.\n\nThis work was done to
get immediate improvements in the security\nsolution UI, but a longer
term consideration will be whether or not the\n`browserFields` is even
necessary anymore as a concept based on what is\navailable via the
`fields` api. We will revisit once our Sourcerer\nrefactoring work is
done.\n\n<img width=\"1728\" alt=\"Screenshot 2025-02-26 at 10 15
29 AM\"\nsrc=\"https://github.com/user-attachments/assets/a25f577f-f758-415e-9c93-5452eadb8020\"\n/>\n\n<img
width=\"1445\" alt=\"Screenshot 2025-02-26 at 10 18
36 AM\"\nsrc=\"https://github.com/user-attachments/assets/d70970d3-991a-47ba-b617-5862d18101b6\"\n/>\n\n<img
width=\"1469\" alt=\"Screenshot 2025-02-26 at 10 19
48 AM\"\nsrc=\"4dff2378-d61b-4770-b46b-41cb37d6ead4)\n\n\n###
After the fix\n(Done on [this
branch](https://github.com/elastic/kibana/pull/212173)\nthat has the
other changes as
well)\n\n\nhttps://github.com/user-attachments/assets/da992296-4eb8-49d4-96ca-b0a19a00f1f0\n\n\n###
Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers
should verify this PR satisfies this list as well.\n\n- [x] [Unit or
functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere
updated or added to match the most common
scenarios","sha":"750e156c26078015025551c4f10d299ba269fa35","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Threat
Hunting","Team:Threat
Hunting:Investigations","backport:version","v8.18.0","v9.1.0","v8.17.3"],"title":"[Performance][Security
Solution][1/4] - Field Browser
Performance","number":212469,"url":"https://github.com/elastic/kibana/pull/212469","mergeCommit":{"message":"[Performance][Security
Solution][1/4] - Field Browser Performance (#212469)\n\n## Summary\nPart
1 of https://github.com/elastic/kibana/pull/212173\n\n### Testing\nFor
setup see testing section
here:\nhttps://github.com/elastic/kibana/pull/212173#issue-2870522020\n\n**Areas
to test:**\n- Alert Table (Alerts, Rule Detail, Rule Preview pages)\n-
Security solution field browser component\n- Flyout table tab.\n\n###
Background\n\nWhen investigating the performance of the security
solution application,\none of the issues that was observed was locking
of the page and field\nbrowser component when the total number of fields
returned were\nsignificantly high.\n\nThis led to cell values not
rendering in the alert table, and the field\nbrowser in all the security
solution pages causing the page to crash.\nThe relevant images can be
seen at the bottom of this description\n\nIn short: The
`push(...fields)` is non-performant at scale, and at a\nsignificant
enough scale (Testing was done with 500k mapped fields),\nfails to run
due to excessive arguments provided to the `push` method.\nIn this PR
improvements are made in the `browserFields` transformations\nthat are
done for the field browser component, expandable flyout table\ntab, and
alert/rule tables via `CellValue` component.\n\nThis work was done to
get immediate improvements in the security\nsolution UI, but a longer
term consideration will be whether or not the\n`browserFields` is even
necessary anymore as a concept based on what is\navailable via the
`fields` api. We will revisit once our Sourcerer\nrefactoring work is
done.\n\n<img width=\"1728\" alt=\"Screenshot 2025-02-26 at 10 15
29 AM\"\nsrc=\"https://github.com/user-attachments/assets/a25f577f-f758-415e-9c93-5452eadb8020\"\n/>\n\n<img
width=\"1445\" alt=\"Screenshot 2025-02-26 at 10 18
36 AM\"\nsrc=\"https://github.com/user-attachments/assets/d70970d3-991a-47ba-b617-5862d18101b6\"\n/>\n\n<img
width=\"1469\" alt=\"Screenshot 2025-02-26 at 10 19
48 AM\"\nsrc=\"4dff2378-d61b-4770-b46b-41cb37d6ead4)\n\n\n###
After the fix\n(Done on [this
branch](https://github.com/elastic/kibana/pull/212173)\nthat has the
other changes as
well)\n\n\nhttps://github.com/user-attachments/assets/da992296-4eb8-49d4-96ca-b0a19a00f1f0\n\n\n###
Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers
should verify this PR satisfies this list as well.\n\n- [x] [Unit or
functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere
updated or added to match the most common
scenarios","sha":"750e156c26078015025551c4f10d299ba269fa35"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","8.17"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/212469","number":212469,"mergeCommit":{"message":"[Performance][Security
Solution][1/4] - Field Browser Performance (#212469)\n\n## Summary\nPart
1 of https://github.com/elastic/kibana/pull/212173\n\n### Testing\nFor
setup see testing section
here:\nhttps://github.com/elastic/kibana/pull/212173#issue-2870522020\n\n**Areas
to test:**\n- Alert Table (Alerts, Rule Detail, Rule Preview pages)\n-
Security solution field browser component\n- Flyout table tab.\n\n###
Background\n\nWhen investigating the performance of the security
solution application,\none of the issues that was observed was locking
of the page and field\nbrowser component when the total number of fields
returned were\nsignificantly high.\n\nThis led to cell values not
rendering in the alert table, and the field\nbrowser in all the security
solution pages causing the page to crash.\nThe relevant images can be
seen at the bottom of this description\n\nIn short: The
`push(...fields)` is non-performant at scale, and at a\nsignificant
enough scale (Testing was done with 500k mapped fields),\nfails to run
due to excessive arguments provided to the `push` method.\nIn this PR
improvements are made in the `browserFields` transformations\nthat are
done for the field browser component, expandable flyout table\ntab, and
alert/rule tables via `CellValue` component.\n\nThis work was done to
get immediate improvements in the security\nsolution UI, but a longer
term consideration will be whether or not the\n`browserFields` is even
necessary anymore as a concept based on what is\navailable via the
`fields` api. We will revisit once our Sourcerer\nrefactoring work is
done.\n\n<img width=\"1728\" alt=\"Screenshot 2025-02-26 at 10 15
29 AM\"\nsrc=\"https://github.com/user-attachments/assets/a25f577f-f758-415e-9c93-5452eadb8020\"\n/>\n\n<img
width=\"1445\" alt=\"Screenshot 2025-02-26 at 10 18
36 AM\"\nsrc=\"https://github.com/user-attachments/assets/d70970d3-991a-47ba-b617-5862d18101b6\"\n/>\n\n<img
width=\"1469\" alt=\"Screenshot 2025-02-26 at 10 19
48 AM\"\nsrc=\"4dff2378-d61b-4770-b46b-41cb37d6ead4)\n\n\n###
After the fix\n(Done on [this
branch](https://github.com/elastic/kibana/pull/212173)\nthat has the
other changes as
well)\n\n\nhttps://github.com/user-attachments/assets/da992296-4eb8-49d4-96ca-b0a19a00f1f0\n\n\n###
Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers
should verify this PR satisfies this list as well.\n\n- [x] [Unit or
functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere
updated or added to match the most common
scenarios","sha":"750e156c26078015025551c4f10d299ba269fa35"}},{"branch":"8.17","label":"v8.17.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
2025-03-04 11:56:13 -06:00
..
actions [8.17] [ResponseOps][Actions] Allow to delete connectors with unsupported connector type using the API (#208033) (#208659) 2025-01-29 03:22:45 +01:00
ai_infra [8.17] [ES3] Enable AI assistant knowledge base (#202210) (#202387) 2024-11-30 09:35:18 -06:00
aiops [8.x] [ML] AiOps: Action for adding Log Rate analysis embeddable to a dashboard (#200557) (#200891) 2024-11-20 06:48:58 -06:00
alerting [8.17] [ResponseOps][Rules] Validate timezone in rule routes (#201508) (#208299) 2025-01-28 09:01:19 +01:00
banners [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
canvas [8.17] chore(NA): upgrade to webpack 5 (#191106) (#211148) 2025-02-19 03:48:08 +00:00
cases [8.17] [Response Ops][Cases] Cases with empty string assignees throwing error (#209973) (#210755) 2025-02-12 16:49:46 +01:00
cloud [8.x] [CLOUD-UI] Cloud onboarding token (#198444) (#200832) 2024-11-19 19:29:49 -06:00
cloud_defend [8.x] Authorized route migration for routes owned by @elastic/kibana-cloud-security-posture (#198189) (#200714) 2024-11-19 08:08:37 -06:00
cloud_integrations [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
cloud_security_posture [8.17] Fix Chart Trend by Severity Bug (#212358) 2025-02-25 13:28:54 +00:00
cross_cluster_replication [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
custom_branding [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
dashboard_enhanced [8.x] Address some of the no_group_crossing dependencies (#198261) (#199165) 2024-11-08 10:46:18 -06:00
data_quality [8.x] Address some of the no_group_crossing dependencies (#198261) (#199165) 2024-11-08 10:46:18 -06:00
data_usage [8.x] [Data Usage] remove autoops.api.tls.ca config (#200808) (#201026) 2024-11-20 14:52:51 -06:00
data_visualizer [8.x] [ML][ES|QL] Adds query guardrails and technical preview badge to ES|QL data visualizer (#200325) (#200677) 2024-11-19 03:59:53 -06:00
discover_enhanced [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
drilldowns [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
ecs_data_quality_dashboard [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
elastic_assistant [8.17] [Bug][Assistant API] - chat/complete endpoint is not persisting the model response to the chosen conversation ID (#11783) (#212122) (#212505) 2025-02-26 16:23:15 +00:00
embeddable_enhanced [8.x] [Dashboard] Fix incorrect actions in edit mode after discarding unsaved changes (#200230) (#200279) 2024-11-14 18:53:56 -06:00
encrypted_saved_objects [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
enterprise_search [8.17] [Search] Don't error out on missing connectors permissions (#212622) (#213091) 2025-03-04 11:27:23 -05:00
entity_manager [8.17] [Inventory] Change discover link to use entity definition (#201433) (#202153) 2024-11-29 08:23:46 -06:00
event_log [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
features [8.x] [Cases] [Security Solution] New cases subfeatures, add comments and reopen cases (#194898) (#200807) 2024-11-19 15:13:06 -06:00
fields_metadata [8.17] Update platform security modules (main) (#206227) (#207316) 2025-01-22 14:01:19 +01:00
file_upload [8.x] Authorized route migration for routes owned by @elastic/kibana-presentation (#198193) (#199979) 2024-11-13 06:59:31 -06:00
fleet [8.17] [Fleet] Fix API key insertion into standalone agent policy (#212440) (#212578) 2025-02-26 21:03:35 +00:00
global_search [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
global_search_bar [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
global_search_providers [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
graph [8.x] [EuiInMemoryTable] Persist table rows per page and sort (#198297) (#200569) 2024-11-18 09:22:33 -06:00
grokdebugger [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
index_lifecycle_management [8.x] [Index Management] Feature add button handler for the ilm banner via extension service (#199077) (#200965) 2024-11-20 11:25:43 -06:00
index_management [8.17] Remove inference_id field if no inference endpoint is selected (#205660) (#209351) 2025-02-03 18:50:32 +01:00
inference [8.x] [inference] add pre-bound versions of &#x60;chatComplete&#x60; and &#x60;output&#x60; APIs (#200568) (#201028) 2024-11-20 14:54:41 -06:00
ingest_pipelines [8.17] chore(NA): upgrade to webpack 5 (#191106) (#211148) 2025-02-19 03:48:08 +00:00
integration_assistant [8.17] [Automatic Import] Fix Structured log flow to handle different type of structured syslogs (#212611) (#212645) 2025-02-27 14:37:36 +00:00
kubernetes_security [8.x] Unauthorized route migration for routes owned by kibana-cloud-security-posture (#198353) (#200282) 2024-11-14 19:39:12 -06:00
lens [8.17] [Lens] Restore dynamic colouring by value for Last value agg (#209110) (#210878) 2025-02-12 19:12:27 +01:00
license_api_guard [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
license_management [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
licensing [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
lists [8.x] Authorized route migration for routes owned by @elastic/security-detection-engine (#198195) (#199752) 2024-11-19 08:42:26 -06:00
logstash [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
maps [8.x] [Papercut] Fix KQL parsing error text styling (#199985) (#200543) 2024-11-18 07:35:14 -06:00
ml [8.17] Update @elastic/ml-ui dependencies (main) (#204210) (#207794) 2025-01-22 16:42:41 +00:00
monitoring [8.17] [Stack Monitoring] Do not add empty include array when retrieving logstash pipelines (#202039) (#202100) 2024-11-28 00:18:55 -06:00
monitoring_collection [8.17] [Monitoring] Make monitoring collection API public again (#205190) (#205199) 2024-12-27 08:02:09 -06:00
notifications [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
observability_solution [8.17] Fix panel to remove query filter for .NET OTel runtime metrics on curated dashboard (#212529) (#212722) 2025-02-28 09:13:34 +00:00
osquery [8.17] [EDR Workflows][Osquery] OpenApi Missing Content (#212032) (#212640) 2025-02-27 14:51:28 +00:00
painless_lab [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
remote_clusters [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
reporting [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
rollup [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
rule_registry [8.17] Set refresh according to stateful vs stateless when indexing alert documents (#201209) (#202221) 2024-11-29 12:54:23 +01:00
runtime_fields [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
saved_objects_tagging [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
screenshotting [8.x] Address some of the no_group_crossing dependencies (#198261) (#199165) 2024-11-08 10:46:18 -06:00
search_assistant [8.17] [Search] Add ML as required plugin to Search Assistant (#204009) (#204036) 2024-12-12 10:24:38 -06:00
search_connectors [8.x] Categorise Kibana plugins - Wave 2 (#199302) (#199461) 2024-11-09 10:18:22 -06:00
search_homepage [8.x] Categorise Kibana plugins - Wave 2 (#199302) (#199461) 2024-11-09 10:18:22 -06:00
search_indices [8.17] [Search] [Onboarding] Update example data for index (#201983) (#203142) 2024-12-05 19:13:37 +00:00
search_inference_endpoints [8.17] Adding Tech Preview badge for Reranker (#202561) (#203026) 2024-12-04 18:08:49 -06:00
search_notebooks [8.x] Categorise Kibana plugins - Wave 2 (#199302) (#199461) 2024-11-09 10:18:22 -06:00
search_playground [8.17] [ES3] Enable AI assistant knowledge base (#202210) (#202387) 2024-11-30 09:35:18 -06:00
searchprofiler [8.17] [search profiler] Move profile button inline with index field (#202253) (#208821) 2025-01-30 08:54:01 +01:00
security [8.17] Log request ID in SAML provider (#211877) (#212218) 2025-02-24 12:57:19 +00:00
security_solution [8.17] [Performance][Security Solution][1/4] - Field Browser Performance (#212469) (#213025) 2025-03-04 11:56:13 -06:00
security_solution_ess [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
security_solution_serverless [8.17] Only show data warning screen on Serverless (#201920) (#210383) 2025-02-10 16:38:34 +01:00
serverless [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
serverless_observability [8.17] [ES3] Enable AI assistant knowledge base (#202210) (#202387) 2024-11-30 09:35:18 -06:00
serverless_search [8.17] [Search] Add ML as required plugin to Search Assistant (#204009) (#204036) 2024-12-12 10:24:38 -06:00
session_view [8.x] Unauthorized route migration for routes owned by kibana-cloud-security-posture (#198353) (#200282) 2024-11-14 19:39:12 -06:00
snapshot_restore [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00
spaces [8.17] [Spaces UI] Role Editor Flyout Should Match in Roles Mgmt (#198182) (#202821) 2024-12-04 19:51:52 -05:00
stack_alerts [8.17] [ResponseOps] Do not change time field when edit (#206858) (#208030) 2025-01-23 13:31:16 +00:00
stack_connectors [8.17] Fix calling Sonnet 3.7 (#212372) (#212411) 2025-02-25 13:32:48 -05:00
task_manager [8.17] Exclude unrecognized tasks from the task manager aggregate API (#202163) (#202682) 2024-12-03 07:20:16 -06:00
telemetry_collection_xpack [8.17] Adding telemetry for the fips config (#201282) (#202453) 2024-12-02 08:52:55 -06:00
threat_intelligence [8.17] [Performance][Security Solution][4/4] - General Performance changes (#212488) (#212992) 2025-03-04 11:55:41 -06:00
timelines [8.17] [Security Solution] Fix timeline dynamic batching (#204034) | [ Security Solution ] Fix Refetch logic with new timeline batching (#205893) (#205676) 2025-02-06 11:09:06 -06:00
transform [8.17] [ML] Transforms: Support wildcards in the alerting rule flyout (#204226) (#204713) 2024-12-18 06:34:58 -06:00
translations [8.17] [Automatic Import] Remove pipeline tests from the package (#211223) (#211405) 2025-02-17 15:57:08 +01:00
triggers_actions_ui [8.17] [Performance][Security Solution][1/4] - Field Browser Performance (#212469) (#213025) 2025-03-04 11:56:13 -06:00
upgrade_assistant [8.17] [UA] Remove noisey warn when deleting from &#x60;.tasks&#x60; (#204720) (#204758) 2024-12-18 09:01:19 -06:00
watcher [8.x] [Sustainable Kibana Architecture] Update plugins (wave #1) (#195375) (#199268) 2024-11-08 07:22:05 -06:00