mirror of
https://github.com/elastic/kibana.git
synced 2025-04-22 08:49:27 -04:00
# Backport This will backport the following commits from `main` to `8.18`: - [[Cases] Improve unit test flakiness (#212489)](https://github.com/elastic/kibana/pull/212489) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Christos Nasikas","email":"christos.nasikas@elastic.co"},"sourceCommit":{"committedDate":"2025-03-14T17:17:13Z","message":"[Cases] Improve unit test flakiness (#212489)\n\n## Summary\n\nThis PR attempts (I have lost count of how many there have been so far)\nto stabilize the flakiness of cases tests.\n\n## Approach\n\nBased on our investigations, I believe a common factor that causes all\ntests to time out is how we wrap the tests with the needed providers.\nInstead of figuring out why they time out (which is very difficult), I\ntried a different approach. I rewrote how we initialize the testing\ndependencies, mocks, and providers on tests. To test my theory, I\ncreated a VM instance in GCloud with the same configuration as the one\nrunning in the CI, specifically the n2-standard-4 (4 vCPUs, 16 GB\nMemory) machine type. I ran the tests 100 times, which took almost two\ndays. In 10 of the runs, a random test was timeouted. I noticed that the\nmachine was stressed while running the tests, and the CPU frequently\nspiked to 100%, especially at the beginning of each test. Then, I\nincreased the timeout for all cases tests to 10 seconds and ran the\ntests again 100 times. No timeouts occurred. Lastly, I created a VM\ninstance in GCloud with the same configuration as the one running in the\nCI, specifically the n2-standard-8 (8 vCPUs, 32 GB Memory) machine type.\nI ran the tests again 100 times. In 1 of the runs, a random test was\ntimeouted. The machine on the CI cannot handle the cases tests. I\nbelieve the work in this PR is a step in the right direction either way,\nand we will benefit from it. I also believe increasing the timeout is a\ngood decision as we need it based on the experiments and the research we\nhave done in the last months.\n\nCPU usage on n2-standard-4\n\n\nhttps://github.com/user-attachments/assets/36b035df-310f-4906-98ba-688d57b97c7e\n\nCPU usage on n2-standard-8\n\n\nhttps://github.com/user-attachments/assets/3b1b6351-d48e-41da-a413-a56e52b54b82\n\n## RLT eslint rules\nI enabled the [RLT eslint\nrules](https://testing-library.com/docs/ecosystem-eslint-plugin-testing-library/)\nfor Cases and resolved any eslint errors. The process revealed small\nbugs in the tests, which I fixed them.\n\n## Testing utils\n\nI removed the `appMockRender` and any usage in favor of the new\n`renderWithTestingProviders` utility function and the improved\n`TestProviders` component. The `renderWithTestingProviders` follows the\nprincipals\n[suggested](https://github.com/testing-library/react-testing-library/issues/780#issuecomment-689053441)\nby the RLT team. Specifically:\n- The `renderWithTestingProviders` is a wrapper of the `render` function\nof the RTL library.\n- The `renderWithTestingProviders` does not create the services or any\ncomponent inside it.\n- The `renderWithTestingProviders` cannot be used in `beforeEach`\nfunctions. It should be called separately on each test.\n- The `renderWithTestingProviders` accepts props to override the default\nmocks.\n- The `renderWithTestingProviders` passes the `TestProviders` in the\n`wrapper` argument of the RLT `render` function.\n- The `TestProviders` component initializes and memoizes all services\nand dependencies. It accepts props to override the default mocks.\n- Mock overrides (like core services) should be created and passed to\n`renderWithTestingProviders` on each test, even if it means duplication.\nWe favor test isolation.\n\n### Checklist\n\nCheck the PR satisfies the following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n\n## Issues\n<details>\n<summary>List</summary>\n\nFixes: https://github.com/elastic/kibana/issues/207712\nFixes: https://github.com/elastic/kibana/issues/192739\nFixes: https://github.com/elastic/kibana/issues/174682\nFixes: https://github.com/elastic/kibana/issues/206366\nFixes: https://github.com/elastic/kibana/issues/207427\nFixes: https://github.com/elastic/kibana/issues/175239\nFixes: https://github.com/elastic/kibana/issues/177334\nFixes: https://github.com/elastic/kibana/issues/208443\nFixes: https://github.com/elastic/kibana/issues/187526\nFixes: https://github.com/elastic/kibana/issues/208310\nFixes: https://github.com/elastic/kibana/issues/192640\nFixes: https://github.com/elastic/kibana/issues/207077\nFixes: https://github.com/elastic/kibana/issues/197304\nFixes: https://github.com/elastic/kibana/issues/207249\nFixes: https://github.com/elastic/kibana/issues/202761\nFixes: https://github.com/elastic/kibana/issues/202115\nFixes: https://github.com/elastic/kibana/issues/193026\nFixes: https://github.com/elastic/kibana/issues/177304\nFixes: https://github.com/elastic/kibana/issues/208415\nFixes: https://github.com/elastic/kibana/issues/174661\nFixes: https://github.com/elastic/kibana/issues/201611\nFixes: https://github.com/elastic/kibana/issues/182364\nFixes: https://github.com/elastic/kibana/issues/175841\nFixes: https://github.com/elastic/kibana/issues/207907\nFixes: https://github.com/elastic/kibana/issues/171177\nFixes: https://github.com/elastic/kibana/issues/196628\nFixes: https://github.com/elastic/kibana/issues/194703\nFixes: https://github.com/elastic/kibana/issues/207241\nFixes: https://github.com/elastic/kibana/issues/206056\nFixes: https://github.com/elastic/kibana/issues/207328\nFixes: https://github.com/elastic/kibana/issues/205953\nFixes: https://github.com/elastic/kibana/issues/176524\nFixes: https://github.com/elastic/kibana/issues/176335\nFixes: https://github.com/elastic/kibana/issues/207404\nFixes: https://github.com/elastic/kibana/issues/207384\nFixes: https://github.com/elastic/kibana/issues/208380\nFixes: https://github.com/elastic/kibana/issues/207248\nFixes: https://github.com/elastic/kibana/issues/207444\nFixes: https://github.com/elastic/kibana/issues/175240\nFixes: https://github.com/elastic/kibana/issues/178001\n\n</details>\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"1054799f9c1139c52d74c9c588f60177182919c7","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:ResponseOps","Feature:Cases","backport:version","v8.18.0","v9.1.0","v8.19.0"],"title":"[Cases] Improve unit test flakiness","number":212489,"url":"https://github.com/elastic/kibana/pull/212489","mergeCommit":{"message":"[Cases] Improve unit test flakiness (#212489)\n\n## Summary\n\nThis PR attempts (I have lost count of how many there have been so far)\nto stabilize the flakiness of cases tests.\n\n## Approach\n\nBased on our investigations, I believe a common factor that causes all\ntests to time out is how we wrap the tests with the needed providers.\nInstead of figuring out why they time out (which is very difficult), I\ntried a different approach. I rewrote how we initialize the testing\ndependencies, mocks, and providers on tests. To test my theory, I\ncreated a VM instance in GCloud with the same configuration as the one\nrunning in the CI, specifically the n2-standard-4 (4 vCPUs, 16 GB\nMemory) machine type. I ran the tests 100 times, which took almost two\ndays. In 10 of the runs, a random test was timeouted. I noticed that the\nmachine was stressed while running the tests, and the CPU frequently\nspiked to 100%, especially at the beginning of each test. Then, I\nincreased the timeout for all cases tests to 10 seconds and ran the\ntests again 100 times. No timeouts occurred. Lastly, I created a VM\ninstance in GCloud with the same configuration as the one running in the\nCI, specifically the n2-standard-8 (8 vCPUs, 32 GB Memory) machine type.\nI ran the tests again 100 times. In 1 of the runs, a random test was\ntimeouted. The machine on the CI cannot handle the cases tests. I\nbelieve the work in this PR is a step in the right direction either way,\nand we will benefit from it. I also believe increasing the timeout is a\ngood decision as we need it based on the experiments and the research we\nhave done in the last months.\n\nCPU usage on n2-standard-4\n\n\nhttps://github.com/user-attachments/assets/36b035df-310f-4906-98ba-688d57b97c7e\n\nCPU usage on n2-standard-8\n\n\nhttps://github.com/user-attachments/assets/3b1b6351-d48e-41da-a413-a56e52b54b82\n\n## RLT eslint rules\nI enabled the [RLT eslint\nrules](https://testing-library.com/docs/ecosystem-eslint-plugin-testing-library/)\nfor Cases and resolved any eslint errors. The process revealed small\nbugs in the tests, which I fixed them.\n\n## Testing utils\n\nI removed the `appMockRender` and any usage in favor of the new\n`renderWithTestingProviders` utility function and the improved\n`TestProviders` component. The `renderWithTestingProviders` follows the\nprincipals\n[suggested](https://github.com/testing-library/react-testing-library/issues/780#issuecomment-689053441)\nby the RLT team. Specifically:\n- The `renderWithTestingProviders` is a wrapper of the `render` function\nof the RTL library.\n- The `renderWithTestingProviders` does not create the services or any\ncomponent inside it.\n- The `renderWithTestingProviders` cannot be used in `beforeEach`\nfunctions. It should be called separately on each test.\n- The `renderWithTestingProviders` accepts props to override the default\nmocks.\n- The `renderWithTestingProviders` passes the `TestProviders` in the\n`wrapper` argument of the RLT `render` function.\n- The `TestProviders` component initializes and memoizes all services\nand dependencies. It accepts props to override the default mocks.\n- Mock overrides (like core services) should be created and passed to\n`renderWithTestingProviders` on each test, even if it means duplication.\nWe favor test isolation.\n\n### Checklist\n\nCheck the PR satisfies the following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n\n## Issues\n<details>\n<summary>List</summary>\n\nFixes: https://github.com/elastic/kibana/issues/207712\nFixes: https://github.com/elastic/kibana/issues/192739\nFixes: https://github.com/elastic/kibana/issues/174682\nFixes: https://github.com/elastic/kibana/issues/206366\nFixes: https://github.com/elastic/kibana/issues/207427\nFixes: https://github.com/elastic/kibana/issues/175239\nFixes: https://github.com/elastic/kibana/issues/177334\nFixes: https://github.com/elastic/kibana/issues/208443\nFixes: https://github.com/elastic/kibana/issues/187526\nFixes: https://github.com/elastic/kibana/issues/208310\nFixes: https://github.com/elastic/kibana/issues/192640\nFixes: https://github.com/elastic/kibana/issues/207077\nFixes: https://github.com/elastic/kibana/issues/197304\nFixes: https://github.com/elastic/kibana/issues/207249\nFixes: https://github.com/elastic/kibana/issues/202761\nFixes: https://github.com/elastic/kibana/issues/202115\nFixes: https://github.com/elastic/kibana/issues/193026\nFixes: https://github.com/elastic/kibana/issues/177304\nFixes: https://github.com/elastic/kibana/issues/208415\nFixes: https://github.com/elastic/kibana/issues/174661\nFixes: https://github.com/elastic/kibana/issues/201611\nFixes: https://github.com/elastic/kibana/issues/182364\nFixes: https://github.com/elastic/kibana/issues/175841\nFixes: https://github.com/elastic/kibana/issues/207907\nFixes: https://github.com/elastic/kibana/issues/171177\nFixes: https://github.com/elastic/kibana/issues/196628\nFixes: https://github.com/elastic/kibana/issues/194703\nFixes: https://github.com/elastic/kibana/issues/207241\nFixes: https://github.com/elastic/kibana/issues/206056\nFixes: https://github.com/elastic/kibana/issues/207328\nFixes: https://github.com/elastic/kibana/issues/205953\nFixes: https://github.com/elastic/kibana/issues/176524\nFixes: https://github.com/elastic/kibana/issues/176335\nFixes: https://github.com/elastic/kibana/issues/207404\nFixes: https://github.com/elastic/kibana/issues/207384\nFixes: https://github.com/elastic/kibana/issues/208380\nFixes: https://github.com/elastic/kibana/issues/207248\nFixes: https://github.com/elastic/kibana/issues/207444\nFixes: https://github.com/elastic/kibana/issues/175240\nFixes: https://github.com/elastic/kibana/issues/178001\n\n</details>\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"1054799f9c1139c52d74c9c588f60177182919c7"}},"sourceBranch":"main","suggestedTargetBranches":["8.18","8.x"],"targetPullRequestStates":[{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/212489","number":212489,"mergeCommit":{"message":"[Cases] Improve unit test flakiness (#212489)\n\n## Summary\n\nThis PR attempts (I have lost count of how many there have been so far)\nto stabilize the flakiness of cases tests.\n\n## Approach\n\nBased on our investigations, I believe a common factor that causes all\ntests to time out is how we wrap the tests with the needed providers.\nInstead of figuring out why they time out (which is very difficult), I\ntried a different approach. I rewrote how we initialize the testing\ndependencies, mocks, and providers on tests. To test my theory, I\ncreated a VM instance in GCloud with the same configuration as the one\nrunning in the CI, specifically the n2-standard-4 (4 vCPUs, 16 GB\nMemory) machine type. I ran the tests 100 times, which took almost two\ndays. In 10 of the runs, a random test was timeouted. I noticed that the\nmachine was stressed while running the tests, and the CPU frequently\nspiked to 100%, especially at the beginning of each test. Then, I\nincreased the timeout for all cases tests to 10 seconds and ran the\ntests again 100 times. No timeouts occurred. Lastly, I created a VM\ninstance in GCloud with the same configuration as the one running in the\nCI, specifically the n2-standard-8 (8 vCPUs, 32 GB Memory) machine type.\nI ran the tests again 100 times. In 1 of the runs, a random test was\ntimeouted. The machine on the CI cannot handle the cases tests. I\nbelieve the work in this PR is a step in the right direction either way,\nand we will benefit from it. I also believe increasing the timeout is a\ngood decision as we need it based on the experiments and the research we\nhave done in the last months.\n\nCPU usage on n2-standard-4\n\n\nhttps://github.com/user-attachments/assets/36b035df-310f-4906-98ba-688d57b97c7e\n\nCPU usage on n2-standard-8\n\n\nhttps://github.com/user-attachments/assets/3b1b6351-d48e-41da-a413-a56e52b54b82\n\n## RLT eslint rules\nI enabled the [RLT eslint\nrules](https://testing-library.com/docs/ecosystem-eslint-plugin-testing-library/)\nfor Cases and resolved any eslint errors. The process revealed small\nbugs in the tests, which I fixed them.\n\n## Testing utils\n\nI removed the `appMockRender` and any usage in favor of the new\n`renderWithTestingProviders` utility function and the improved\n`TestProviders` component. The `renderWithTestingProviders` follows the\nprincipals\n[suggested](https://github.com/testing-library/react-testing-library/issues/780#issuecomment-689053441)\nby the RLT team. Specifically:\n- The `renderWithTestingProviders` is a wrapper of the `render` function\nof the RTL library.\n- The `renderWithTestingProviders` does not create the services or any\ncomponent inside it.\n- The `renderWithTestingProviders` cannot be used in `beforeEach`\nfunctions. It should be called separately on each test.\n- The `renderWithTestingProviders` accepts props to override the default\nmocks.\n- The `renderWithTestingProviders` passes the `TestProviders` in the\n`wrapper` argument of the RLT `render` function.\n- The `TestProviders` component initializes and memoizes all services\nand dependencies. It accepts props to override the default mocks.\n- Mock overrides (like core services) should be created and passed to\n`renderWithTestingProviders` on each test, even if it means duplication.\nWe favor test isolation.\n\n### Checklist\n\nCheck the PR satisfies the following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n\n## Issues\n<details>\n<summary>List</summary>\n\nFixes: https://github.com/elastic/kibana/issues/207712\nFixes: https://github.com/elastic/kibana/issues/192739\nFixes: https://github.com/elastic/kibana/issues/174682\nFixes: https://github.com/elastic/kibana/issues/206366\nFixes: https://github.com/elastic/kibana/issues/207427\nFixes: https://github.com/elastic/kibana/issues/175239\nFixes: https://github.com/elastic/kibana/issues/177334\nFixes: https://github.com/elastic/kibana/issues/208443\nFixes: https://github.com/elastic/kibana/issues/187526\nFixes: https://github.com/elastic/kibana/issues/208310\nFixes: https://github.com/elastic/kibana/issues/192640\nFixes: https://github.com/elastic/kibana/issues/207077\nFixes: https://github.com/elastic/kibana/issues/197304\nFixes: https://github.com/elastic/kibana/issues/207249\nFixes: https://github.com/elastic/kibana/issues/202761\nFixes: https://github.com/elastic/kibana/issues/202115\nFixes: https://github.com/elastic/kibana/issues/193026\nFixes: https://github.com/elastic/kibana/issues/177304\nFixes: https://github.com/elastic/kibana/issues/208415\nFixes: https://github.com/elastic/kibana/issues/174661\nFixes: https://github.com/elastic/kibana/issues/201611\nFixes: https://github.com/elastic/kibana/issues/182364\nFixes: https://github.com/elastic/kibana/issues/175841\nFixes: https://github.com/elastic/kibana/issues/207907\nFixes: https://github.com/elastic/kibana/issues/171177\nFixes: https://github.com/elastic/kibana/issues/196628\nFixes: https://github.com/elastic/kibana/issues/194703\nFixes: https://github.com/elastic/kibana/issues/207241\nFixes: https://github.com/elastic/kibana/issues/206056\nFixes: https://github.com/elastic/kibana/issues/207328\nFixes: https://github.com/elastic/kibana/issues/205953\nFixes: https://github.com/elastic/kibana/issues/176524\nFixes: https://github.com/elastic/kibana/issues/176335\nFixes: https://github.com/elastic/kibana/issues/207404\nFixes: https://github.com/elastic/kibana/issues/207384\nFixes: https://github.com/elastic/kibana/issues/208380\nFixes: https://github.com/elastic/kibana/issues/207248\nFixes: https://github.com/elastic/kibana/issues/207444\nFixes: https://github.com/elastic/kibana/issues/175240\nFixes: https://github.com/elastic/kibana/issues/178001\n\n</details>\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"1054799f9c1139c52d74c9c588f60177182919c7"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> |
||
---|---|---|
.. | ||
hooks | ||
pipeline-resource-definitions | ||
pipeline-utils | ||
pipelines | ||
scripts | ||
.mocharc.json | ||
.npmrc | ||
disabled_jest_configs.json | ||
ftr_base_serverless_configs.yml | ||
ftr_configs_manifests.json | ||
ftr_oblt_serverless_configs.yml | ||
ftr_oblt_stateful_configs.yml | ||
ftr_platform_stateful_configs.yml | ||
ftr_search_serverless_configs.yml | ||
ftr_search_stateful_configs.yml | ||
ftr_security_serverless_configs.yml | ||
ftr_security_stateful_configs.yml | ||
package-lock.json | ||
package.json | ||
pull_requests.json | ||
README.md | ||
tsconfig.json | ||
tsconfig.test.json |
Kibana / Buildkite
Directory Structure
hooks
- special directory used by Buildkite agents for hookspipeline-utils
- Shared TypeScript utils for use in pipeline scriptspipelines
- contains pipeline definitionsscripts/common
- scripts that getsource
d by other scripts to set environment variables or import shared functionsscripts/lifecycle
- general scripts for tasks that run before or after individual steps or the entire buildscripts/steps
- scripts that define something that will run for a step defined in a pipelinescripts/*
- all other scripts are building blocks that make up the tasks in pipelines. They may be run by other scripts, but should not besource
d