mirror of
https://github.com/elastic/kibana.git
synced 2025-04-24 01:38:56 -04:00
# Backport This will backport the following commits from `main` to `8.9`: - [[Security Solution] [Flaky test] Fix double ftr test flakiness (#162990)](https://github.com/elastic/kibana/pull/162990) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Wafaa Nasr","email":"wafaa.nasr@elastic.co"},"sourceCommit":{"committedDate":"2023-08-09T14:40:55Z","message":"[Security Solution] [Flaky test] Fix double ftr test flakiness (#162990)\n\n## Summary\r\n\r\n- Addresses https://github.com/elastic/kibana/issues/155122 \r\n\r\n\r\n## Investigations\r\n\r\n\r\n *First attempt*\r\n\r\n> While investigating the three failing pipelines and examining the\r\nArtifacts, specifically target/test_failures/*.log, I discovered the\r\nroot cause to be\r\n> Did not get an expected 200 \"ok\" when waiting for a list item\r\n(waitForListItem) yet. Retrying until we get a 200 \"ok\". body:\r\n{\"message\":\"list_id: \\\"list_items.txt\\\" item of 1.0 does not\r\nexist\",\"status_code\":404}, status: 404\r\n> \r\n>\r\n\r\n> \r\n\r\n*Second attempt*\r\n\r\n\r\n> Upon incorporating additional logs to pinpoint the underlying issue,\r\nit was realized that the test will persist in its attempts until the\r\npreceding error is rectified, and it was determined that the\r\nlist_items.txt file contains the correct item once located.\r\n> \r\n> The problem seemed to be associated with incomplete indexing of all\r\nindices signalsOpen.hits.hits when the test verifies against the\r\nanticipated value.\r\n> \r\n> To investigate the matter, two iterations were conducted, totaling 490\r\nattempts. During these iterations, logs, and warnings were deliberately\r\ninserted to facilitate the tracing of the issue.\r\n> \r\n>\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2770#0189bb29-c258-4586-986a-2ac5e128eb52\r\n>\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2768#0189baeb-8f09-488e-936f-69e72d089787\r\n> \r\n> In this\r\n[Atifact](0189bb29
-c258-4586-986a-2ac5e128eb52_d98d35afa0abd1ad03e28a68882291c4.log?response-content-type=text%2Fplain&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQPCP3C7LXS7MSJGH%2F20230808%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230808T102039Z&X-Amz-Expires=600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEOH%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIQDYG9De6RYDlVNcO0JGqO6Qc4dpzCQG1s%2BgaXTSimWc9QIgfkZgXlnVcScJCO56vz2VQx1buVTNEDEaVYhYmDgupEgq%2BgMIiv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAAGgwwMzIzNzk3MDUzMDMiDOz2Gl2kfcuTHVaWyirOA4iHfkFBI7KBoy%2F97CuLMAxGLNIR8h4FtLCsY664lMhgDGd%2FBVi82zu%2B2Y5DAGEboOmb7C1q%2FX6CyesjDKP7sknXdfT7q8B4jBRK4QcVw2AAvC9%2BqMeN37eCfRXWT%2FrQAEdaWzxkJxk4BtFcATds3LJmmAlbYOT2L7vItmGzGs%2FnkRYevX%2Fq6hwxWqmOKNNot9hZ9sMJ3TZvGsA3045459BfyfObwIlzb13Pj6V7%2FMyTb87RjQazFiK%2Bb7qOXB%2B7ItaVAcpG7diXzPAif%2F%2FEiJedTHtBVh4LsMIKQtiASbcTh4X90kYtVMUcH%2BSG0B41WjWYPok0C3tHfQnGhGBIbSnqvMGmWgwGlhHNF1G7nIDYyWs32PQ3M3WmTGQuLIwp2hPzk4wFtgH%2FDQPWevRFRT%2FhzYJf05hyBT9BMcST9eeKTqEqs52PhCHDlCgo7K5bGjT86OO16MzEKdA77HRro8GqIh6mgTMyQ5gKpH38ueUWvQZnLix4T5HYw0EyJgRLRfiVIpKcH0jRobaFwKSfSYc%2FZ2FPHsAK31weedhy6tafwU7elNYiAaMRv4AQhB%2FlrGWYd6IjOrdgSFowlkd1m5%2FrYTY3k7%2F8D0vsxgjUETDX9semBjqlAWkLq%2FLaRV5KokMNXfT3lCVNJzpYeN1%2FdnInE%2BQn3Ub86kRgMOfTN2OgDHkuYSKsQ95tFHH49NfRn6rTw8tI2Tjs3pToqp65gCYh9MzUKTd2EWsgqdNn6y%2BbHRGFJxd7cacrX9Ghr4XyNme5aAeWAVYNXSqJ1KIbtWmRcwwhThAJHxWk5%2FoBwScZRUSYGWmkhaxnZXDjXg6LBuH7uZOJjf%2FLuFVdtQ%3D%3D&X-Amz-SignedHeaders=host&X-Amz-Signature=8a44c3a5e3b88b632097dc344ecec1ba6d6cb8e80e10d9058e96883e43d925b5)\r\n> \r\n>\r\n\r\n> \r\n> \r\n\r\n## Solution \r\n\r\nBy modifying the third parameter of the `await\r\nwaitForSignalsToBePresent(supertest, log, 1, [id]);` function to a value\r\nof 3 instead of the initial 1, we signify the number of alerts we need\r\nto await before conducting assertions. This change is made to\r\naccommodate the assertion of three alerts being generated, accounting\r\nfor the exclusion of a single 1.0 value.","sha":"f48d422a2e5114e899df5697a18de10918beb779","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","test-failure-flaky","Feature:Rule Exceptions","v8.9.0","Team:Detection Engine","v8.10.0","v8.11.0"],"number":162990,"url":"https://github.com/elastic/kibana/pull/162990","mergeCommit":{"message":"[Security Solution] [Flaky test] Fix double ftr test flakiness (#162990)\n\n## Summary\r\n\r\n- Addresses https://github.com/elastic/kibana/issues/155122 \r\n\r\n\r\n## Investigations\r\n\r\n\r\n *First attempt*\r\n\r\n> While investigating the three failing pipelines and examining the\r\nArtifacts, specifically target/test_failures/*.log, I discovered the\r\nroot cause to be\r\n> Did not get an expected 200 \"ok\" when waiting for a list item\r\n(waitForListItem) yet. Retrying until we get a 200 \"ok\". body:\r\n{\"message\":\"list_id: \\\"list_items.txt\\\" item of 1.0 does not\r\nexist\",\"status_code\":404}, status: 404\r\n> \r\n>\r\n\r\n> \r\n\r\n*Second attempt*\r\n\r\n\r\n> Upon incorporating additional logs to pinpoint the underlying issue,\r\nit was realized that the test will persist in its attempts until the\r\npreceding error is rectified, and it was determined that the\r\nlist_items.txt file contains the correct item once located.\r\n> \r\n> The problem seemed to be associated with incomplete indexing of all\r\nindices signalsOpen.hits.hits when the test verifies against the\r\nanticipated value.\r\n> \r\n> To investigate the matter, two iterations were conducted, totaling 490\r\nattempts. During these iterations, logs, and warnings were deliberately\r\ninserted to facilitate the tracing of the issue.\r\n> \r\n>\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2770#0189bb29-c258-4586-986a-2ac5e128eb52\r\n>\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2768#0189baeb-8f09-488e-936f-69e72d089787\r\n> \r\n> In this\r\n[Atifact](0189bb29
-c258-4586-986a-2ac5e128eb52_d98d35afa0abd1ad03e28a68882291c4.log?response-content-type=text%2Fplain&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQPCP3C7LXS7MSJGH%2F20230808%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230808T102039Z&X-Amz-Expires=600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEOH%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIQDYG9De6RYDlVNcO0JGqO6Qc4dpzCQG1s%2BgaXTSimWc9QIgfkZgXlnVcScJCO56vz2VQx1buVTNEDEaVYhYmDgupEgq%2BgMIiv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAAGgwwMzIzNzk3MDUzMDMiDOz2Gl2kfcuTHVaWyirOA4iHfkFBI7KBoy%2F97CuLMAxGLNIR8h4FtLCsY664lMhgDGd%2FBVi82zu%2B2Y5DAGEboOmb7C1q%2FX6CyesjDKP7sknXdfT7q8B4jBRK4QcVw2AAvC9%2BqMeN37eCfRXWT%2FrQAEdaWzxkJxk4BtFcATds3LJmmAlbYOT2L7vItmGzGs%2FnkRYevX%2Fq6hwxWqmOKNNot9hZ9sMJ3TZvGsA3045459BfyfObwIlzb13Pj6V7%2FMyTb87RjQazFiK%2Bb7qOXB%2B7ItaVAcpG7diXzPAif%2F%2FEiJedTHtBVh4LsMIKQtiASbcTh4X90kYtVMUcH%2BSG0B41WjWYPok0C3tHfQnGhGBIbSnqvMGmWgwGlhHNF1G7nIDYyWs32PQ3M3WmTGQuLIwp2hPzk4wFtgH%2FDQPWevRFRT%2FhzYJf05hyBT9BMcST9eeKTqEqs52PhCHDlCgo7K5bGjT86OO16MzEKdA77HRro8GqIh6mgTMyQ5gKpH38ueUWvQZnLix4T5HYw0EyJgRLRfiVIpKcH0jRobaFwKSfSYc%2FZ2FPHsAK31weedhy6tafwU7elNYiAaMRv4AQhB%2FlrGWYd6IjOrdgSFowlkd1m5%2FrYTY3k7%2F8D0vsxgjUETDX9semBjqlAWkLq%2FLaRV5KokMNXfT3lCVNJzpYeN1%2FdnInE%2BQn3Ub86kRgMOfTN2OgDHkuYSKsQ95tFHH49NfRn6rTw8tI2Tjs3pToqp65gCYh9MzUKTd2EWsgqdNn6y%2BbHRGFJxd7cacrX9Ghr4XyNme5aAeWAVYNXSqJ1KIbtWmRcwwhThAJHxWk5%2FoBwScZRUSYGWmkhaxnZXDjXg6LBuH7uZOJjf%2FLuFVdtQ%3D%3D&X-Amz-SignedHeaders=host&X-Amz-Signature=8a44c3a5e3b88b632097dc344ecec1ba6d6cb8e80e10d9058e96883e43d925b5)\r\n> \r\n>\r\n\r\n> \r\n> \r\n\r\n## Solution \r\n\r\nBy modifying the third parameter of the `await\r\nwaitForSignalsToBePresent(supertest, log, 1, [id]);` function to a value\r\nof 3 instead of the initial 1, we signify the number of alerts we need\r\nto await before conducting assertions. This change is made to\r\naccommodate the assertion of three alerts being generated, accounting\r\nfor the exclusion of a single 1.0 value.","sha":"f48d422a2e5114e899df5697a18de10918beb779"}},"sourceBranch":"main","suggestedTargetBranches":["8.9"],"targetPullRequestStates":[{"branch":"8.9","label":"v8.9.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/162990","number":162990,"mergeCommit":{"message":"[Security Solution] [Flaky test] Fix double ftr test flakiness (#162990)\n\n## Summary\r\n\r\n- Addresses https://github.com/elastic/kibana/issues/155122 \r\n\r\n\r\n## Investigations\r\n\r\n\r\n *First attempt*\r\n\r\n> While investigating the three failing pipelines and examining the\r\nArtifacts, specifically target/test_failures/*.log, I discovered the\r\nroot cause to be\r\n> Did not get an expected 200 \"ok\" when waiting for a list item\r\n(waitForListItem) yet. Retrying until we get a 200 \"ok\". body:\r\n{\"message\":\"list_id: \\\"list_items.txt\\\" item of 1.0 does not\r\nexist\",\"status_code\":404}, status: 404\r\n> \r\n>\r\n\r\n> \r\n\r\n*Second attempt*\r\n\r\n\r\n> Upon incorporating additional logs to pinpoint the underlying issue,\r\nit was realized that the test will persist in its attempts until the\r\npreceding error is rectified, and it was determined that the\r\nlist_items.txt file contains the correct item once located.\r\n> \r\n> The problem seemed to be associated with incomplete indexing of all\r\nindices signalsOpen.hits.hits when the test verifies against the\r\nanticipated value.\r\n> \r\n> To investigate the matter, two iterations were conducted, totaling 490\r\nattempts. During these iterations, logs, and warnings were deliberately\r\ninserted to facilitate the tracing of the issue.\r\n> \r\n>\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2770#0189bb29-c258-4586-986a-2ac5e128eb52\r\n>\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2768#0189baeb-8f09-488e-936f-69e72d089787\r\n> \r\n> In this\r\n[Atifact](0189bb29
-c258-4586-986a-2ac5e128eb52_d98d35afa0abd1ad03e28a68882291c4.log?response-content-type=text%2Fplain&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQPCP3C7LXS7MSJGH%2F20230808%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230808T102039Z&X-Amz-Expires=600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEOH%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIQDYG9De6RYDlVNcO0JGqO6Qc4dpzCQG1s%2BgaXTSimWc9QIgfkZgXlnVcScJCO56vz2VQx1buVTNEDEaVYhYmDgupEgq%2BgMIiv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAAGgwwMzIzNzk3MDUzMDMiDOz2Gl2kfcuTHVaWyirOA4iHfkFBI7KBoy%2F97CuLMAxGLNIR8h4FtLCsY664lMhgDGd%2FBVi82zu%2B2Y5DAGEboOmb7C1q%2FX6CyesjDKP7sknXdfT7q8B4jBRK4QcVw2AAvC9%2BqMeN37eCfRXWT%2FrQAEdaWzxkJxk4BtFcATds3LJmmAlbYOT2L7vItmGzGs%2FnkRYevX%2Fq6hwxWqmOKNNot9hZ9sMJ3TZvGsA3045459BfyfObwIlzb13Pj6V7%2FMyTb87RjQazFiK%2Bb7qOXB%2B7ItaVAcpG7diXzPAif%2F%2FEiJedTHtBVh4LsMIKQtiASbcTh4X90kYtVMUcH%2BSG0B41WjWYPok0C3tHfQnGhGBIbSnqvMGmWgwGlhHNF1G7nIDYyWs32PQ3M3WmTGQuLIwp2hPzk4wFtgH%2FDQPWevRFRT%2FhzYJf05hyBT9BMcST9eeKTqEqs52PhCHDlCgo7K5bGjT86OO16MzEKdA77HRro8GqIh6mgTMyQ5gKpH38ueUWvQZnLix4T5HYw0EyJgRLRfiVIpKcH0jRobaFwKSfSYc%2FZ2FPHsAK31weedhy6tafwU7elNYiAaMRv4AQhB%2FlrGWYd6IjOrdgSFowlkd1m5%2FrYTY3k7%2F8D0vsxgjUETDX9semBjqlAWkLq%2FLaRV5KokMNXfT3lCVNJzpYeN1%2FdnInE%2BQn3Ub86kRgMOfTN2OgDHkuYSKsQ95tFHH49NfRn6rTw8tI2Tjs3pToqp65gCYh9MzUKTd2EWsgqdNn6y%2BbHRGFJxd7cacrX9Ghr4XyNme5aAeWAVYNXSqJ1KIbtWmRcwwhThAJHxWk5%2FoBwScZRUSYGWmkhaxnZXDjXg6LBuH7uZOJjf%2FLuFVdtQ%3D%3D&X-Amz-SignedHeaders=host&X-Amz-Signature=8a44c3a5e3b88b632097dc344ecec1ba6d6cb8e80e10d9058e96883e43d925b5)\r\n> \r\n>\r\n\r\n> \r\n> \r\n\r\n## Solution \r\n\r\nBy modifying the third parameter of the `await\r\nwaitForSignalsToBePresent(supertest, log, 1, [id]);` function to a value\r\nof 3 instead of the initial 1, we signify the number of alerts we need\r\nto await before conducting assertions. This change is made to\r\naccommodate the assertion of three alerts being generated, accounting\r\nfor the exclusion of a single 1.0 value.","sha":"f48d422a2e5114e899df5697a18de10918beb779"}},{"branch":"8.11","label":"v8.11.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
This commit is contained in:
parent
5a890ddfae
commit
4c62619b31
1 changed files with 1 additions and 2 deletions
|
@ -489,7 +489,6 @@ export default ({ getService }: FtrProviderContext) => {
|
|||
expect(hits).to.eql([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('working against string values in the data set', () => {
|
||||
it('will return 3 results if we have a list that includes 1 double', async () => {
|
||||
await importFile(supertest, log, 'double', ['1.0'], 'list_items.txt');
|
||||
|
@ -508,7 +507,7 @@ export default ({ getService }: FtrProviderContext) => {
|
|||
],
|
||||
]);
|
||||
await waitForRuleSuccess({ supertest, log, id });
|
||||
await waitForSignalsToBePresent(supertest, log, 1, [id]);
|
||||
await waitForSignalsToBePresent(supertest, log, 3, [id]);
|
||||
const signalsOpen = await getSignalsById(supertest, log, id);
|
||||
const hits = signalsOpen.hits.hits.map((hit) => hit._source?.double).sort();
|
||||
expect(hits).to.eql(['1.1', '1.2', '1.3']);
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue