[8.11] [docs] Add comment about ftr test environment differences (#169596) (#169665)

# Backport

This will backport the following commits from `main` to `8.11`:
- [[docs] Add comment about ftr test environment differences
(#169596)](https://github.com/elastic/kibana/pull/169596)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Christiane (Tina)
Heiligers","email":"christiane.heiligers@elastic.co"},"sourceCommit":{"committedDate":"2023-10-24T14:46:03Z","message":"[docs]
Add comment about ftr test environment differences (#169596)\n\nSkipped
tests are release blockers and fixing flaky tests isn't always\r\neasy,
especially if we assume that if a test passes locally, it will\r\nalso
pass in CI.\r\n\r\nWe need to keep the differences between our local
environments and CI in\r\nmind and make appropriate changes that take
these differences into\r\naccount but to do that, we first need to know
what these differences\r\nare!\r\n\r\nThis PR adds a high level
description of a couple of key differences.\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [X] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[x]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or
tutorials\r\n\r\n---------\r\n\r\nCo-authored-by: Alejandro Fernández
Haro
<afharo@gmail.com>","sha":"456d980e771846ee82ae246c7fca204d4a4c9f50","branchLabelMapping":{"^v8.12.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:all-open","v8.12.0"],"number":169596,"url":"https://github.com/elastic/kibana/pull/169596","mergeCommit":{"message":"[docs]
Add comment about ftr test environment differences (#169596)\n\nSkipped
tests are release blockers and fixing flaky tests isn't always\r\neasy,
especially if we assume that if a test passes locally, it will\r\nalso
pass in CI.\r\n\r\nWe need to keep the differences between our local
environments and CI in\r\nmind and make appropriate changes that take
these differences into\r\naccount but to do that, we first need to know
what these differences\r\nare!\r\n\r\nThis PR adds a high level
description of a couple of key differences.\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [X] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[x]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or
tutorials\r\n\r\n---------\r\n\r\nCo-authored-by: Alejandro Fernández
Haro
<afharo@gmail.com>","sha":"456d980e771846ee82ae246c7fca204d4a4c9f50"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.12.0","labelRegex":"^v8.12.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/169596","number":169596,"mergeCommit":{"message":"[docs]
Add comment about ftr test environment differences (#169596)\n\nSkipped
tests are release blockers and fixing flaky tests isn't always\r\neasy,
especially if we assume that if a test passes locally, it will\r\nalso
pass in CI.\r\n\r\nWe need to keep the differences between our local
environments and CI in\r\nmind and make appropriate changes that take
these differences into\r\naccount but to do that, we first need to know
what these differences\r\nare!\r\n\r\nThis PR adds a high level
description of a couple of key differences.\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [X] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[x]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or
tutorials\r\n\r\n---------\r\n\r\nCo-authored-by: Alejandro Fernández
Haro
<afharo@gmail.com>","sha":"456d980e771846ee82ae246c7fca204d4a4c9f50"}}]}]
BACKPORT-->

Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
This commit is contained in:
Kibana Machine 2023-10-24 11:08:46 -04:00 committed by GitHub
parent 53294881df
commit cc59e06bff
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23

View file

@ -10,6 +10,8 @@ Consistently writing functional tests that aren't flaky is impossible. There are
When you watch tests execute locally it can be tempting to think "after I click this button I can click this other button" but this assumes that the first button click will always execute its click handler immediately, or that the render of the second button will be done immediately. The reality is that user interfaces are super complex and these types of assumptions are by far the most common cause of flakiness in tests.
We also have to remember that we can't assume a test passing locally will mean it will pass in CI. The two environments are different. There is a lot we could mention, but at a high level, most functional tests are run on 4 core 16 GB machines, and these machines are virtualized, which means neighbors can cause modest but variable levels of performance. Additionally, end-to-end tests in CI are run against {kib} distributions, using default memory configurations, while we run the tests under the `--dev` flag locally with, most likely, a different memory configuration.
There are all sorts of things that can happen to delay a click handler, or react render, and we need to be prepared for those in our tests. We can do this using appropriate timeouts for specific actions, retry logic, and validating our assumptions with code.
## Become familiar with the retry/timing logic of each common service method