[8.x] Deployment agnostic tests - add support for env skip tags (#208154) (#208197)

# Backport

This will backport the following commits from `main` to `8.x`:
- [Deployment agnostic tests - add support for env skip tags
(#208154)](https://github.com/elastic/kibana/pull/208154)

<!--- Backport version: 9.6.4 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Robert
Oskamp","email":"robert.oskamp@elastic.co"},"sourceCommit":{"committedDate":"2025-01-24T11:27:56Z","message":"Deployment
agnostic tests - add support for env skip tags (#208154)\n\n##
Summary\r\n\r\nThis PR adds support for two new suite tags in FTR
deployment agnostic\r\ntests:\r\n* `skipStateful` to remove the suite
from all stateful test runs (local\r\n+ ECH)\r\n* `skipServerless` to
remove the suite from all serverless test runs\r\n(local +
MKI)","sha":"f28c293973b59d02aa484fd2cccf6dce840f5887","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","v8.18.0"],"title":"Deployment
agnostic tests - add support for env skip
tags","number":208154,"url":"https://github.com/elastic/kibana/pull/208154","mergeCommit":{"message":"Deployment
agnostic tests - add support for env skip tags (#208154)\n\n##
Summary\r\n\r\nThis PR adds support for two new suite tags in FTR
deployment agnostic\r\ntests:\r\n* `skipStateful` to remove the suite
from all stateful test runs (local\r\n+ ECH)\r\n* `skipServerless` to
remove the suite from all serverless test runs\r\n(local +
MKI)","sha":"f28c293973b59d02aa484fd2cccf6dce840f5887"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/208154","number":208154,"mergeCommit":{"message":"Deployment
agnostic tests - add support for env skip tags (#208154)\n\n##
Summary\r\n\r\nThis PR adds support for two new suite tags in FTR
deployment agnostic\r\ntests:\r\n* `skipStateful` to remove the suite
from all stateful test runs (local\r\n+ ECH)\r\n* `skipServerless` to
remove the suite from all serverless test runs\r\n(local +
MKI)","sha":"f28c293973b59d02aa484fd2cccf6dce840f5887"}},{"branch":"8.x","label":"v8.18.0","branchLabelMappingKey":"^v8.18.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

x-pack/test/api_integration/deployment_agnostic/README.md

@@ -246,10 +246,28 @@ node scripts/functional_test_runner --config x-pack/test/api_integration/deploym
 ```
 
 ## Tagging and Skipping the Tests
-Since deployment-agnostic tests are designed to run both locally and on MKI/Cloud, we believe no extra tagging is required. If a test is not working on MKI/Cloud or both, there is most likely an issue with the FTR service or the configuration file it uses.
+Since deployment-agnostic tests are designed to run both locally and on MKI/Cloud, we believe no extra tagging is required in general (read below for exceptions). If a test is not working on MKI/Cloud or both, there is most likely an issue with the FTR service or the configuration file it uses.
 
 When a test fails on CI, automation will apply `.skip` to the top-level describe block. This means the test will be skipped in **both serverless and stateful environments**. If a test is unstable in a specific environment only, it is probably a sign that the test is not truly deployment-agnostic.
 
+### Excluding a suite from test environments
+As pointed out above, deployment agnostic tests should be designed to run in stateful and serverless, locally and in cloud (ECH, MKI). However, there are situations where a test suite should only run on a subset of these environments. **This should be an exception.**
+
+Here are the supported suite labels to control execution in test environments:
+* `skipStateful` - this will exclude the suite from **all stateful test runs, local and ECH**
+* `skipCloud` - this will exclude the suite from **stateful cloud / ECH test runs**
+* `skipServerless` - this will exclude the suite from **all serverless test runs, local and MKI**
+* `skipMKI` - this will exclude the suite from **serverless cloud / MKI test runs**
+
+Note that tags can not be applied to an arrow function suite like `describe('test suite', () => {`. Here's an example of how to apply a suite tag:
+```ts
+describe('test suite', function () {
+  // add a comment to explain why this suite is excluded from that test environment
+  this.tags(['skipMKI']);
+  [...]
+});
+```
+
 ## Migrating existing tests
 If your tests align with the outlined criteria and requirements, you can migrate them to deployment-agnostic by following these steps:
 

x-pack/test/api_integration/deployment_agnostic/default_configs/serverless.config.base.ts

@@ -130,7 +130,10 @@ export function createServerlessTestConfig<T extends DeploymentAgnosticCommonSer
       },
       testFiles: options.testFiles,
       junit: options.junit,
-      suiteTags: options.suiteTags,
+      suiteTags: {
+        include: options.suiteTags?.include,
+        exclude: [...(options.suiteTags?.exclude || []), 'skipServerless'],
+      },
     };
   };
 }

x-pack/test/api_integration/deployment_agnostic/default_configs/stateful.config.base.ts

@@ -101,7 +101,10 @@ export function createStatefulTestConfig<T extends DeploymentAgnosticCommonServi
       // services can be customized, but must extend DeploymentAgnosticCommonServices
       services: options.services || services,
       junit: options.junit,
-      suiteTags: options.suiteTags,
+      suiteTags: {
+        include: options.suiteTags?.include,
+        exclude: [...(options.suiteTags?.exclude || []), 'skipStateful'],
+      },
 
       esTestCluster: {
         ...xPackAPITestsConfig.get('esTestCluster'),