github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-23 17:28:26 -04:00
Code Issues Projects Releases Packages Wiki Activity

[ftr] fix scripts/functional_tests to respect cli flags (#155734)

Browse source 
## Summary

This
[failure](https://buildkite.com/elastic/kibana-on-merge/builds/29091#01879988-ecd2-4e4f-bfb4-108939f145a1)
clearly shows that `--bail` flag is ignored when passed to
`scripts/functional_tests.js` script, and since
`scripts/functional_tests.js --help` list these flags I think we need to
fix it:

```
    --include-tag        Tags that suites must include to be run, can be included multiple times
    --exclude-tag        Tags that suites must NOT include to be run, can be included multiple times
    --include            Files that must included to be run, can be included multiple times
    --exclude            Files that must NOT be included to be run, can be included multiple times
    --grep               Pattern to select which tests to run
    --bail               Stop the test run at the first failure
    --dry-run            Report tests without executing them
    --updateBaselines    Replace baseline screenshots with whatever is generated from the test
    --updateSnapshots    Replace inline and file snapshots with whatever is generated from the test
```

I was able to reproduce it locally:
1. Break
[test/functional/apps/console/_console.ts](test/functional/apps/console/_console.ts)
by adding `expect(1).to.be(2);` in the first `it` function
2. Run `node scripts/functional_tests.js --bail --config
test/functional/apps/console/config.ts`
Actual: Tests continue to run after failure
Expected: Stop tests after first failure

It turned out `scripts/functional_test_runner.js` respects the flags so
I just copied the logic from
[packages/kbn-test/src/functional_test_runner/cli.ts](https://github.com/elastic/kibana/blob/main/packages/kbn-test/src/functional_test_runner/cli.ts#L41-L63)

Let me know if you think we need to add jest tests.


Tested:

```
node scripts/functional_tests.js --bail --config test/functional/apps/console/config.ts --grep "multiple requests output"
```
This commit is contained in:
Dzmitry Lemechko 2023-04-26 18:33:54 +02:00 • committed by GitHub
parent 947b75741c
commit 4211e03a5f
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
2 changed files with 26 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

8
packages/kbn-test/src/functional_tests/run_tests/flags.ts
View file

@ -78,12 +78,12 @@ export function parseFlags(flags: FlagsReader) {
installDir: flags.path('kibana-install-dir'),
grep: flags.string('grep'),
suiteTags: {
include: flags.arrayOfStrings('include-tag'),
exclude: flags.arrayOfStrings('exclude-tag'),
include: flags.arrayOfStrings('include-tag') ?? [],
exclude: flags.arrayOfStrings('exclude-tag') ?? [],
},
suiteFilters: {
include: flags.arrayOfPaths('include'),
exclude: flags.arrayOfPaths('exclude'),
include: flags.arrayOfPaths('include') ?? [],
exclude: flags.arrayOfPaths('exclude') ?? [],
},
};
}

23
packages/kbn-test/src/functional_tests/run_tests/run_tests.ts
View file

@ -36,6 +36,27 @@ export async function runTests(log: ToolingLog, options: RunTestsOptions) {
log.warning('❗️❗️❗️');
}
const settingOverrides = {
mochaOpts: {
bail: options.bail,
dryRun: options.dryRun,
grep: options.grep,
},
kbnTestServer: {
installDir: options.installDir,
},
suiteFiles: {
include: options.suiteFilters.include,
exclude: options.suiteFilters.exclude,
},
suiteTags: {
include: options.suiteTags.include,
exclude: options.suiteTags.exclude,
},
updateBaselines: options.updateBaselines,
updateSnapshots: options.updateSnapshots,
};
for (const [i, path] of options.configs.entries()) {
await log.indent(0, async () => {
if (options.configs.length > 1) {
@ -43,7 +64,7 @@ export async function runTests(log: ToolingLog, options: RunTestsOptions) {
log.write(`--- [${progress}] Running ${Path.relative(REPO_ROOT, path)}`);
}
const config = await readConfigFile(log, options.esVersion, path);
const config = await readConfigFile(log, options.esVersion, path, settingOverrides);
const hasTests = await checkForEnabledTestsInFtrConfig({
config,