* [ftr] automatically determine config run order
* split lens config into two groups
* support ftr configs always running against CI
* Split detection_engine_api_integration rule exception list tests
* Add configs from previous commit
* [ftr] remove testMetadata and maintain a unique lifecycle instance per run
* Revert "[ftr] remove testMetadata and maintain a unique lifecycle instance per run"
This reverts commit d2b4fdb824.
* Split alerting_api_integration/security_and_spaces tests
* Add groups to yaml
* Revert "Revert "[ftr] remove testMetadata and maintain a unique lifecycle instance per run""
This reverts commit 56232eea68.
* stop ES more forcefully and fix timeout
* only cleanup lifecycle phases when the cleanup is totally complete
* only use kill when cleaning up an esTestInstance
* fix broken import
* fix runOptions.alwaysUseSource implementation
* fix config access
* fix x-pack/ccs config
* fix ml import file paths
* update kibana build id
* revert array.concat() change
* fix baseConfig usage
* fix pie chart data
* split up maps tests
* pull in all of group5 so that es archives are loaded correctly
* add to ftr configs.yml
* fix pie chart data without breaking legacy version
* fix more pie_chart stuff in new vis lib
* restore normal PR tasks
* bump kibana-buildkite-library
* remove ciGroup validation
* remove the script which is no longer called from checks.sh
* [CI] Auto-commit changed files from 'yarn kbn run build -i @kbn/pm'
* adapt flaky test runner scripts to handle ftrConfig paths
* fix types in alerting_api_integration
* improve flaky config parsing and use non-local var name for passing explicit configs to ftr_configs.sh
* Split xpack dashboard tests
* Add configs
* [flaky] remove key from ftr-config steps
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* restore cypress builds
* remove ciGroups from FTR config files
* fixup some docs
* add temporary script to hunt for FTR config files
* use config.base.js naming for clarity
* use script to power ftr_configs.yml
* remove usage of removed x-pack/scripts/functional_tests
* fix test names in dashboard snapshots
* bump kibana-buildkite-library
* Try retrying only failed configs
* be a little quieter about trying to get testStats from configs with testRunners defined
* Remove test code
* bump kibana-buildkite-library
* update es_snapshot and on_merge jobs too
* track duration and exit code for each config and print it at the end of the script
* store results in order, rather than by key, in case there are duplicates in $config
* bash is hard
* fix env source and use +e rather than disabling e for whole file
* bash sucks
* print config summary in jest jobs too
* define results in jest_parallel.sh
* simplify config summary print, format times a little better
* fix reference to unbound time variable, use better variable name
* skip the newline between each result
* finish with the nitpicking
* sync changes with ftr_configs.sh
* refuse to execute config files which aren't listed in the .buildkite/ftr_configs.yml
* fix config.edge.js base config import paths
* fix some readmes
* resolve paths from ftr_configs manifest
* fix readConfigFile tests
* just allow __fixtures__ configs
* list a few more cypress config files
* install the main branch of kibana-buildkite-library
* split up lens group1
* move ml data_visualizer tests to their own config
* fix import paths
* fix more imports
* install specific commit of buildkite-pipeline-library
* sort configs in ftr_configs.yml
* bump kibana-buildkite-library
* remove temporary script
* fix env var for limiting config types
* Update docs/developer/contributing/development-functional-tests.asciidoc
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
* produce a JUnit report for saved objects field count
* apply standard concurrency limits from flaky test runner
* support customizing FTR concurrency via the env
Co-authored-by: Brian Seeders <brian.seeders@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
* Update/refactor some cypress documentation
* Fixes some whitespace/grammar/typos
* Condenses the explanation/instructions for the different modes of
execution
* Condense Artifacts section
This is a big sprawling file; trying to cut down on the noise.
* Move test-running section to top of README
This is going to be what 90% of readers are looking for, methinks.
* Adds Security Solution's cypress suite to x-pack testing README
* Fix broken link
This file was moved as part of #64368.
* Remove broken link
This file was deleted in #67138.
* Apply suggestions from code review
Co-authored-by: Devin W. Hurley <snowmiser111@gmail.com>
* Fix typo
Co-authored-by: Devin W. Hurley <snowmiser111@gmail.com>
Over the last few years we've increased the number of test runners.
Entry points by test type have become too lage of a category for unique
names, so this moves top level test scripts under yarn and replaces test
types with the runner name.
e.g. `yarn test:browser` -> `yarn test:karma`
Closes#41133
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* docs: Add test file location -> test runner table
* Use scripts/mocha instead of if yarn test:mocha and clarify arguments
* docs: add some examples of running tests suites with grep patterns
* Split out x-pack testing docs, add test runner column
* Split out x-pack testing docs, add test runner column #2
* Use node scripts for jest and split functional runner/server
* progress
* progress
* cleanup and elastic configs
* make upgrades to support adding aditional users, with
* use defaultDeep to ensure settings pass correctly
* move needed configs to start servers into kbn_server (except x-pack plugin paths and users)
* move xpack config to an export
* add more time
* diff rollbacks
* roll back prettier diff
* revert setupUsers signature
* remove more bluebird
* update bluebird for fixes with jest compatability
* fix ts errors
* dont allow jest to keep going making errors confising
* Separates configs for jest integration core/x-pack.
* Pass nested kbn config parameters.
* Adds example x-pack integration test using live es.
* Cloud detectors should be configurable for tests.
* Cloud detectors should use native promises only.
* No erroneous comments...
* Util is only for promisify, duh!
* New tests should have docuementation to help those looking to utilize them.
* Doc section headings should be consistent with each other.
* With git there is no need to commit commented code.
* Reporting test readme
* Use full urls
* more full paths
* Don't use link to session folder, it's not in repo.
* updates
* Consolidate all reporting information into the readme and link from main x-pack readme.
* be consistent with Note: styling
* Add windows steps for downloading the correct packages.
Restructure testing with kbn-test package
- Run with multiple configs, move cli options to config
- Package-ify kbn-test
- Eventually we'll have functional_test_runner live in a package
of its own, and then this kbn-test will use that as a dependency,
probably still as a devDependency.
- Implement functional_tests_server
- Collapse single and multiple config apis into one command
Use kbn-es
Replace es_test_cluster + es_test_config with kbn/test utils
Implement new createEsTestCluster
Improve scripts, jsdocs, cli top-level tools
Lift error handling to the top level
