* [ftr] add first-class support for playwrite journeys
* [CI] Auto-commit changed files from 'node scripts/generate codeowners'
* fix jest test
* remove ability to customize kibana server args, if we need it we can add it back
* remove dev dir that doesn't exist
* fix typo
* prevent duplicated array converstion logic by sharing flag reader
* remove destructuring of option
* fix scalability config and config_path import
* fix start_servers args and tests
* include simple readme
* fix jest tests and support build re-use when changes are just to jest tests
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* [ftr] automatically determine config run order
* split lens config into two groups
* support ftr configs always running against CI
* Split detection_engine_api_integration rule exception list tests
* Add configs from previous commit
* [ftr] remove testMetadata and maintain a unique lifecycle instance per run
* Revert "[ftr] remove testMetadata and maintain a unique lifecycle instance per run"
This reverts commit d2b4fdb824.
* Split alerting_api_integration/security_and_spaces tests
* Add groups to yaml
* Revert "Revert "[ftr] remove testMetadata and maintain a unique lifecycle instance per run""
This reverts commit 56232eea68.
* stop ES more forcefully and fix timeout
* only cleanup lifecycle phases when the cleanup is totally complete
* only use kill when cleaning up an esTestInstance
* fix broken import
* fix runOptions.alwaysUseSource implementation
* fix config access
* fix x-pack/ccs config
* fix ml import file paths
* update kibana build id
* revert array.concat() change
* fix baseConfig usage
* fix pie chart data
* split up maps tests
* pull in all of group5 so that es archives are loaded correctly
* add to ftr configs.yml
* fix pie chart data without breaking legacy version
* fix more pie_chart stuff in new vis lib
* restore normal PR tasks
* bump kibana-buildkite-library
* remove ciGroup validation
* remove the script which is no longer called from checks.sh
* [CI] Auto-commit changed files from 'yarn kbn run build -i @kbn/pm'
* adapt flaky test runner scripts to handle ftrConfig paths
* fix types in alerting_api_integration
* improve flaky config parsing and use non-local var name for passing explicit configs to ftr_configs.sh
* Split xpack dashboard tests
* Add configs
* [flaky] remove key from ftr-config steps
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* restore cypress builds
* remove ciGroups from FTR config files
* fixup some docs
* add temporary script to hunt for FTR config files
* use config.base.js naming for clarity
* use script to power ftr_configs.yml
* remove usage of removed x-pack/scripts/functional_tests
* fix test names in dashboard snapshots
* bump kibana-buildkite-library
* Try retrying only failed configs
* be a little quieter about trying to get testStats from configs with testRunners defined
* Remove test code
* bump kibana-buildkite-library
* update es_snapshot and on_merge jobs too
* track duration and exit code for each config and print it at the end of the script
* store results in order, rather than by key, in case there are duplicates in $config
* bash is hard
* fix env source and use +e rather than disabling e for whole file
* bash sucks
* print config summary in jest jobs too
* define results in jest_parallel.sh
* simplify config summary print, format times a little better
* fix reference to unbound time variable, use better variable name
* skip the newline between each result
* finish with the nitpicking
* sync changes with ftr_configs.sh
* refuse to execute config files which aren't listed in the .buildkite/ftr_configs.yml
* fix config.edge.js base config import paths
* fix some readmes
* resolve paths from ftr_configs manifest
* fix readConfigFile tests
* just allow __fixtures__ configs
* list a few more cypress config files
* install the main branch of kibana-buildkite-library
* split up lens group1
* move ml data_visualizer tests to their own config
* fix import paths
* fix more imports
* install specific commit of buildkite-pipeline-library
* sort configs in ftr_configs.yml
* bump kibana-buildkite-library
* remove temporary script
* fix env var for limiting config types
* Update docs/developer/contributing/development-functional-tests.asciidoc
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
* produce a JUnit report for saved objects field count
* apply standard concurrency limits from flaky test runner
* support customizing FTR concurrency via the env
Co-authored-by: Brian Seeders <brian.seeders@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
* added cypress setup for fleet
* updated readme
* fixed types
* fix unused
* starting up fleet server in cypress
* added more fleet tests
* added package update test
* updated readme
* trying jenkins run
* trying jenkins run
* trying jenkins run
* fix types, cypress config
* example of mocked requests
* added getBySel
* fix test after refactor
* removed duplication
* fix tests
* added to buildkite
* updated tests
* starting agent with docker
* trying to fix ip address
* ifconfig
* ip address
* ip address
* ip address
* type fix
* remove extra logging
* fixed test
* fix buildkite
* cleanup
* cleanup
* using docker for fleet_server
* fix
* fix
* trying to fix
* update config
* test
* test
* test
* test
* revert changes
* test
* test
* static ip on linux
* docker version
* try again
* fixed review comments
* fixed types
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
This PR include three key changes:
1. Run tasks as soon as they have been marked as running, rather than wait for the whole batch to me marked
2. Use a custom refresh setting of refresh: false where possible, in place of wait_for, in order to speed up Task Manager's internal workflow
3. Instrumentation of Task Manager exposing Activity / Inactivity metrics in Performance test runs
Restructure testing with kbn-test package
- Run with multiple configs, move cli options to config
- Package-ify kbn-test
- Eventually we'll have functional_test_runner live in a package
of its own, and then this kbn-test will use that as a dependency,
probably still as a devDependency.
- Implement functional_tests_server
- Collapse single and multiple config apis into one command
Use kbn-es
Replace es_test_cluster + es_test_config with kbn/test utils
Implement new createEsTestCluster
Improve scripts, jsdocs, cli top-level tools
Lift error handling to the top level
