* [ftr] automatically determine config run order
* split lens config into two groups
* support ftr configs always running against CI
* Split detection_engine_api_integration rule exception list tests
* Add configs from previous commit
* [ftr] remove testMetadata and maintain a unique lifecycle instance per run
* Revert "[ftr] remove testMetadata and maintain a unique lifecycle instance per run"
This reverts commit d2b4fdb824.
* Split alerting_api_integration/security_and_spaces tests
* Add groups to yaml
* Revert "Revert "[ftr] remove testMetadata and maintain a unique lifecycle instance per run""
This reverts commit 56232eea68.
* stop ES more forcefully and fix timeout
* only cleanup lifecycle phases when the cleanup is totally complete
* only use kill when cleaning up an esTestInstance
* fix broken import
* fix runOptions.alwaysUseSource implementation
* fix config access
* fix x-pack/ccs config
* fix ml import file paths
* update kibana build id
* revert array.concat() change
* fix baseConfig usage
* fix pie chart data
* split up maps tests
* pull in all of group5 so that es archives are loaded correctly
* add to ftr configs.yml
* fix pie chart data without breaking legacy version
* fix more pie_chart stuff in new vis lib
* restore normal PR tasks
* bump kibana-buildkite-library
* remove ciGroup validation
* remove the script which is no longer called from checks.sh
* [CI] Auto-commit changed files from 'yarn kbn run build -i @kbn/pm'
* adapt flaky test runner scripts to handle ftrConfig paths
* fix types in alerting_api_integration
* improve flaky config parsing and use non-local var name for passing explicit configs to ftr_configs.sh
* Split xpack dashboard tests
* Add configs
* [flaky] remove key from ftr-config steps
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* restore cypress builds
* remove ciGroups from FTR config files
* fixup some docs
* add temporary script to hunt for FTR config files
* use config.base.js naming for clarity
* use script to power ftr_configs.yml
* remove usage of removed x-pack/scripts/functional_tests
* fix test names in dashboard snapshots
* bump kibana-buildkite-library
* Try retrying only failed configs
* be a little quieter about trying to get testStats from configs with testRunners defined
* Remove test code
* bump kibana-buildkite-library
* update es_snapshot and on_merge jobs too
* track duration and exit code for each config and print it at the end of the script
* store results in order, rather than by key, in case there are duplicates in $config
* bash is hard
* fix env source and use +e rather than disabling e for whole file
* bash sucks
* print config summary in jest jobs too
* define results in jest_parallel.sh
* simplify config summary print, format times a little better
* fix reference to unbound time variable, use better variable name
* skip the newline between each result
* finish with the nitpicking
* sync changes with ftr_configs.sh
* refuse to execute config files which aren't listed in the .buildkite/ftr_configs.yml
* fix config.edge.js base config import paths
* fix some readmes
* resolve paths from ftr_configs manifest
* fix readConfigFile tests
* just allow __fixtures__ configs
* list a few more cypress config files
* install the main branch of kibana-buildkite-library
* split up lens group1
* move ml data_visualizer tests to their own config
* fix import paths
* fix more imports
* install specific commit of buildkite-pipeline-library
* sort configs in ftr_configs.yml
* bump kibana-buildkite-library
* remove temporary script
* fix env var for limiting config types
* Update docs/developer/contributing/development-functional-tests.asciidoc
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
* produce a JUnit report for saved objects field count
* apply standard concurrency limits from flaky test runner
* support customizing FTR concurrency via the env
Co-authored-by: Brian Seeders <brian.seeders@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
* feat(12976): node version validation at runtime.
* refact(12976): move the code into a static utilities class.
* test(12976): added first test case using jest.
* test(12976): added test cases for node_version.
* feat(12976): create setup env node to bootstrap babel, ts-node and node version validator.
* refact(12976): migrated node version code from es6 to es5.
* feat(12976): node version validation at runtime.
* refact(12976): move the code into a static utilities class.
* test(12976): added first test case using jest.
* test(12976): added test cases for node_version.
* feat(12976): create setup env node to bootstrap babel, ts-node and node version validator.
* refact(12976): migrated node version code from es6 to es5.
* fix(12976): remove one level from ts node register cache directory link.
* chore(12976): added caret to semver dependecy in order to support minor versions.
* refact(12976): small change from named import to default import on node version validator.
* refact(12976): removed ts_node_register and add the code to babel_register.
* feat(12976): split eslint config in order to properly support files built to run before and after node version validator. refact(12976): convert script files to es5 code. refact(12976): delete inline eslint configs from node version check related files.
* refact(12976): remove ts node register file.
* refact(12976): completely port setup_node_env to es5.
* refact(12976): remove babel_register invokation from external dependencies in scripts.
* refact(12976): move node_version code directly into node_version_validator inside setup_node_env folder.
* refact(12976): only node version validator for kbn script.
Restructure testing with kbn-test package
- Run with multiple configs, move cli options to config
- Package-ify kbn-test
- Eventually we'll have functional_test_runner live in a package
of its own, and then this kbn-test will use that as a dependency,
probably still as a devDependency.
- Implement functional_tests_server
- Collapse single and multiple config apis into one command
Use kbn-es
Replace es_test_cluster + es_test_config with kbn/test utils
Implement new createEsTestCluster
Improve scripts, jsdocs, cli top-level tools
Lift error handling to the top level
* [babel] create babel-preset and babel-register modules
* [babel-preset] add comments to babel-preset-env.target choices
* [babel-preset] use more normal BUILT_WITH_BABEL signal
* [babel-register] extended comment about filtering ./src in dist
* [babel-preset] stage of class properties has changed, sorta
* As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces.
Differences:
- Streaming implementation allows for much larger archives
- CLI for creating and using archives
- Configurable archive location
- Stores the data in gzipped files (better for source control, searching, large archives)
- Automatically identifies and upgrades Kibana config documents
Methods:
- `#load(name)`: import an archive
- `#loadIfNeeded(name)`: import an archive, but skip the documents what belong to any existing index
- `#unload(name)`: delete the indexes stored in an archive
CLI operations:
- `./bin/es_archiver save <name> [index patterns...]`: save the mapping and documents in one or more indexes that match the wild-card patterns into an the `<name>` archive
- `./bin/es_archiver load <name>`: load the mapping and documents from the `<name>` archive
* [functional_tests/common/nagivate] check for statusPage
* [es_archiver] move bins into new scripts dir
* [functional_tests/apps/context] use esArchiver
* [esArchiver] general improvements after showing to a few folks
- remove auto-upgrading config doc logic (until we have better access to kibana version info)
- export unload command
- remove preemptive checks in favor of reacting to errors
- use type "doc" vs "hit" for doc records (consistency)
- wrote a bunch of pending tests to think though and plan
* [esArchiver] make log a stream that writes to itself
* [esArchiver] fill in stats and archive format tests
* [esArchiver] splitup action logic
* [esArchiver/cli] fix cli --help output and comment
* [esArchiver] remove type-based param coercion
* [esArchiver/log] use strings for log levels
* [esArchvier] remove unused var
* [esArchiver/indexDocRecordsStream] add tests
* [esArchive] fill in remaining tests
* [esArchiver] fix dem tests
* [eslint] remove unused vars
* [esArchiver/loadIfNeeded] fix call to load()
* [esArchiver] remove loadDumpData helpers
