## Summary
This PR is the Step 1 to enable performance journeys run for serverless
projects.
The focus is to re-design journeys to be compatible both for stateful &
serverless Kibana.
I created `KibanaPage` class to have some shared UI actions across
different journeys.
`ProjectPage` extends `KibanaPage` and allows us to override actions,
that are different (or have different locators) in Kibana Project UI
(generic project at the moment)
`kibanaPage` is available in Step context and based on TEST_SERVERLESS
env var appropriate class instance is used.
```typescript
.step('Go to Discover Page', async ({ page, kbnUrl, kibanaPage }) => {
await page.goto(kbnUrl.get(`/app/discover`));
await kibanaPage.waitForHeader();
await page.waitForSelector('[data-test-subj="discoverDocTable"][data-render-complete="true"]');
await page.waitForSelector(subj('globalLoadingIndicator-hidden'));
})
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
While elastic/appex-qa is in process to take ownership over FTR related
code, from elastic/kibana-operations it will be good to keep track on
any changes happening and participate in the review process.
## Summary
This PR adds changes and instructions how to run existing performance
journeys against ESS instances.
As previously discussed with @pheyos , there are few manual steps to be
done before running the actual journey:
1. Create cloud deployment and re-configure it the way APM traces are
reported the monitoring cluster
2. Check out the branch matching deployment version (main ->
8.9.0-SNAPSHOT), create user with `superuser` role (we don't test
functional features, only performance metrics)
Then you can run the journey like a regular cloud test suite:
```
export TEST_KIBANA_URL=https://<username>:<password>@<kibana_url>
export TEST_ES_URL=https://<username>:<password>@j<elasticsearch_url>:<port>
export TEST_CLOUD=1
node scripts/functional_test_runner.js --config x-pack/performance/journeys/$YOUR_JOURNEY_NAME.ts
```
<img width="1357" alt="image"
src="https://user-images.githubusercontent.com/10977896/236499836-2738fecb-a5a9-4e58-a10b-915b962af037.png">
---------
Co-authored-by: Jon <jon@budzenski.me>
Enables APM instrumentation for Elasticsearch when running performance
journeys, for a more complete understanding of where time is being
spent:
<img width="1418" alt="CleanShot 2023-04-18 at 20 30 32@2x"
src="https://user-images.githubusercontent.com/352732/232870884-151aed23-347b-485e-8490-4b56b6beaa33.png">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Related to #151613
There might be cases when we need to add extra wait for Kibana plugin to
be ready before starting loading test data with `esArchiver` /
`kbnArchiver`.
Currently journey lifecycle looks like this:
- `onSetup` wrapped with mocha `before` hook
- in parallel
- `setupBrowserAndPage` (including EBT tracker setup)
- load ES data / Kibana saved objects
- `setupApm`
- steps execution (each step wrapped with mocha `it` function)
- `onTeardown` wrapped with mocha `after` hook
- `tearDownBrowserAndPage` (including closing EBT tracker)
- `teardownApm`
- load ES data / Kibana saved objects
beforeSteps hook purpose is to make sure Kibana/ES state is ready for
journey execution and not load test data, since it won't be unloaded
during `after` hook:
- `onSetup` wrapped with mocha `before` hook
- `setupBrowserAndPage` (including EBT tracker setup)
- run beforeSteps hook -> prepare Kibana/ES for data ingestion / steps
execution
- load ES data / Kibana saved objects
- `setupApm`
How to use:
```
export const journey = new Journey({
beforeSteps: async ({ kibanaServer, retry }) => {
retry.try(async () => {
const response = await kibanaServer.request({
path: '/internal/cloud_security_posture/status?check=init',
method: 'GET',
});
return response.status === 200;
});
},
esArchives: [...],
kbnArchives: [...],
})
.step({...})
.step({...})
```
## Summary
Currently journeys use pre-defined base FTR config and there is no way
to re-configure Kibana to be loaded with extras, e.g. Fleet plugin
configuration.
f443109eea/packages/kbn-journeys/journey/journey_ftr_config.ts (L32-L34)
Probably the easiest way to address it is to path custom FTR config
directly in the journey.
```
export const journey = new Journey({
ftrConfigPath: 'x-pack/test/cloud_security_posture_api/config.ts',
...
})
```
It can be also considered as a step towards using journeys as future
alternative to Webdriver functional tests.
## Summary
I noticed some noise in [Performance
dashboard](dd0473ac-826f-5621-9a10-25319700326e?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-24h%2Fh,to:now)))
and think it is better to disable Telemetry for journeys by default.
We use it to report performance events and this PR enables it in the
performance pipeline via env variable `PERFORMANCE_ENABLE_TELEMETRY`.
For other pipelines (PRs,
[performance-data-set-extraction](https://buildkite.com/elastic/kibana-performance-data-set-extraction))
running on regular workers or local troubleshooting there is no much
value to collect inconsistent values.
## Summary
This PR adds capability to run capacity testing for single apis #143066
Currently in main we have to 2 types of performance tests:
- single user performance journey that simulates single end-user
experience in browser
- scalability journey that uses APM traces from single user performance
journey to simulate multiple end-users experience
This new type of performance tests allow to better understand how each
single server api scale under the similar load.
How to run locally:
make sure to clone the latest main branch of
[elastic/kibana-load-testing](https://github.com/elastic/kibana-load-testing)
in Kibana repo run:
`node scripts/run_scalability.js --journey-path
x-pack/test/scalability/apis/api.core.capabilities.json`
How it works:
FTR is used to start Kibana/ES and run Gatling simulation with json file
as input. After run the latest report matching journey name is parsed to
get perf metrics and report using EBT to the Telemetry cluster.
How will it run after merge:
I plan to run pipeline every 3 hours on bare metal machine and report
metrics to Telemetry staging cluster.
<img width="2023" alt="image"
src="https://user-images.githubusercontent.com/10977896/208771628-f4f5dbcb-cb73-40c6-9aa1-4ec3fbf5285b.png">
APM traces are collected and reported to Kibana stats cluster:
<img width="1520" alt="image"
src="https://user-images.githubusercontent.com/10977896/208771323-4cca531a-eeea-4941-8b01-50b890f932b1.png">
What metrics are collected:
1. warmupAvgResponseTime - average response time during warmup phase
2. rpsAtWarmup - average requests per second during warmup phase
3. warmupDuration
4. responseTimeMetric (default: 85%) Gatling has response time
25/50/75/80/85/90/95/99 percentiles, as well as min/max values
5. threshold1ResponseTime (default 3000 ms)
6. rpsAtThreshold1 requests per second when `responseTimeMetric` first
reach threshold1ResponseTime
7. threshold2ResponseTime
8. rpsAtThreshold2 (default 9000 ms)
9. threshold3ResponseTime
10. rpsAtThreshold3 (default 15000 ms)
As long as we agree on metrics I will update indexer for telemetry.
Co-authored-by: Alejandro Fernández Haro <alejandro.haro@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Dearest Reviewers 👋
I've been working on this branch with @mistic and @tylersmalley and
we're really confident in these changes. Additionally, this changes code
in nearly every package in the repo so we don't plan to wait for reviews
to get in before merging this. If you'd like to have a concern
addressed, please feel free to leave a review, but assuming that nobody
raises a blocker in the next 24 hours we plan to merge this EOD pacific
tomorrow, 12/22.
We'll be paying close attention to any issues this causes after merging
and work on getting those fixed ASAP. 🚀
---
The operations team is not confident that we'll have the time to achieve
what we originally set out to accomplish by moving to Bazel with the
time and resources we have available. We have also bought ourselves some
headroom with improvements to babel-register, optimizer caching, and
typescript project structure.
In order to make sure we deliver packages as quickly as possible (many
teams really want them), with a usable and familiar developer
experience, this PR removes Bazel for building packages in favor of
using the same JIT transpilation we use for plugins.
Additionally, packages now use `kbn_references` (again, just copying the
dx from plugins to packages).
Because of the complex relationships between packages/plugins and in
order to prepare ourselves for automatic dependency detection tools we
plan to use in the future, this PR also introduces a "TS Project Linter"
which will validate that every tsconfig.json file meets a few
requirements:
1. the chain of base config files extended by each config includes
`tsconfig.base.json` and not `tsconfig.json`
1. the `include` config is used, and not `files`
2. the `exclude` config includes `target/**/*`
3. the `outDir` compiler option is specified as `target/types`
1. none of these compiler options are specified: `declaration`,
`declarationMap`, `emitDeclarationOnly`, `skipLibCheck`, `target`,
`paths`
4. all references to other packages/plugins use their pkg id, ie:
```js
// valid
{
"kbn_references": ["@kbn/core"]
}
// not valid
{
"kbn_references": [{ "path": "../../../src/core/tsconfig.json" }]
}
```
5. only packages/plugins which are imported somewhere in the ts code are
listed in `kbn_references`
This linter is not only validating all of the tsconfig.json files, but
it also will fix these config files to deal with just about any
violation that can be produced. Just run `node scripts/ts_project_linter
--fix` locally to apply these fixes, or let CI take care of
automatically fixing things and pushing the changes to your PR.
> **Example:** [`64e93e5`
(#146212)](64e93e5806)
When I merged main into my PR it included a change which removed the
`@kbn/core-injected-metadata-browser` package. After resolving the
conflicts I missed a few tsconfig files which included references to the
now removed package. The TS Project Linter identified that these
references were removed from the code and pushed a change to the PR to
remove them from the tsconfig.json files.
## No bazel? Does that mean no packages??
Nope! We're still doing packages but we're pretty sure now that we won't
be using Bazel to accomplish the 'distributed caching' and 'change-based
tasks' portions of the packages project.
This PR actually makes packages much easier to work with and will be
followed up with the bundling benefits described by the original
packages RFC. Then we'll work on documentation and advocacy for using
packages for any and all new code.
We're pretty confident that implementing distributed caching and
change-based tasks will be necessary in the future, but because of
recent improvements in the repo we think we can live without them for
**at least** a year.
## Wait, there are still BUILD.bazel files in the repo
Yes, there are still three webpack bundles which are built by Bazel: the
`@kbn/ui-shared-deps-npm` DLL, `@kbn/ui-shared-deps-src` externals, and
the `@kbn/monaco` workers. These three webpack bundles are still created
during bootstrap and remotely cached using bazel. The next phase of this
project is to figure out how to get the package bundling features
described in the RFC with the current optimizer, and we expect these
bundles to go away then. Until then any package that is used in those
three bundles still needs to have a BUILD.bazel file so that they can be
referenced by the remaining webpack builds.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
After moving away from composite projects in the IDE we now have an
issue where projects like security solutions are getting `@types/jest`
and `@types/mocha` loaded up, even though the "types" compiler option in
security solutions focuses on jest. To fix this I've removed the
`@types/mocha` package, implemented/copied a portion of the mocha types
into a new `@kbn/ambient-ftr-types` package which can be used in ftr
packages to define the describe/it/etc. globals.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Rewrite the performance journey runner using TypeScript, to avoid
dangling ES\node processes during test execution.
Results are stable with this runner, as verified on CI
Co-authored-by: Spencer <email@spalger.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* [ftr] add first-class support for playwrite journeys
* [CI] Auto-commit changed files from 'node scripts/generate codeowners'
* fix jest test
* remove ability to customize kibana server args, if we need it we can add it back
* remove dev dir that doesn't exist
* fix typo
* prevent duplicated array converstion logic by sharing flag reader
* remove destructuring of option
* fix scalability config and config_path import
* fix start_servers args and tests
* include simple readme
* fix jest tests and support build re-use when changes are just to jest tests
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
