Commit graph

114 commits

Author SHA1 Message Date
Tre
46707e46de
[FTR] Mv test subjs svc to shared location (#174048)
## Summary

Refactoring general ui service to a kbn package.

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2024-01-09 11:43:39 +00:00
Dzmitry Lemechko
f9a4962b48
[ftr] abort retry on invalid webdriver session (#174092)
## Summary

Since many our e2e tests use Retry service, we might have a situation
with retry running for some time (**20,30, 120 seconds** ) before
reaching timeout while browser already crashed and Webdriver is not
functioning properly.

This PR updates Retry service with checking error name before retry
attempt: if it is the WebDriver critical error, retry is aborted and we
fail test fast.

It should help with long useless logging messages as well:

Before:

```
         │ debg --- retry.try error: no such window: target window already closed
         │      from unknown error: web view not found
         │        (Session info: chrome=120.0.6099.129)
         │ debg Find.findByCssSelector('[data-test-subj="canvasExpressionInput"]') with timeout=10000
         │ debg --- retry.try failed again with the same message...
         │ debg Find.findByCssSelector('[data-test-subj="canvasExpressionInput"]') with timeout=10000
         │ debg --- retry.try failed again with the same message...
         ...
         │ERROR Browser is closed, no artifacts were captured for the failure
         └- ✖ fail: Canvas Canvas app expression editor shows autocomplete when typing
         │      retry.try timeout: NoSuchWindowError: no such window: target window already closed
         │ from unknown error: web view not found
         │   (Session info: chrome=120.0.6099.129)
         │     at Object.throwDecodedError (/Users/dmle/github/kibana/node_modules/selenium-webdriver/lib/error.js:524:15)
         │     at parseHttpResponse (/Users/dmle/github/kibana/node_modules/selenium-webdriver/lib/http.js:601:13)
         │     at Executor.execute (/Users/dmle/github/kibana/node_modules/selenium-webdriver/lib/http.js:529:28)
         │     at processTicksAndRejections (node:internal/process/task_queues:95:5)
         │     at Task.exec (prevent_parallel_calls.ts:28:20)
         │   Error: retry.try timeout: NoSuchWindowError: no such window: target window already closed
         │   from unknown error: web view not found
         │     (Session info: chrome=120.0.6099.129)
         │       at Object.throwDecodedError (node_modules/selenium-webdriver/lib/error.js:524:15)
         │       at parseHttpResponse (node_modules/selenium-webdriver/lib/http.js:601:13)
         │       at Executor.execute (node_modules/selenium-webdriver/lib/http.js:529:28)
         │       at processTicksAndRejections (node:internal/process/task_queues:95:5)
         │       at Task.exec (prevent_parallel_calls.ts:28:20)
         │       at onFailure (retry_for_success.ts:17:9)
         │       at retryForSuccess (retry_for_success.ts:59:13)
         │       at RetryService.try (retry.ts:31:12)
         │       at Proxy.clickByCssSelector (find.ts:417:5)
         │       at TestSubjects.click (test_subjects.ts:164:5)
         │       at Context.<anonymous> (expression.ts:92:7)
         │       at Object.apply (wrap_function.js:73:16)
         │
         │
       └-> "after all" hook for "shows autocomplete when typing"
         │ debg unloading docs from archive at /Users/dmle/github/kibana/x-pack/test/functional/fixtures/kbn_archiver/canvas/default.json
         │ info deleting 1 objects { space: undefined }
         │ succ 1 saved objects deleted
       └-> "after all" hook: afterTestSuite.trigger for "shows autocomplete when typing"
         │ERROR Browser window is already closed
``` 

After:

```
         │ debg --- retry.try error: no such window: target window already closed
         │      from unknown error: web view not found
         │        (Session info: chrome=120.0.6099.129)
         │ERROR Browser is closed, no artifacts were captured for the failure
         └- ✖ fail: Canvas Canvas app expression editor shows autocomplete when typing
         │      Error: WebDriver session is invalid, retry was aborted
         │       at retryForSuccess (retry_for_success.ts:64:13)
         │       at RetryService.try (retry.ts:31:12)
         │       at MonacoEditorService.getCodeEditorValue (monaco_editor.ts:25:5)
         │       at Context.<anonymous> (expression.ts:83:34)
         │       at Object.apply (wrap_function.js:73:16)
         │
         │
       └-> "after all" hook for "shows autocomplete when typing"
         │ debg unloading docs from archive at /Users/dmle/github/kibana/x-pack/test/functional/fixtures/kbn_archiver/canvas/default.json
         │ info deleting 1 objects { space: undefined }
         │ succ 1 saved objects deleted
       └-> "after all" hook: afterTestSuite.trigger for "shows autocomplete when typing"
         │ERROR Browser window is already closed
     └-> "after all" hook in "Canvas app"
       │ debg set roles = superuser
       │ debg creating user test_user
       │ debg created user test_user
     └-> "after all" hook: afterTestSuite.trigger in "Canvas app"
       │ERROR Browser window is already closed
   └-> "after all" hook: afterTestSuite.trigger in "Canvas"
     │ERROR Browser window is already closed

5 passing (17.0s)
1 failing

1)    Canvas
       Canvas app
         expression editor
           shows autocomplete when typing:

      Error: WebDriver session is invalid, retry was aborted
       at retryForSuccess (retry_for_success.ts:64:13)
       at RetryService.try (retry.ts:31:12)
       at MonacoEditorService.getCodeEditorValue (monaco_editor.ts:25:5)
       at Context.<anonymous> (expression.ts:83:34)
       at Object.apply (wrap_function.js:73:16)


```
2024-01-02 22:01:26 +01:00
Yulia Čech
a29b0f4573
[Console] Add api integration tests for serverless (#172306)
## Summary

This PR adds api integration test for Console routes: some were missing
on stateful and all are now enabled on serverless.
2023-12-05 10:26:51 +01:00
Maxim Palenov
58adee01a0
[Security Solution] Support Serverless Cypress tests with different roles (#169017)
**Addresses:** https://github.com/elastic/kibana/issues/164451

## Summary

This PR allows to run role based reused between ESS and Serverless Cypress tests.

## Details

The main idea behind is to make environmental differences for tests unnoticeable. As Serverless env already has roles and users but ESS env allows to create any possible role and user we just need to create Serverless roles and corresponding users + specific ESS roles and corresponding users in ESS env before running any ESS tests. This way tests will run in a similar env and don't have to bother by roles/users creation in test suites. This is achieved by using separate Cypress support files (Cypress includes `support/e2e.js` by default) `ess_e2e.ts` and `serverless_e2e.ts` executed for corresponding environments. `ess_e2e.ts` contains logic to create mentioned above roles and users while `serverless_e2e.ts` doesn't contain such logic.

_Only one user created per role and user has the same name as its corresponding role with `changeme` password._

To have an ability to create roles we need to store their definitions somewhere. It's also convenient to have JSON definitions instead of YAML. Plus Serverless roles should be pulled from `project-controller` repo but it's not addressed in this PR. I've chosen the following locations

- Serverless Security roles in `packages/kbn-es/src/serverless_resources/security_roles.json`. While `@kbn/es` is a common package it has `serverless_resources` folder containing `roles.yml` with a mix of `https://github.com/elastic/project-controller/blob/main/internal/project/observability/config/roles.yml`, `https://github.com/elastic/project-controller/blob/main/internal/project/esproject/config/roles.yml` and `https://github.com/elastic/project-controller/blob/main/internal/project/security/config/roles.yml` copied from `project-controller` and used for ES data restore. As there is no automation yet it looks logical to keep Security roles subset next to ES Serverless resources.
- ESS Security specific roles in `x-pack/plugins/security_solution/common/test/ess_roles.json`

On top of that the following has been done

- `reader` role replaced  with `t1_analyst` where possible in tests (besides `e2e/explore/cases/attach_alert_to_case.cy.ts` but it's purely ESS test so it's fine) as `reader` is ESS specific and make harder to run the same tests in ESS and Serverless environments but both roles are almost equivalent
- `login()` helper function accepts all known roles (Serverless + ESS) but throws an exception if a custom ESS role is used under Serverless env
- `x-pack/plugins/security_solution/server/lib/detection_engine/scripts/roles_users` isn't necessary anymore as `security_roles.json` + `ess_roles.json` contain all the necessary data to create roles and users

### Does it enable role support for MKI environments?

No. This PR only enabling role support for Non-MKI Serverless environments. MKI env has predefined roles but not users. This will be addressed in a follow up PR.

## Flaky test runner

Two unskiped in this PR Serverless Cypress tests using non default role `detection_response/detection_alerts/missing_privileges_callout.cy.ts` and `detection_response/prebuilt_rules/prebuilt_rules_install_update_authorization.cy.ts`  [150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3723) 🟢 (there is one env related failure but it doesn't look related to the changes in this PR)
2023-10-31 09:39:47 -07:00
Dzmitry Lemechko
ac8d73ac6d
[ftr] fix test users for serverless (#161280)
## Summary

This PR fixes few issues occurring while running FTR API tests against
actual serverless project.

How to run:
```
TEST_CLOUD=1 ES_SECURITY_ENABLED=1 NODE_TLS_REJECT_UNAUTHORIZED=0 TEST_ES_URL=<your_es_url_with_credentials> TEST_KIBANA_URL=<your_es_url_with_credentials> node  --no-warnings scripts/functional_test_runner --es-version=8.9.0 --config x-pack/test_serverless/api_integration/test_suites/search/config.ts --bail
```

The first error is faced during Elasticsearch version validation

```
ERROR Error: attempted to use the "es" service to fetch Elasticsearch version info but the request failed: ResponseError: {"ok":false,"message":"Unknown resource."}
          at SniffingTransport.request (/Users/dmle/github/kibana/node_modules/@elastic/transport/src/Transport.ts:535:17)
          at processTicksAndRejections (node:internal/process/task_queues:96:5)
          at Client.InfoApi [as info] (/Users/dmle/github/kibana/node_modules/@elastic/elasticsearch/src/api/api/info.ts:60:10)
          at FunctionalTestRunner.validateEsVersion (functional_test_runner.ts:129:16)
          at functional_test_runner.ts:64:11
          at FunctionalTestRunner.runHarness (functional_test_runner.ts:251:14)
          at FunctionalTestRunner.run (functional_test_runner.ts:48:12)
          at log.defaultLevel (cli.ts:112:32)
          at run.ts:70:7
          at withProcRunner (with_proc_runner.ts:29:5)
          at run (run.ts:69:5)
          at FunctionalTestRunner.validateEsVersion (functional_test_runner.ts:131:13)
          at processTicksAndRejections (node:internal/process/task_queues:96:5)
          at functional_test_runner.ts:64:11
          at FunctionalTestRunner.runHarness (functional_test_runner.ts:251:14)
          at FunctionalTestRunner.run (functional_test_runner.ts:48:12)
          at log.defaultLevel (cli.ts:112:32)
          at run.ts:70:7
          at withProcRunner (with_proc_runner.ts:29:5)
          at run (run.ts:69:5)
```

Since there is no version term in case of serverless, we can skip
version check by using newly added to FTR schema `serverless` property
(`false` by default). It is set to `true` in root FTR config
`/shared/config.base`.

The next error is related to ESArchiver relying on `ES` FTR service to
provide ESClient.

```
ResponseError: security_exception
   │ 	Root causes:
   │ 		security_exception: unable to authenticate user [system_indices_superuser] for REST request [/kibana_sample_data_flights]
```

It is fixed by using the default user (from host url) instead of
`system_indices_superuser` we use in stateful run.
2023-07-10 10:09:07 +02:00
Lukas Olson
4b7d18b5c3
[bfetch] Use versioned router (#161317)
## Summary

Part of https://github.com/elastic/kibana/issues/157095.

Uses the new versioned router capabilities for the bfetch plugin.

### Checklist

- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-07-07 16:48:02 -07:00
Matthew Kime
26d4ba5c3e
[data views] Use versioned router for REST routes (#158608)
## Summary

Version alllll the data view routes.

Best viewed with whitespace hidden -
https://github.com/elastic/kibana/pull/158608/files?diff=unified&w=1

In this PR:
- All REST (public and internal) routes are versioned
- Internal routes are called with version specified
- Internal and public routes are now stored in directories labeled as
such
- All routes have a response schema
- All responses are typed with `response` types, separate from internal
api types. This is to help prevent unacknowledged changes to the api.
- Moves some functional tests from js => ts

For follow up PRs:
- Move to `internal` path for internal routes
- Proper typing and schema for `fields_for_wildcard` filter

Closes https://github.com/elastic/kibana/issues/157099
Closes https://github.com/elastic/kibana/issues/157100

---------

Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>
2023-06-12 22:01:03 -05:00
Lukas Olson
34ada8a9a6
[data.search] Use versioned router (#158520)
## Summary

Step 1 of https://github.com/elastic/kibana/issues/157095.

Uses the new versioned router capabilities for the search routes (`POST`
and `DELETE`).

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

### For maintainers

- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Matthias Wilhelm <matthias.wilhelm@elastic.co>
2023-06-07 10:33:39 +02:00
John Dorlus
20ebb175df
Added Rollups CCS Test (#144074)
* Removed comment of the issue that was referenced for the skip. But the tests were already skipped.

* Unskipping test as a fix has been made. 138510

* Made CCS test for rollups and made it conditional based on configuration.

* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'

* Fixed issues in build.

* Added comment to rollups test and using super user until the perms issue is fixed.

Co-authored-by: cuffs <cuffs@cuffss-Office-MacBook-Pro.local>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2022-10-28 15:20:50 -04:00
Spencer
50b3b57d9e
[ftr] add first-class support for playwrite journeys (#140680)
* [ftr] add first-class support for playwrite journeys

* [CI] Auto-commit changed files from 'node scripts/generate codeowners'

* fix jest test

* remove ability to customize kibana server args, if we need it we can add it back

* remove dev dir that doesn't exist

* fix typo

* prevent duplicated array converstion logic by sharing flag reader

* remove destructuring of option

* fix scalability config and config_path import

* fix start_servers args and tests

* include simple readme

* fix jest tests and support build re-use when changes are just to jest tests

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2022-09-22 01:06:46 -07:00
Dzmitry Lemechko
7b0033f68c
[scalability testing] store relative path to kbnArchives/esArchives in dataset extractor output file (#139891)
* update journeys, save testData in output json

* fix config inheritance

Co-authored-by: spalger <spencer@elastic.co>
2022-09-01 18:34:40 +02:00
Tim Sullivan
160058a8c1
[search/public] expose showWarnings(inspector) method on search service (#138342)
* add showWarning to search service

* add comments

* add unit tests

* test foo

* cleanup

* add s to property name in test

* comments for api items

* use the warnings when calling showWarnings

* change showWarning to just show a single warning

* put handleWarnings on the request adapter

* comment

* simplify 1

* fix lens unit test

* remove underscoring for unused variables

* revert inspector changes, extract the response warnings in the search service

* fix bug

* remove console.log

* re-apply typescript fixes to app test code

* declutter

* add test, improve comments

* fix some unexported public api items

* include rawResponse in the warning structure

* fix lint

* tweak clean up example app

* SearchResponseWarnings and SearchResponseWarningNotification

* fix export bug

* not include shardStats if there are no warnings

* Update src/plugins/data/common/search/types.ts

* simplify SearchResponseWarnings interface

* remove array copying

* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'

* comments for api_docs

* simplify per feedback

* Pass callback to handleResponse in showWarnings

* export more public types

* update example to make possible to show shard failure

* pr cleanup

* eslint fix

* allow example app to not show default warnings

* move extractWarning and related types to inspector plugin

* wip functional test of example app

* fix test references

* finish functional test

* relocate extractWarnings back to search/fetch

* fix test

* remove need for isTimeout, isShardFailure

* ts fix

* improve test

* Change showWarnings to accept the RequestAdapter

* use showWarnings in vis_types/timeseries

* more tests

* use handle_warning name

* fix ts

* add reason field to SearchResponseWarning

* fix component snapshot

* update comments

* test cleanup

* fix test

* ensure notification appears only once

* fix and cleanup

* fix ts

* fix response.json bug

* use top-level type, and lower-level reason.type

* cleanup

* fix shard failure warning in tsvb per feedback

cc @flash1293

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>
2022-08-31 11:22:24 -07:00
Spencer
f3d69b8197
[@kbn/dev-utils] break out more pieces (#132292)
* [@kbn/dev-utils] break out more pieces

* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2022-05-17 11:19:20 -05:00
Dzmitry Lemechko
80427ea1ba
[scalability testing] extend FTR config with optional scalability configuration (#132047)
* [scalability testing] extend FTR config

* update schema with defaults for testData
2022-05-11 20:29:30 +02:00
Spencer
2a78f350e1
break out parts of @kbn/dev-utils (#130509)
* break out parts of @kbn/dev-utils

* autofix imports and kbn/pm dist

* update readme for @kbn/stdio-dev-helpers

* finish renames
2022-04-19 12:24:58 -05:00
spalger
3730dd0779 fix all violations 2022-04-16 01:37:30 -05:00
Dmitry Tomashevich
0427952e76
[Discover] Extend Elasticsearch query rule with search source based data fetching (#124534)
* [Discover] introduce .index-threshold rule

* [Discover] change filters in alert expression

* [Discover] fix cursor issue

* [Discover] add loading

* [Discover] separate validation params

* [Discover] add view alert route

* [Discover] enable "view in app" for alert created from discover

* [Discover] fix filter popover

* [Discover] fix linting, unit tests

* [Discover] fix remaining tests

* [Discover] add unit tests, add link back to stack management for es query

* Update src/plugins/discover/public/application/view_alert/view_alert_route.tsx

* [Discover] add tool tip for data view without time field

* [Discover] add info alert about possible document difference that triggered alert and displayed documents

* [Discover] update unit test

* [Discover] fix unit tests

* Update x-pack/plugins/stack_alerts/public/alert_types/es_query/expression/search_source_expression.tsx

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>

* Update x-pack/plugins/stack_alerts/server/alert_types/es_query/alert_type/alert_type.ts

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>

* Update x-pack/plugins/stack_alerts/server/alert_types/es_query/alert_type/alert_type.ts

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>

* Update x-pack/plugins/stack_alerts/server/alert_types/es_query/alert_type/alert_type.ts

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>

* Update src/plugins/discover/public/application/main/components/top_nav/open_alerts_popover.tsx

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>

* Update x-pack/plugins/stack_alerts/public/alert_types/es_query/expression/search_source_expression.tsx

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>

* [Discover] fix unit tests

* [Discover] fix security solution alerts

* [Discover] fix eslint errors

* [Discover] fix unit tests

* Update x-pack/plugins/stack_alerts/server/alert_types/es_query/alert_type/alert_type.ts

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>

* Update x-pack/plugins/stack_alerts/server/alert_types/es_query/alert_type/alert_type.ts

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>

* [Discover] apply suggestions

* [Discover] fix tests

* Update x-pack/plugins/stack_alerts/server/alert_types/es_query/alert_type/alert_type.ts

* [Discover] remove close button in filters

* Improve code structure

* Fix missing name in fetchEsQuery

* Fix messages

* Fix messages, again

* Refactor

* Refactor, add tests + a bit more of documentation

* Move size field, change text

* Implement readonly callout

* change icon in callout

* add padding to popover

* Hide query and filter UI if there are no values to display

* [Discover] add unit test, improve comparator types

* [Discover] fix linting and unit test

* [Discover] add es query alert integration tests

* [Discover] fix linting

* [Discover] uncomment one expect

* [Discover] fix latesTimestamp for searchSource type, unify test logic

* Update x-pack/plugins/stack_alerts/public/alert_types/es_query/expression/search_source_expression.tsx

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>

* [Discover] apply suggestions

* [Discover] make searchType optional, adjust tests

* [Discover] remove updated translations

* [Discover] apply suggestions

* [Discover] fix unit test

* [Discover] close popover on alert rule creation

* [Discover] apply suggestions

* [Discover] add first functional test

* [Discover] implement tests

* Move functionals x-pack since ssl is needed

* Fix potential flakiness in functional test

* [Discover] remove timeout waiter

* Fix functional test

- adding permissions to fix the functional

* [Discover] add logger

* [Discover] add more log points

* [Discover] wait for indices creation finished

* Try to fix the functional flakiness
- by creating data views in a serial way
- lets see if that work

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>
Co-authored-by: Matthias Wilhelm <matthias.wilhelm@elastic.co>
Co-authored-by: andreadelrio <andrea.delrio@elastic.co>
2022-04-01 14:57:57 +05:00
Spencer
0821c31fa5
[ftr] implement support for accessing ES through CCS (#126547)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2022-03-07 15:27:41 -07:00
Spencer
be1028c345
[functional/security/test-user] remove naked boolean (#126652)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2022-03-03 08:56:56 -07:00
Robert Oskamp
befefc3347
FTR - check ES security before creating system_indices_superuser (#124948)
This PR adds a check if ES security is enabled before creating the system_indices_superuser in the security service.
2022-02-08 16:32:10 +01:00
Robert Oskamp
8989ead2d6
Functional test runner creates system_indices_superuser (#124008)
This PR fixes the functional test runner for execution against cloud (and other existing deployments) by making sure the system_indices_superuser exists.
2022-02-04 11:23:47 +01:00
Matthew Kime
f3ff2291f5
less deprecated index pattern apis (#124313) 2022-02-02 10:39:59 -06:00
Tyler Smalley
23739f5f4d
Support for superuser not having write access (#123337)
Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
Co-authored-by: Timothy Sullivan <tsullivan@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2022-01-24 09:39:08 -08:00
Aleh Zasypkin
2ba281b8aa
Cleaning up after DLS functional tests. (#122426) 2022-01-11 09:50:57 +01:00
Frank Hassanabad
ae7b5a9be9
[Security Solutions] Adds bsearch service to FTR e2e tests to reduce flake, boilerplate, and technique choices (#116211)
## Summary

Fixes flake tests of:
https://github.com/elastic/kibana/issues/115918
https://github.com/elastic/kibana/issues/103273
https://github.com/elastic/kibana/issues/108640
https://github.com/elastic/kibana/issues/109447
https://github.com/elastic/kibana/issues/100630
https://github.com/elastic/kibana/issues/94535
https://github.com/elastic/kibana/issues/104260

Security solution has been using `bsearch` and has encountered flake in various forms. Different developers have been fixing the flake in a few odd ways (myself included) which aren't 100%. This PR introduces a once-in-for-all REST API retry service called `bsearch` which will query `bsearch` and if `bsearch` is not completed because of async occurring due to slower CI runtimes it will continuously call into the `bsearch` with the correct API to ensure it gets a complete response before returning.


## Usage

Anyone can use this service like so:
```ts
const bsearch = getService('bsearch');
const response = await bsearch.send<MyType>({
 supertest,
 options: {
   defaultIndex: ['large_volume_dns_data'],
}
  strategy: 'securitySolutionSearchStrategy',
});
```

If you're using a custom auth then you can set that beforehand like so:
```ts
const bsearch = getService('bsearch');
const supertestWithoutAuth = getService('supertestWithoutAuth');
const supertest supertestWithoutAuth.auth(username, password);
const response = await bsearch.send<MyType>({
 supertest,
 options: {
   defaultIndex: ['large_volume_dns_data'],
  }
  strategy: 'securitySolutionSearchStrategy',
});
```

## Misconceptions in the tests leading to flake
* Can you just call the bsearch REST API and it will always return data first time? Not always true, as when CI slows down or data increases `bsearch` will give you back an async reference and then your test will blow up.
* Can we wrap the REST API in `retry` to fix the flake? Not always but mostly true, as when CI slows down or data increases `bsearch` could return the async version continuously which could then fail your test. It's also tedious to tell everyone in code reviews to wrap everything in `retry` instead of just fixing it with a service as well as inform new people why we are constantly wrapping these tests in `retry`.
* Can we manually parse the `bsearch` if it has `async` for each test? This is true but is error prone and I did this for one test and it's ugly and I had issues as I have to wrap 2 things in `retry` and test several conditions. Also it's harder for people to read the tests rather than just reading there is a service call. Also people in code reviews missed where I had bugs with it. Also lots of boiler plate.
* Can we just increase the timeout with `wait_for_completion_timeout` and the tests will pass for sure then? Not true today but maybe true later, as this hasn't been added as plumbing yet. See this [open ticket](https://github.com/elastic/kibana/issues/107241). Even if it is and we increase the timeout to a very large number bsearch might return with an `async` or you might want to test the `async` path. Either way, if/when we add the ability we can increase it within 1 spot which is this service for everyone rather than going to each individual test to add it. If/when it's added if people don't use the bsearch service we can remove it later if we find this is deterministic enough and no one wants to test bsearch features with their strategies down the road.

## Manual test of bsearch service
If you want to manually watch the bsearch operate as if the CI system is running slow or to cause an `async` manually you manually modify this setting here:
https://github.com/elastic/kibana/blob/master/src/plugins/data/server/search/strategies/ese_search/request_utils.ts#L61

To be of a lower number such as `1ms` and then you will see it enter the `async` code within `bsearch` consistently

## Reference PRs
We cannot set the wait_for_complete just yet
https://github.com/elastic/kibana/issues/107241 so we decided this was the best way to reduce flake for testing for now. 

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-10-27 14:22:45 -06:00
Mikhail Shustov
3c8fa527a7
[ES] Upgrade client to v8.0 (#113950)
* bump to a pre-8.0 version

* export KibanaClient from /lib sub-folder

* workaround the problem of the absence of estypes

* update es client usage in pacakges

* export estypes from another path

* import errors from root

* import errors from root 2

* update transport import

* update import path for /api/types

* update import path for /api/types

* import errors from top export

* use TransportResult instead if ApiResponse

* fix errors in client_config

* fix src/core/server/saved_objects/migrationsv2/actions/integration_tests/actions.test.ts

* use KibanaClient in mock. we dont export the original Client

* fix client mocks

* fix errors on SO

* fix remaining core errors

* update estype import path

* fix errors in data plugin

* fix data_views

* fix es_ui_shared

* fix errors in interactive_setup

* fix errors in ./test folder

* add @elastic/transport to the runtime deps

* fix errors in packages

* fix erros in src/core

* fix errors in test/

* fix an error in actions plugin

* woraround and fix errors in APM plugin

* fix errors in canvas

* fix errors in event_log

* fix errors in fleet

* fix errors in ILM

* fix errors in infra

* fix errors in ingest_pipeline

* fix errors in lens

* fix errors in license_management

* fix errors in licensing

* fix errors in logstash

* fix errors in ml

* fix errors in monitoring

* fix errors in observability

* fix errors in rule_registry

* fix errors in reporting

* fix errors in rule_registry

* fix errors in security

* fix errors in security_solution

* fix errors in snapshot_restore

* fix errors in transform

* fix errors in UA

* fix errors in uptime

* fix errors in x-pack/test

* fix eslint errors

* fix new errors

* use default HTTP Connection. Undici does not support agent config options keepAlive and maxSockets

* create does not accept require_alias option

* update deps

* use transport types exported from ES client package

* fix ErrorCause | string errors

* do not use enum

* fix errors in data plugin

* update x-pack code

* fix transport

* fix apm search request

* do not crash on reporting

* fix kbn-test build

* mute reporting error to start

* fix ftr build

* another attempt

* update import path

* address or mute new errors

* REMOVE me. pin transport version temporarily.

* remove deep imports from transport package

* fix jest crash

* fix product check tests

* remove unnecessary ts-expect-error

* fix a few failed unit tests

* bump to canary 24

* remove unnecessary ts-expect-error

* remove dependency on transport

* fix types in tests

* mute errors in xpack tests

* product check doesn;t  spam in logs anymore

* filterPath --> filter_path

* ignoreUnavailable --> ignore_unavailable

* ignoreUnavailable --> ignore_unavailable

* trackScores --> track_scores

* trackTotalHits --> track_total_hits

* fix es-arcives

* fix data plugin crashes

* fix watcher test utils

* rollback unnecessary changes

* fix another problem in es-archiver

* fix scroll. for whatever reason scroll fails when request scroll_id in body

* add meta: true in kbn-securitysolution-es-utils

* bump client to canary 25

* fix errors in accordance with the es client spec

* update securityscolution-es-utils

* unify scroll api in reporting and fix tests

* fix unit tests in watcher

* refactor APM to abort request with AbortController API

* fix missing es client calls in tests

* fix missing meta in detection engine FTR tests

* fix another bunch of errors in js tests

* fix wrong coercion

* remove test-grep pattern

* fix apm unit test

* rename terminateAfter to terminate_after in infra plugin

* rename terminateAfter to terminate_after in uptime plugin

* rename terminateAfter to terminate_after in apm plugin

* fix security roles FTR tests

* fix reference

* fix post_privilidges test

* fix post_privilidges

* bump client to 26

* add meta for index_management test helpers

* remove ts-expect-error caused by bad type in reason

* bump client to 27

* REMOVE me. workaround until fixed in the es client

* fix incorrect type casting

* swtich from camelCase params

* use `HttpConnection` for FTR-related clients

* bump client to 29

* Revert "REMOVE me. workaround until fixed in the es client"

This reverts commit c038850c09.

* fix new util

* revert repository changes

* do not crash if cannot store event_loop data

* fix new estypes imports

* fix more types

* fix security test types and add ts-ignore for custom ES client

* fix more estypes imports

* yet more ts violations

* line by line fixing is hard

* adapt `evaluateAlert` from infra as it's also used from FTR tests

* use convertToKibanaClient in FTR test instead of meta:true in plugin code

* migrate from deprecated API in fleet

* fix intergration tests

* fix fleet tests

* fix another fleet test

* fix more tests

* let's call it a day

* Removes custom header check on 404 responses, includes es client ProductNotSupportedError in EsUnavailableError conditional (#116029)

* Removes custom header check on 404 responses, includes es client ProductNotSupportedError in EsUnavailableError conditional

* Updates proxy response integration test

* disable APM until compatible with client v8

* skip async_search FTR test

* use kbnClient in integration tests

* bump version to 29

* bump to 30

* have configureClient return a KibanaClient instead of Client, remove resolved violations.

* bump to 31

* bump to 31

* Revert "bump to 31"

This reverts commit 5ac713e640.

* trigger stop to unusubscribe

* update generated docs

* remove obsolete test

* put "as" back

* cleanup

* skip test

* remove new type errors in apm package

* remove ErrorCause casting

* update a comment

* bump version to 32

* remove unnecessary ts-expect-error in apm code

* update comments

* update to client v33

* remove outdated type definition

* bump to 34 without params mutation

* unskip the test that should not fail anymore

* remove unnecessary ts-expect-error comments

* update to v35. body can be string

* move `sort` to body and use body friendly syntax

* fix a failing test. maps register the same SO that has been already registered by home

Co-authored-by: pgayvallet <pierre.gayvallet@gmail.com>
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
2021-10-26 14:08:22 +02:00
Thomas Watson
f152787a68
Remove deprecated xpack.security.enabled config option (#111681)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-10-25 14:12:05 -04:00
Robert Oskamp
f7174809e1
Functional tests - fix retry.waitFor not timing out correctly (#113905)
This PR fixes bugs in test services, where retry.waitForWithTimeout and find.descendantExistsByCssSelector do not time out correctly.
2021-10-07 11:23:33 +02:00
Tre
240716d2ee
[QA][SO INFO SVC] Drop in-test jq filtering (#113655) 2021-10-04 21:02:51 +01:00
Tre
0b4a8c081c
[QA][SO INFO SVC] Add jq filtering (#113302) 2021-10-01 16:52:20 +01:00
Tyler Smalley
4681a80317
[DX] Upgrade prettier to v2.4.0 (#112359)
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
2021-09-19 22:34:30 -07:00
Tre
458a57939b
[QA][SO INFO SVC] Docs for using with jq (#112129) 2021-09-14 17:40:00 +01:00
Tre
9762da497d
[QA][SO INFO SVC] Add json only option (#112061) 2021-09-14 15:31:08 +01:00
Mikhail Shustov
641cef7ca6
Functional tests for execution context (#110299)
* remove unnecessary ts-ignore

* add context propagation to x-opaque-id header tests

* run tests on CI

* simplify logging. the action purpose follows from the context name

* extend tests with the assertion against execution_context from the Kibana logs

* split JSON log records only

* apply suggestions proposed by Spencer
2021-09-03 05:48:32 -04:00
Tyler Smalley
689d974729
Removes supertest-as-promised dependency (#100486)
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
2021-08-16 18:01:34 -07:00
Tre
8a571c2f7d
[QA][SO INFO SVC] Add cli (#108353)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-08-16 13:36:22 +01:00
Anton Dosov
6450df1885
[FieldFormats] Add editors tests (#107770) 2021-08-10 09:44:24 -04:00
Mikhail Shustov
41b6a99282
Remove legacyEs test service (#107567)
* migrate uptime tests to the new client. it didn't use legacy one anyway

* remove legacyEs service
2021-08-04 14:21:35 +02:00
Tre
d0b5c9e016
[QA] Extend Saved Objects Info Svc (#105800) 2021-07-16 15:02:39 +01:00
Spencer
7917e3c9d9
[kbnArchiver] convert archive names to root-relative paths (#101839)
* [kbnArchiver] convert archive names to root-relative paths

* ensure that newly multiline hooks are explicitly async

* missed a newly multiline hook

* fix exists check

* avoid extra lines by wrapping arrow body in {}

* one block more

* fix errant `name` variable

Co-authored-by: spalger <spalger@users.noreply.github.com>
2021-06-09 20:13:00 -04:00
Spencer
f466ebf1a3
[esArchiver] drop support for --dir, use repo-relative paths instead (#101345)
Co-authored-by: spalger <spalger@users.noreply.github.com>
2021-06-08 17:37:42 -04:00
Spencer
bdafd27e19
[ts] migrate x-pack/test to composite ts project (#101441)
Co-authored-by: spalger <spalger@users.noreply.github.com>
2021-06-08 12:54:05 -04:00
Mikhail Shustov
d920682e4e
Update @elastic/elasticsearch to 8.0.0-canary13 (#98266)
* bump @elastic/elasticsearch to canary.7

* address errors in core

* address errors in data plugin

* address errors in Alerting team plugins

* remove outdated messages in Lens

* remove unnecessary comments in ML

* address errors in Observability plugin

* address errors in reporting plugin

* address errors in Rule registry plugin

* fix errors in Security plugins

* fix errors in ES-UI plugin

* remove unnecessary union.

* update core tests

* fix kbn-es-archiver

* update to canary 8

* bump to v9

* use new typings

* fix new errors in core

* fix errors in core typeings

* fix type errors in data plugin

* fix type errors in telemetray plugin

* fix data plugin tests

* fix search examples type error

* fix errors in discover plugin

* fix errors in index_pattern_management

* fix type errors in vis_type_*

* fix errors in typings/elasticsearch

* fix type errors in actions plugin

* fix type errors in alerting and apm plugins

* fix type errors in canvas and cases

* fix errors in event_log

* fix type errors in ILM and ingest_pipelines

* fix errors in lens plugin

* fix errors in lists plugin

* fix errors in logstash

* fix errors in metrics_entities

* fix errors in o11y

* fix errors in watcher

* fix errors in uptime

* fix errors in upgrade_assistant

* fix errors in task_manager

* fix errors in stack_alerts

* fix errors in security_solution

* fix errors in rule_registry

* fix errors in snapshot_restore

* fix remaining errors

* fix search intergration tests

* adjust assetion

* bump version to canary.10

* adapt code to new naming schema

* use mapping types provided by the client library

* Revert "adjust assetion"

This reverts commit 19b8fe0464.

* fix so intergration tests

* fix http integration tests

* bump version to canary 11

* fix login test

* fix http integration test

* fix apm test

* update docs

* fixing some ml types

* fix new errors in data plugin

* fix new errors in alerting plugin

* fix new errors in lists plugin

* fix new errors in reporting

* fix or mute errors in rule_registry plugin

* more ML type fixes

* bump to canary 12

* fix errors after merge conflict

* additional ML fixes

* bump to canary 13

* fix errors in apm plugin

* fix errors in fleet plugin

* fix errors in infra plugin

* fix errors in monitoring plugin

* fix errors in osquery plugin

* fix errors in security solution plugins

* fix errors in transform plugin

* Update type imports for ES

* fix errors in x-pack plugins

* fix errors in tests

* update docs

* fix errors in x-pack/test

* update error description

* fix errors after master merge

* update comment in infra plugin

* fix new errors on xpack tests/

Co-authored-by: James Gowdy <jgowdy@elastic.co>
Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
2021-06-08 15:06:06 +02:00
Spencer
111e15a054
[ftr] implement FtrService classes and migrate common services (#99546)
Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-05-25 09:25:09 -07:00
Tyler Smalley
537be25754
[FTR][CI] Use default distribution for all tests (#94968)
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
2021-05-12 15:24:25 -04:00
Aleh Zasypkin
808959e316
Handle 401 Unauthorized errors in a more user-friendly way (#94927) 2021-04-27 19:09:54 +02:00
Tre
9ec466e5a1
[QA] Add saved objects info svc (#96364)
Add svc and fn to return saved object types in a given index,
defaulted to ".kibana" index.
2021-04-08 14:46:52 -06:00
Tomas Della Vedova
238791b942
ES client : use the new type definitions (#83808)
* Use client from branch

* Get type checking working in core

* Fix types in other plugins

* Update client types + remove type errors from core

* migrate Task Manager Elasticsearch typing from legacy library to client library

* use SortOrder instead o string in alerts

* Update client types + fix core type issues

* fix maps ts errors

* Update Lens types

* Convert Search Profiler body from a string to an object to conform to SearchRequest type.

* Fix SOT types

* Fix/mute Security/Spaces plugins type errors.

* Fix bootstrap types

* Fix painless_lab

* corrected es typing in Event Log

* Use new types from client for inferred search responses

* Latest type defs

* Integrate latest type defs for APM/UX

* fix core errors

* fix telemetry errors

* fix canvas errors

* fix data_enhanced errors

* fix event_log errors

* mute lens errors

* fix or mute maps errors

* fix reporting errors

* fix security errors

* mute errors in task_manager

* fix errors in telemetry_collection_xpack

* fix errors in data plugins

* fix errors in alerts

* mute errors in index_management

* fix task_manager errors

* mute or fix lens errors

* fix upgrade_assistant errors

* fix or mute errors in index_lifecycle_management

* fix discover errors

* fix core tests

* ML changes

* fix core type errors

* mute error in kbn-es-archiver

* fix error in data plugin

* fix error in telemetry plugin

* fix error in discover

* fix discover errors

* fix errors in task_manager

* fix security errors

* fix wrong conflict resolution

* address errors with upstream code

* update deps to the last commit

* remove outdated comments

* fix core errors

* fix errors after update

* adding more expect errors to ML

* pull the lastest changes

* fix core errors

* fix errors in infra plugin

* fix errors in uptime plugin

* fix errors in ml

* fix errors in xpack telemetry

* fix or mute errors in transform

* fix errors in upgrade assistant

* fix or mute fleet errors

* start fixing apm errors

* fix errors in osquery

* fix telemetry tests

* core cleanup

* fix asMutableArray imports

* cleanup

* data_enhanced cleanup

* cleanup events_log

* cleaup

* fix error in kbn-es-archiver

* fix errors in kbn-es-archiver

* fix errors in kbn-es-archiver

* fix ES typings for Hit

* fix SO

* fix actions plugin

* fix fleet

* fix maps

* fix stack_alerts

* fix eslint problems

* fix event_log unit tests

* fix failures in data_enhanced tests

* fix test failure in kbn-es-archiver

* fix test failures in index_pattern_management

* fixing ML test

* remove outdated comment in kbn-es-archiver

* fix error type in ml

* fix eslint errors in osquery plugin

* fix runtime error in infra plugin

* revert changes to event_log cluser exist check

* fix eslint error in osquery

* fixing ML endpoint argument types

* fx types

* Update api-extractor docs

* attempt fix for ese test

* Fix lint error

* Fix types for ts refs

* Fix data_enhanced unit test

* fix lens types

* generate docs

* Fix a number of type issues in monitoring and ml

* fix triggers_actions_ui

* Fix ILM functional test

* Put search.d.ts typings back

* fix data plugin

* Update typings in typings/elasticsearch

* Update snapshots

* mute errors in task_manager

* mute fleet errors

* lens. remove unnecessary ts-expect-errors

* fix errors in stack_alerts

* mute errors in osquery

* fix errors in security_solution

* fix errors in lists

* fix errors in cases

* mute errors in search_examples

* use KibanaClient to enforce promise-based API

* fix errors in test/ folder

* update comment

* fix errors in x-pack/test folder

* fix errors in ml plugin

* fix optional fields in ml api_integartoon tests

* fix another casting problem in ml tests

* fix another ml test failure

* fix fleet problem after conflict resolution

* rollback changes in security_solution. trying to fix test

* Update type for discover rows

* uncomment runtime_mappings as its outdated

* address comments from Wylie

* remove eslint error due to any

* mute error due to incompatibility

* Apply suggestions from code review

Co-authored-by: John Schulz <github.com@jfsiii.org>

* fix type error in lens tests

* Update x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_service.ts

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

* Update x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_service.test.ts

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

* update deps

* fix errors in core types

* fix errors for the new elastic/elasticsearch version

* remove unused type

* remove unnecessary manual type cast and put optional chaining back

* ML: mute Datafeed is missing indices_options

* Apply suggestions from code review

Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com>

* use canary pacakge instead of git commit

Co-authored-by: Josh Dover <me@joshdover.com>
Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com>
Co-authored-by: Gidi Meir Morris <github@gidi.io>
Co-authored-by: Nathan Reese <reese.nathan@gmail.com>
Co-authored-by: Wylie Conlon <wylieconlon@gmail.com>
Co-authored-by: CJ Cenizal <cj@cenizal.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
Co-authored-by: restrry <restrry@gmail.com>
Co-authored-by: James Gowdy <jgowdy@elastic.co>
Co-authored-by: John Schulz <github.com@jfsiii.org>
Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
2021-03-25 04:47:16 -04:00
Spencer
0aabc317ec
[kbn/test] add import/export support to KbnClient (#92526)
Co-authored-by: Tre' Seymour <wayne.seymour@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: spalger <spalger@users.noreply.github.com>
2021-02-25 16:13:27 -07:00
Tyler Smalley
b39ad86b5d
Use default ES distribution for functional tests (#88737)
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-02-08 14:23:10 -08:00