## Summary
Resolves: https://github.com/elastic/kibana/issues/192201
- Expose `TryWithRetriesOptions`
- Tune timeouts to pass ci
- Add attempt count debug info to `retry/retry_for_success.ts`
- Helps with tuning timeout params
- Move exposure of `AlertingApiProvider` from
`x-pack/test_serverless/api_integration/services/index.ts` ->
`x-pack/test_serverless/shared/services/deployment_agnostic_services.ts`
- This exposes the alerting api under Deployment Agnostic Services (DA),
and DA is exposed within
`x-pack/test_serverless/functional/services/index.ts` (Shared Services
[Serverless])
- Collapse helper script functions into just another object literal
stanza within `AlertingApiProvider`
- Update all references
- Refactor alerting api to use `retry` service, instead of p-retry
(following [this pr](https://github.com/elastic/kibana/pull/178660))
### Additional debug logging
Run in debug mode (add `-v`):
```
node scripts/functional_tests \
--config x-pack/test_serverless/api_integration/test_suites/search/common_configs/config.group1.ts \
--grep "Summary actions"
-v
```
#### After
```
│ sill retry.tryWithRetries('Alerting API - waitForDocumentInIndex, retryOptions: {"retryCount":5,"retryDelay":200}', [object AsyncFunction], [object Object])
│ debg --- retry.tryWithRetries error: index_not_found_exception
│ Root causes:
│ index_not_found_exception: no such index [alert-action-es-query] - Attempt #: 1
│ sill es.search([object Object])
│ debg --- retry.tryWithRetries failed again with the same message... - Attempt #: 2
│ sill es.search([object Object])
│ debg --- retry.tryWithRetries failed again with the same message... - Attempt #: 3
│ sill es.search([object Object])
│ debg --- retry.tryWithRetries failed again with the same message... - Attempt #: 4
│ sill es.search([object Object])
│ debg --- retry.tryWithRetries failed again with the same message... - Attempt #: 5
...
// Msg after all attempts fail:
│ Error: retry.tryWithRetries reached the limit of attempts waiting for 'Alerting API - waitForDocumentInIndex, retryOptions: {"retryCount":5,"retryDelay":200}': 5 out of 5
│ ResponseError: index_not_found_exception
│ Root causes:
│ index_not_found_exception: no such index [alert-action-es-query]
│ at SniffingTransport._request (node_modules/@elastic/transport/src/Transport.ts:601:17)
│ at processTicksAndRejections (node:internal/process/task_queues:95:5)
│ at /Users/trezworkbox/dev/main.worktrees/cleanup-alerting-api/node_modules/@elastic/transport/src/Transport.ts:704:22
│ at SniffingTransport.request (node_modules/@elastic/transport/src/Transport.ts:701:14)
│ at Proxy.SearchApi (node_modules/@elastic/elasticsearch/src/api/api/search.ts:96:10)
│ at alerting_api.ts:123:28
│ at runAttempt (retry_for_success.ts:30:15)
│ at retryForSuccess (retry_for_success.ts:99:21)
│ at Proxy.tryWithRetries (retry.ts:113:12)
│ at Object.waitForDocumentInIndex (alerting_api.ts:120:14)
│ at Context.<anonymous> (summary_actions.ts:146:20)
│ at Object.apply (wrap_function.js:74:16)
│ at Object.apply (wrap_function.js:74:16)
│ at onFailure (retry_for_success.ts:18:9)
│ at retryForSuccess (retry_for_success.ts:75:7)
│ at Proxy.tryWithRetries (retry.ts:113:12)
│ at Object.waitForDocumentInIndex (alerting_api.ts:120:14)
│ at Context.<anonymous> (summary_actions.ts:146:20)
│ at Object.apply (wrap_function.js:74:16)
│ at Object.apply (wrap_function.js:74:16)
```
### Notes
Was put back in draft to additional scope detailed in issue linked
above.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Use api keys for api calls, to act as a real user **not the operator
user,** within:
`x-pack/test_serverless/api_integration/test_suites/common/index_management/`
Update `test_serverless/README.md`
## Important (More helpers)
In order to create severless versions of statefull helpers that require
role credentials, the pr adds a number of helpers to
`x-pack/test_serverless/api_integration/services/index_management/svl_*.ts`
With these additions, any given serverless api integration test can call
the serverless version of a helper, and the stateful version of the
helper, is left untouched.
Further, the type checker is satisfied (cant share code across
boundaries, etc. mostly within the tsconfig.json files).
## For Reviewers
### Issues raised:
index_templates: https://github.com/elastic/kibana/issues/187368
datastreams: https://github.com/elastic/kibana/issues/187372
settings: https://github.com/elastic/kibana/issues/187369
Contributes to: https://github.com/elastic/kibana/issues/180834
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
closes [186450](https://github.com/elastic/kibana/issues/186450)
## Summary
Add tests for aggregate critical path and service map endpoints to
serverless test suite
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR fixes an issue with serverless search tests when running against
MKI. Due to a different test run order there, the search project default
data view does not necessarily exist anymore when the test runs (as
another test might have cleaned it up already). To fix this, the default
data view is re-created during `before` of the default data view test
suite.
As part of this PR, the `data_view_api` service is moved from the
`api_integration` services to the `shared` services, so it can be used
from functional tests as well.
## 🍒 Summary
This PR adds basic serverless integration tests for SLO and covers 2
scenarios:
- SLO creation
- SLO deletion
There is another PR that adds SLO integration tests for stateful and
covers more scenarios:
- create
- delete
- update
- reset
- get/find
Current PR will cover only the create and delete scenarios. I want to
check the flakiness of these tests, before introducing new ones. Another
reason for not covering all scenarios in this PR is because we don't
want to have duplicate effort with @dominiqueclarke's
[PR](https://github.com/elastic/kibana/pull/173236). Once stateful tests
are reviewed and merged, we can come up with a plan on how/if to
continue with serverless tests and more scenarios.
**TODO**: Create a github issue to track the SLO integration tests
effort. Also we need to add privilege & space related test cases
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
We try to reuse existing FTR services for serverless test as much as
possible. Most of the services are deployment agnostic, however some
won't work on serverless and we add its own serverless implementation.
Currently we import all the services from api integration tests directly
into the serverless services. The same with functional ones. This PR
loads the minimum required services for api integration and functional
tests respectively.
This way we can control which services are deployment agnostic and
explicitly remove any of it when we decide to have its own
implementation for serverless.
## Summary
This PR fixes the serverless cases tests for MKI runs against
Observability and Security projects.
### The issues we were seeing
There were actually two issues where things worked locally but not in
MKI:
* A hard-coded `elastic_serverless` user
* Deleting from system indices directly
### How this PR solves them
* Replace the hard-coded `elastic_serverless` user with the one obtained
via the test config
* Replace deletion from system indices with saved object API `clean`
calls
### Other changes
I've noticed, that the tests are using local helper methods, but
observability and security helper methods were 99.9% the same code. In
order to make the code easier to re-use and also allow usage of other
services from within the helper methods, we recommend putting helper
methods into services. I've refactored the code to do this:
* Create a new serverless API integrations service `svlCases`
* Combine
`x-pack/test_serverless/api_integration/test_suites/observability/cases/helpers/api.ts`
and
`x-pack/test_serverless/api_integration/test_suites/security/cases/helpers/api.ts`
into the `svlCases.api` service
* Combine
`x-pack/test_serverless/api_integration/test_suites/observability/cases/helpers/omit.ts`
and
`x-pack/test_serverless/api_integration/test_suites/security/cases/helpers/omit.ts`
into the `svlCases.omit` service
* Clean up dependencies a bit ( e.g. no need to pass `supertest` around
since `svlCases.api` can load the supertest service on its own now)
Closes#161337
## Summary
Uses build flavor(see #161930) to disable specific Kibana security,
spaces, and encrypted saved objects HTTP API routes in serverless (see
details in #161337). HTTP APIs that will be public in serverless have
been handled in #162523.
**IMPORTANT: This PR leaves login, user, and role routes enabled. The
primary reason for this is due to several testing mechanisms that rely
on basic authentication and custom roles (UI, Cypress). These tests will
be modified to use SAML authentication and serverless roles in the
immediate future. Once this occurs, we will disable these routes.**
### Testing
This PR also implements testing API access in serverless.
- The testing strategy for disabled routes in serverless is to verify a
`404 not found `response.
- The testing strategy for internal access routes in serverless is to
verify that without the internal request header
(`x-elastic-internal-origin`), a `400 bad request response` is received,
then verify that with the internal request header, a `200 ok response`
is received.
- The strategy for public routes in serverless is to verify a `200 ok`
or `203 redirect` is received.
~~blocked by #161930~~
~~blocked by #162149 for test implementation~~
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@elastic.co>
## Summary
It seems beneficial to have feature flag tests in a separate test config
file
- tests are still run on Kibana CI automatically
- tests are not run on MKI projects automatically, but you can deploy
custom project and run tests via feature flags config
All the feature flags within the same project should be places in the
same config to make sure there is no arguments conflict.
When the flag is moved to the yml configuration, we can rely on Kibana
CI and manually triggered deployment to make sure projects are
functioning correctly.
---------
Co-authored-by: Robert Oskamp <robert.oskamp@elastic.co>
## Summary
This PR converts helpers into FTR services that should provide better
test capabilities and improve stability/error output.
Before change:
```
└-> should be active
└-> "before each" hook: global before each for "should be active"
└- ✖ fail: serverless observability API Threshold rule - AVG - PCT - FIRED Rule creation should be active
│ Error: Timeout of 360000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/dmle/github/kibana/x-pack/test_serverless/api_integration/test_suites/observability/threshold_rule/avg_pct_fired.ts)
│ at listOnTimeout (node:internal/timers:559:17)
│ at processTimers (node:internal/timers:502:7)
```
The test is not handling its state properly and fails on mocha timeout
(6 minutes) without proper error message. It leads the test suite
usually taking ~**5** min keep running for over [1
hour](https://buildkite.com/elastic/appex-qa-kibana-serverless-ftr-tests/builds/30#01898d63-a0e1-4cb0-b460-b4824ce2d5b1)
After change:
```
2) serverless observability API
│ Threshold rule - AVG - PCT - FIRED
│ Rule creation
│ should be active:
│
│ Error: 'ruleId' is undefined
│ at Object.waitForRuleStatus (alerting_api.ts:31:15)
│ at Context.apply (avg_pct_fired.ts:122:51)
│ at Object.apply (wrap_function.js:73:30)
```
Since the follow-up tests depend on the first test to success, we can
either move Rule creation to before hook or check for `ruleId` to be
defined. If it is defined, `retry` service will pull for status up to 90
sec (important to keep below Mocha timeout so you can get error) and
fail if it is not matching expected one.
## Summary
This PR adds boilerplate code and a few initial end-to-end tests to
serverless plugins.
Note that the tests defined in this PR are not part of any CI run yet,
this will be done in a follow-up after this PR is merged.
### Details
The serverless test structure corresponds to what we have in
`x-pack/test` with API tests in `api_integration` and UI tests in
`functional`, each with their set of helper methods and sub-directories
for
- `common` functionality shared across serverless projects (core, shared
UX, ...)
- `observability` project specific functionality
- `search` project specific functionality
- `security` project specific functionality
The `shared` directory contains fixtures, services, ... that are shared
across `api_integration` abd `functional` tests.
```
x-pack/test_serverless/
├─ api_integration
│ ├─ services
│ ├─ test_suites
│ │ ├─ common
│ │ ├─ observability
│ │ ├─ search
│ │ ├─ security
├─ functional
│ ├─ page_objects
│ ├─ services
│ ├─ test_suites
│ │ ├─ common
│ │ ├─ observability
│ │ ├─ search
│ │ ├─ security
├─ shared
│ ├─ services
│ ├─ types
```
See also `x-pack/test_serverless/README.md`
### Run tests
Similar to how functional tests are run in `x-pack/test`, you can point
the functional tests server and test runner to config files in this
`x-pack/test_serverless` directory, e.g. from the `x-pack` directory
run:
```
node scripts/functional_tests_server.js --config test_serverless/api_integration/test_suites/common/config.ts
```
and
```
node scripts/functional_test_runner.js --config test_serverless/api_integration/test_suites/common/config.ts
```
### Additional changes
- The stateful `common_page` page object used the existence of the
global nav to determine `isChromeVisible` and `isChromeHidden`, which is
not working when the global nav is disabled. To solve this, a
`data-test-subj` that indicates the chrome visible state is added to the
Kibana app wrapper and is used for the checks.
- Add a few `data-test-subj` entries to the Observability overview page.
- Add optional `dataTestSubj` to the `Navigation` component and use that
for the serverless search nav.
- Add optional `titleDataTestSubj` to the `SolutionNav` component and
use it for the serverless security nav.
- Add a data-test-subj entry to the Search overview page.
