kibana/x-pack/test_serverless

Kibana Serverless Tests

The tests and helper methods (services, page objects) defined here in x-pack/test_serverless cover the serverless functionality introduced by the serverless, serverless_observability, serverless_search and serverless_security plugins.

For how to set up Docker for serverless ES images, please refer to packages/kbn-es/README.

Serverless testing structure and conventions

Overview

The serverless test structure corresponds to what we have in x-pack/test with API tests in api_integration and UI tests in functional, each with their set of helper methods and sub-directories for

• common functionality shared across serverless projects (core, shared UX, ...)
• observability project specific functionality
• search project specific functionality
• security project specific functionality

The shared directory contains fixtures, services, ... that are shared across api_integration abd functional tests.

x-pack/test_serverless/
├─ api_integration
│  ├─ services
│  ├─ test_suites
│  │  ├─ common
│  │  ├─ observability
│  │  ├─ search
│  │  ├─ security
├─ functional
│  ├─ page_objects
│  ├─ services
│  ├─ test_suites
│  │  ├─ common
│  │  ├─ observability
│  │  ├─ search
│  │  ├─ security
├─ shared
│  ├─ services
│  ├─ types

Common tests

As outlined above, tests in the common API integration and functional test suites are covering functionality that's shared across serverless projects. That's why these tests don't have a dedicated config file and instead need to be included in project specific configurations.

If you add a new api_integration or functional common sub-directory, remember to add it to the corresponding common_configs of all projects (x-pack/test_serverless/[api_integration|functional]/test_suites/[observability|search|security]/common_configs).

In case a common test needs to be skipped for one of the projects (in both regular pipelines that start KBN in serverless mode [against serverless ES] & pipelines creating serverless projects in MKI [Cloud]), there are the following suite tags available to do so: skipSvlOblt, skipSvlSearch, skipSvlSec, which can be added like this to a test suite:

describe('my test suite', function () {
  this.tags(['skipSvlOblt', 'skipSvlSearch', 'skipSvlSec']);
  // or for a single tag: this.tags('skipSvlSec');
  [...]
});

Tests that are designed to only run in one of the projects should be added to the project specific test directory and not to common with two skips.

Note, that common tests are invoked three times in a full test run: once per project to make sure the covered shared functionality works correctly in every project. So when writing tests there, be mindful about the test run time.

See also the README files for Serverless Common API Integration Tests and Serverless Common Functional Tests.

Shared services and page objects

Test services and page objects from x-pack/test/[api_integration|functional] are available for reuse.

Serverless specific services and page objects are implemented in x-pack/test_serverless/[api_integration|functional|shared] only and may not be added to or make modifications in x-pack/test.

With this helper method reuse, we have to avoid name clashes and go with the following namespaces:

project	namespace for helper methods
common	svlCommon
observability	svlOblt
search	svlSearch
security	svlSec

Adding Serverless Tests

As outlined above, serverless tests are separated from stateful tests (except the reuse of helper methods), which includes a new base configuration. All tests that should run in a serverless environment have to be added to the x-pack/test_serverless.

Tests in this area should be clearly designed for the serverless environment, particularly when it comes to timing for API requests and UI interaction.

Roles-based testing

Each serverless project has its own set of SAML roles with specfic permissions defined in roles.yml and in oder to properly test Kibana functionality, test design requires to login with a project-supported SAML role. FTR provides svlUserManager service to do SAML authentication, that allows UI tests to set the SAML cookie in the browser context and generates api key to use in the api integration tests. See examples below.

General recommendations:

• use the minimal required role to access tested functionality
• when feature logic depends on both project type & role, make sure to add separate tests
• avoid using basic authentication, unless it is the actual test case
• run the tests against real project(s) on MKI to validate it is stable

Functional UI test example

Recommendations:

• in each test file top level describe suite should start with loginWithRole call in before hook
• no need to log out, you can change role by calling loginWithRole again.
• for the common tests you can use loginWithPrivilegedRole to login as Editor/Developer

describe("my test suite", async function() {
  before(async () => {
    await PageObjects.svlCommonPage.loginWithRole('viewer');
    await esArchiver.load(...);
    await PageObjects.dashboard.navigateToApp();
  });

  it('test step', async() => {
    await PageObjects.dashboard.loadSavedDashboard('old dashboard');
    await PageObjects.dashboard.waitForRenderComplete();
    ...
  });
});

API integration test example

Recommendations:

• in each test file top level describe suite should start with createM2mApiKeyWithRoleScope call in before hook
• don't forget to invalidate api key using invalidateApiKeyWithRoleScope in after hook
• make api calls using supertestWithoutAuth with generated api key header

describe("my test suite", async function() {
    before(async () => {
      roleAuthc = await svlUserManager.createM2mApiKeyWithRoleScope('viewer');
      commonRequestHeader = svlCommonApi.getCommonRequestHeader();
      internalRequestHeader = svlCommonApi.getInternalRequestHeader();
    });

    after(async () => {
      await svlUserManager.invalidateApiKeyWithRoleScope(roleAuthc);
    });

    it(''test step', async () => {
      const { body, status } = await supertestWithoutAuth
        .delete('/api/spaces/space/default')
        .set(commonRequestHeader)
        .set(roleAuthc.apiKeyHeader);
      ...
    });
});

Testing with feature flags

tl;dr: Tests specific to functionality behind a feature flag need special handling and are by default only tested locally / in CI but excluded from regular test runs in MKI.

New features might be gated behind a feature flag and can only be enabled through a yml configuration entry. By default, these features are not enabled so they're not available in a regular serverless MKI project, which would make end-to-end tests for such a feature fail. In order to still have tests for features behind a feature flag, these tests need to be separated from the regular tests.

For every project's test_suites directory, there are feature flags specific config (config.feature_flags.ts) and index (index.feature_flags.ts) files next to the regular config.ts and index.ts. These extra files are used to cover all feature flag tests of the respective area. If you want to add feature flag specific tests:

• Add your feature flag(s) to the kbnServerArgs in the config.feature_flags.ts file
• Load your test file(s) in the index.feature_flags.ts file

As mentioned above, these tests are not part of the regular test run against MKI projects. If you still want to run feature flag tests against an MKI project, this requires a Kibana docker build that has the feature flags enabled by default. This docker image can then be used to create a project in serverless QA and the feature flags tests can be pointed to the project.

Run tests

Similar to how functional tests are run in x-pack/test, you can point the functional tests server and test runner to config files in this x-pack/test_serverless directory, e.g. from the x-pack directory run:

node scripts/functional_tests_server.js --config test_serverless/api_integration/test_suites/search/config.ts

node scripts/functional_test_runner.js --config test_serverless/api_integration/test_suites/search/config.ts

Run tests on MKI

There is no need to start servers locally, you just need to create MKI project and copy urls for Elasticsearch and Kibana. Make sure to update urls with username/password and port 443 for Elasticsearch. FTR has no control over MKI and can't update your projects so make sure your config.ts does not specify any custom arguments for Kibana or Elasticsearch. Otherwise, it will be ignored. You can run the tests from the x-pack directory:

TEST_CLOUD=1 TEST_CLOUD_HOST_NAME="CLOUD_HOST_NAME" TEST_ES_URL="https://elastic:PASSWORD@ES_HOSTNAME:443" TEST_KIBANA_URL="https://elastic:PASSWORD@KIBANA_HOSTNAME" node scripts/functional_test_runner --config test_serverless/api_integration/test_suites/search/config.ts --exclude-tag=skipMKI

Steps to follow to run on QA environment:

• Go to CLOUD_HOST_NAME and create a project.

• Go to CLOUD_HOST_NAME/account/keys and create Cloud specific API Key.

• We need the key from step 2 to obtain basic auth credentials for ES and Kibana. Make a POST request to the following endpoint.

  POST CLOUD_HOST_NAME/api/v1/serverless/projects/<project-type>/<project-id>/_reset-internal-credentials
  Authorization: ApiKey <Cloud-API-key>
  Content-Type: application/json
  

  In response you should get credentials.

  {
    "password": "testing-internal_pwd",
    "username": "testing-internal"
  }
  

  We would use these credentials for TEST_ES_URL="https://USERNAME:PASSWORD@ES_HOSTNAME:443" and TEST_KIBANA_URL="https://USERNAME:PASSWORD@KIBANA_HOSTNAME"

• Now we need to create a user with the roles we want to test. Go to members page - CLOUD_HOST_NAME/account/members and click [Invite member].

  • Select the access level you want to grant and your project type. For example, to create a user with viewer role, toggle [Instanse access], select project (should correspond to your project type, i.e Security), select Viewer role.
  • Create .ftr/role_users.json in the root of Kibana repo. Add record for created user.

    {
      "viewer": {
        "password": "xxxx",
        "email": "email_of_the_elastic_cloud_account"
      }
    }
    

• Now run the tests from the x-pack directory

TEST_CLOUD=1 TEST_CLOUD_HOST_NAME="CLOUD_HOST_NAME" TEST_ES_URL="https://testing-internal:testing-internal_pwd@ES_HOSTNAME:443" TEST_KIBANA_URL="https://testing-internal:testing-internal_pwd@KIBANA_HOSTNAME:443" node scripts/functional_test_runner.js --config test_serverless/functional/test_suites/security/common_configs/config.group1.ts --exclude-tag=skipMKI

Skipping tests for MKI run

The tests that are listed in the the regular config.ts generally should work in both Kibana CI and MKI. However some tests might not work properly against MKI projects by design. Tag the tests with skipMKI to be excluded for MKI run. It works only for the describe block:

describe("my test suite", async function() {
    this.tags(['skipMKI']);
    ...
});

If you are running tests from your local against MKI projects, make sure to add --exclude-tag=skipMKI to your FTR command.