[Security][Serverless] Add Product types in FTR API Integration tests. (#184309)

# Pull Request Overview
This pull request (PR):
1. Enables Product Types for FTR API Integration Test Suites in
Serverless MKI:
- Previously, the test suites ignored product types entirely. With this
PR, scripts to run the tests have been relocated to
x-pack/test/security_solution_api_integration/scripts.
- Users can now run tests from the API Integration tests directory for
security solutions by using the command:
TARGET_SCRIPT={script_from_package_json} node
./scripts/mki_start_api_ftr_execution. This will execute the following
steps:
1. Create a security serverless project, respecting the product types
specified in the serverless configuration found in the config folder of
the relevant test suite.
      2. Reset credentials.
3. Wait for Elasticsearch (ES) and Kibana to be ready and available.
      4. Execute the tests.
      5. Delete the project upon completion.
2. Adds Proxy Services Organizations to .ftr Role Users Files:
- This PR updates the .ftr role_users files to include all proxy
services organizations, ensuring they have the necessary permissions for
the tests.

# Implementation Details
Product Types

  - Previous Setup:
- A project was created and handed over to the test suite to run the API
tests against, without considering product types.
  - Changes Introduced:
- The script execution for tests has moved to
x-pack/test/security_solution_api_integration/scripts.
- Tests can be initiated using the command from the API Integration
tests folder where package.json exists:
      ```
TARGET_SCRIPT={script_from_package_json} node
./scripts/mki_start_api_ftr_execution
      ```
- The mki_start_api_ftr_execution script performs several steps to run
the tests, including creating a security serverless project with
specified product types, resetting credentials, ensuring ES and Kibana
readiness, executing tests, and cleaning up the project afterward.
- The script reads extra configuration (currently only product types are
supported) in the specific format as is, from the following file :
[api_configs.json](https://github.com/elastic/kibana/pull/184309/files#diff-1122baffe7ff843b1f486cee95468bed5851a9a4934be747f540bd42dc9a07daR2).
The key for the JSON file is the name of the script in
[package.json](https://github.com/elastic/kibana/pull/184309/files#diff-c6af1c81947b3a77bed431c688c7ad38c8969bd52e1c3ea92d643f09d422eb61R296)
- If a specific configuration is not required and the default complete
project is needed for the test to run, the key and configuration in the
`api_configs.json` file can be ommitted.

# Summary

This PR enhances the flexibility and functionality of the FTR API
integration test suites for serverless MKI by incorporating product type
considerations and updating the role_users configuration to include
proxy services organizations.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This commit is contained in:
dkirchan 2024-06-20 17:30:35 +03:00 committed by GitHub
parent 7e19cc5660
commit 6ce61db2ff
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 272 additions and 33 deletions

View file

@ -15,7 +15,11 @@ buildkite-agent meta-data set "${BUILDKITE_JOB_ID}_is_test_execution_step" "true
source .buildkite/scripts/pipelines/security_solution_quality_gate/prepare_vault_entries.sh
echo "--- Running test script $1"
TARGET_SCRIPT=$1 node .buildkite/scripts/pipelines/security_solution_quality_gate/api_integration/start_api_ftr_execution
cd x-pack/test/security_solution_api_integration
set +e
TARGET_SCRIPT=$1 node ./scripts/mki_start_api_ftr_execution
cmd_status=$?
echo "Exit code with status: $cmd_status"
exit $cmd_status

View file

@ -1,10 +0,0 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/
require('../../../../../src/setup_node_env');
require('./api_ftr_execution').cli();

View file

@ -8,6 +8,13 @@ vault_get security-quality-gate/role-users data -format=json > .ftr/role_users.j
vault_get security-quality-gate/role-users/sec-sol-auto-01 data -format=json > .ftr/sec-sol-auto-01.json
vault_get security-quality-gate/role-users/sec-sol-auto-02 data -format=json > .ftr/sec-sol-auto-02.json
vault_get security-quality-gate/role-users/sec-sol-auto-03 data -format=json > .ftr/sec-sol-auto-03.json
vault_get security-quality-gate/role-users/sec-sol-auto-04 data -format=json > .ftr/sec-sol-auto-04.json
vault_get security-quality-gate/role-users/sec-sol-auto-05 data -format=json > .ftr/sec-sol-auto-05.json
vault_get security-quality-gate/role-users/sec-sol-auto-06 data -format=json > .ftr/sec-sol-auto-06.json
vault_get security-quality-gate/role-users/sec-sol-auto-07 data -format=json > .ftr/sec-sol-auto-07.json
vault_get security-quality-gate/role-users/sec-sol-auto-08 data -format=json > .ftr/sec-sol-auto-08.json
vault_get security-quality-gate/role-users/sec-sol-auto-09 data -format=json > .ftr/sec-sol-auto-09.json
vault_get security-quality-gate/role-users/sec-sol-auto-10 data -format=json > .ftr/sec-sol-auto-10.json
# The vault entries relevant to QA Cloud
export CLOUD_QA_API_KEY=$(vault_get security-solution-quality-gate qa_api_key)

View file

@ -47,6 +47,13 @@ ex:
3. In these new configuration files, include references to the base configurations located under the config directory to inherit CI configurations, environment variables, and other settings.
4. Append a new entry in the `ftr_configs.yml` file to enable the execution of the newly added tests within the CI pipeline.
## Adding tests for MKI which rely onto NON default project configuration
The default project type configuration in Serverless is complete. If for the needs of a test suite a different configuration is required, e.g. [PLI - Essentials](https://github.com/elastic/kibana/blob/36578e82fa0a0440c1657a0ca688106c895d5e4e/x-pack/test/security_solution_api_integration/test_suites/entity_analytics/risk_engine/basic_license_essentials_tier/configs/serverless.config.ts#L13), the already mentioned configuration in the permalink **does not work** for MKI. The override is needed to be added in the `./scripts/api_configs.json` file under the key with exact same name as the one of the script in `package.json` file which is running.
There are already configurations in the `./scripts/api_configs.json` which you can follow in order to add yours when it is needed. The currently supported configuration, allows **ONLY** the PLIs to be configured. Thus, experimental feature flags **are not yet supported** and the test should be skipped until further notice.
**NOTE**: If a target script living in `package.json` file, does not require any further configuration, then the entry in `./scripts/api_configs.json` file, **can be omitted!**
# Testing locally

View file

@ -297,11 +297,11 @@
"rule_read:basic:server:ess": "npm run initialize-server:rm:basic_essentials rule_read ess",
"rule_read:basic:runner:ess": "npm run run-tests:rm:basic_essentials rule_read ess essEnv",
"rules_management:essentials:server:serverless": "npm run initialize-server:rm:basic_essentials rule_management serverless",
"rules_management:essentials:runner:serverless": "npm run run-tests:rm:basic_essentials rule_management serverless serverlessEnv",
"rules_management:essentials:qa:serverless": "npm run run-tests:rm:basic_essentials rule_management serverless qaPeriodicEnv",
"rules_management:essentials:qa:serverless:release": "npm run run-tests:rm:basic_essentials rule_management serverless qaEnv",
"rules_management:basic:server:ess": "npm run initialize-server:rm:basic_essentials rule_management ess",
"rules_management:basic:runner:ess": "npm run run-tests:rm:basic_essentials rule_management ess essEnv"
"rules_management:essentials:server:serverless": "npm run initialize-server:rm:basic_essentials rule_management serverless",
"rules_management:essentials:runner:serverless": "npm run run-tests:rm:basic_essentials rule_management serverless serverlessEnv",
"rules_management:essentials:qa:serverless": "npm run run-tests:rm:basic_essentials rule_management serverless qaPeriodicEnv",
"rules_management:essentials:qa:serverless:release": "npm run run-tests:rm:basic_essentials rule_management serverless qaEnv",
"rules_management:basic:server:ess": "npm run initialize-server:rm:basic_essentials rule_management ess",
"rules_management:basic:runner:ess": "npm run run-tests:rm:basic_essentials rule_management ess essEnv"
}
}

View file

@ -0,0 +1,198 @@
{
"nlp_cleanup_task:essentials:qa:serverless": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" },
{ "product_line": "cloud", "product_tier": "essentials" }
]
},
"nlp_cleanup_task:essentials:qa:serverless:release": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" },
{ "product_line": "cloud", "product_tier": "essentials" }
]
},
"entity_analytics:essentials:qa:serverless": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" },
{ "product_line": "cloud", "product_tier": "essentials" }
]
},
"entity_analytics:essentials:qa:serverless:release": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" },
{ "product_line": "cloud", "product_tier": "essentials" }
]
},
"exception_workflows:essentials:qa:serverless": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"exception_workflows:essentials:qa:serverless:release": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"exception_operators_date_numeric_types:essentials:qa:serverless": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"exception_operators_date_numeric_types:essentials:qa:serverless:release": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"exception_operators_keyword:essentials:qa:serverless": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"exception_operators_keyword:essentials:qa:serverless:release": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"exception_operators_ips:essentials:qa:serverless": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"exception_operators_ips:essentials:qa:serverless:release": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"exception_operators_long:essentials:qa:serverless": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"exception_operators_long:essentials:qa:serverless:release": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"exception_operators_text:essentials:qa:serverless": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"exception_operators_text:essentials:qa:serverless:release": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"alerts:essentials:qa:serverless": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"alerts:essentials:qa:serverless:release": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"rule_creation:essentials:qa:serverless": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"rule_creation:essentials:qa:serverless:release": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"rule_update:essentials:qa:serverless": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"rule_update:essentials:qa:serverless:release": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"rule_patch:essentials:qa:serverless": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"rule_patch:essentials:qa:serverless:release": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"rule_delete:essentials:qa:serverless": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"rule_delete:essentials:qa:serverless:release": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"rule_import_export:essentials:qa:serverless": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"rule_import_export:essentials:qa:serverless:release": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"rules_management:essentials:qa:serverless": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"rules_management:essentials:qa:serverless:release": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"rule_read:essentials:qa:serverless": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
},
"rule_read:essentials:qa:serverless:release": {
"productTypes": [
{ "product_line": "security", "product_tier": "essentials" },
{ "product_line": "endpoint", "product_tier": "essentials" }
]
}
}

View file

@ -1,17 +1,19 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { run } from '@kbn/dev-cli-runner';
import { ToolingLog } from '@kbn/tooling-log';
import { exec } from 'child_process';
import crypto from 'crypto';
import type { ProjectHandler } from '@kbn/security-solution-plugin/scripts/run_cypress/project_handler/project_handler';
import fs from 'fs';
import type {
ProductType,
ProjectHandler,
} from '@kbn/security-solution-plugin/scripts/run_cypress/project_handler/project_handler';
import { CloudHandler } from '@kbn/security-solution-plugin/scripts/run_cypress/project_handler/cloud_project_handler';
import { ProxyHandler } from '@kbn/security-solution-plugin/scripts/run_cypress/project_handler/proxy_project_handler';
import {
@ -25,28 +27,33 @@ const BASE_ENV_URL = `${process.env.QA_CONSOLE_URL}`;
const PROJECT_NAME_PREFIX = 'kibana-ftr-api-integration-security-solution';
// Function to execute a command and return a Promise with the status code
function executeCommand(command: string, envVars: any, workDir: string): Promise<number> {
function executeCommand(
command: string,
envVars: any,
log: ToolingLog,
workDir?: string
): Promise<number> {
return new Promise((resolve, reject) => {
const childProcess = exec(command, { env: envVars, cwd: workDir }, (error, stdout, stderr) => {
const childProcess = exec(command, { env: envVars }, (error, stdout, stderr) => {
if (error) {
console.error(`exec error: ${error}`);
log.error(`exec error: ${error}`);
process.exitCode = error.code;
}
});
// Listen and print stdout data
childProcess.stdout?.on('data', (data) => {
console.log(data);
log.info(data);
});
// Listen and print stderr data
childProcess.stderr?.on('data', (data) => {
console.log(data);
log.info(data);
});
// Listen for process exit
childProcess.on('exit', (code) => {
console.log(`Node process for target ${process.env.TARGET_SCRIPT} exits with code : ${code}`);
log.info(`Node process for target ${process.env.TARGET_SCRIPT} exits with code : ${code}`);
if (code !== 0) {
reject(code);
return;
@ -56,6 +63,24 @@ function executeCommand(command: string, envVars: any, workDir: string): Promise
});
}
async function parseProductTypes(log: ToolingLog): Promise<ProductType[] | undefined> {
if (!process.env.TARGET_SCRIPT) {
log.error('TARGET_SCRIPT environment variable is not provided. Aborting...');
return process.exit(1);
}
const apiConfigs = JSON.parse(await fs.promises.readFile('./scripts/api_configs.json', 'utf8'));
try {
const productTypes: ProductType[] = apiConfigs[process.env.TARGET_SCRIPT]
.productTypes as ProductType[];
return productTypes && productTypes.length > 0 ? productTypes : undefined;
} catch (err) {
// If the configuration for the script is not needed, it can be omitted from the json file.
log.warning(`Extended configuration was not found for script : ${process.env.TARGET_SCRIPT}`);
return undefined;
}
}
export const cli = () => {
run(
async (context) => {
@ -89,14 +114,13 @@ export const cli = () => {
const id = crypto.randomBytes(8).toString('hex');
const PROJECT_NAME = `${PROJECT_NAME_PREFIX}-${id}`;
const productTypes = await parseProductTypes(log);
// Creating project for the test to run
const project = await cloudHandler.createSecurityProject(PROJECT_NAME);
log.info(project);
const project = await cloudHandler.createSecurityProject(PROJECT_NAME, productTypes);
if (!project) {
log.error('Failed to create project.');
return process.exit(1);
}
let statusCode: number = 0;
@ -132,7 +156,6 @@ export const cli = () => {
const testCloud = 1;
const testEsUrl = `https://${credentials.username}:${credentials.password}@${FORMATTED_ES_URL}`;
const testKibanaUrl = `https://${credentials.username}:${credentials.password}@${FORMATTED_KB_URL}`;
const workDir = 'x-pack/test/security_solution_api_integration';
const envVars = {
...process.env,
TEST_CLOUD: testCloud.toString(),
@ -140,7 +163,7 @@ export const cli = () => {
TEST_KIBANA_URL: testKibanaUrl,
};
statusCode = await executeCommand(command, envVars, workDir);
statusCode = await executeCommand(command, envVars, log);
} catch (err) {
log.error('An error occured when running the test script.');
log.error(err.message);

View file

@ -0,0 +1,9 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
require('../../../../src/setup_node_env');
require('./mki_api_ftr_execution').cli();

View file

@ -47,5 +47,6 @@
"@kbn/timelines-plugin",
"@kbn/ftr-common-functional-ui-services",
"@kbn/test-subj-selector",
"@kbn/dev-cli-runner",
]
}