mirror of
https://github.com/elastic/kibana.git
synced 2025-06-28 11:05:39 -04:00
85c130c141
# Backport This will backport the following commits from `main` to `9.0`: - [[Security Assistant] Adds BuildKite pipeline for running Security GenAI Evaluations weekly (#215254)](https://github.com/elastic/kibana/pull/215254) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Garrett Spong","email":"spong@users.noreply.github.com"},"sourceCommit":{"committedDate":"2025-04-24T17:46:57Z","message":"[Security Assistant] Adds BuildKite pipeline for running Security GenAI Evaluations weekly (#215254)\n\n## Summary\n\nIntroduces a new `security_solution/gen_ai_evals.yml` BuildKite pipeline\nfor automatically running our Assistant and Attack Discovery evaluation\nsuites weekly.\n\n### To Run Locally:\nEnsure you are authenticated with vault for LLM + LangSmith creds:\n\n> See [internal\ndocs](https://github.com/elastic/infra/blob/master/docs/vault/README.md#login-with-your-okta)\nfor setup/login instructions.\n\nFetch Connectors and LangSmith creds:\n\n> [!NOTE]\n> In discussion with @elastic/kibana-operations it was preferred to use\nthe ci-prod secrets vault, so we cannot self-manage the secrets. To test\nthis locally though, you can grab the secrets and follow the\ninstructions in this [paste\nbin](https://p.elstc.co/paste/q7k+zYOc#PN0kasw11u2J0XWC2Ls5PMNWreKzKTpgWA1wtsPzeH+).\n\n```\ncd x-pack/test/security_solution_api_integration\nnode scripts/genai/vault/retrieve_secrets.js \n```\n\n\nNavigate to api integration directory, load the env vars, and start\nserver:\n```\ncd x-pack/test/security_solution_api_integration\nexport KIBANA_SECURITY_TESTING_AI_CONNECTORS=$(base64 -w 0 < scripts/genai/vault/connector_config.json) && export KIBANA_SECURITY_TESTING_LANGSMITH_KEY=$(base64 -w 0 < scripts/genai/vault/langsmith_key.txt)\nyarn genai_evals:server:ess\n```\n\nThen in another terminal, load vars and run the tests:\n```\ncd x-pack/test/security_solution_api_integration\nexport KIBANA_SECURITY_TESTING_AI_CONNECTORS=$(base64 -w 0 < scripts/genai/vault/connector_config.json) && export KIBANA_SECURITY_TESTING_LANGSMITH_KEY=$(base64 -w 0 < scripts/genai/vault/langsmith_key.txt)\nyarn genai_evals🏃ess\n```\n\n### To manually run on BuildKite:\nNavigate to\n[BuildKite](https://buildkite.com/elastic?filter=ftr-security-solution-gen-ai-evaluations)\nand run `ftr-security-solution-gen-ai-evaluations` pipeline.\n\n### To manually run on BuildKite for specific PR:\nIn `.buildkite/ftr_security_stateful_configs.yml`, temporarily move the\n`genai/evaluations/trial_license_complete_tier/configs/ess.config.ts`\nline down to the `enabled` section. Will see if we can do this without\nrequiring a commit. @elastic/kibana-operations is it possible to set a\nbuildkite env var that can be read in FTR tests when a specific GitHub\nlabel is added to the PR? I.e. can I create a `SecurityGenAI:Run Evals`\nlabel that when added will run this suite as part of the build?\n\n> [!NOTE]\n> Currently the connectors secrets only include `gpt-4o` and\n`gpt-4o-mini`. Waiting on finalized list w/ credentials from @jamesspi\nand @peluja1012 and then we can have ops update using the scripts\nincluded in this PR.\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Patryk Kopycinski <patryk.kopycinski@elastic.co>","sha":"e57663a0cf5a1692a5be10413c1d745dd260a24f","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Security Generative AI","Feature:Assistant Evaluation","backport:version","v9.1.0","v8.19.0","v9.0.1","ci:security-genai-run-evals"],"title":"[Security Assistant] Adds BuildKite pipeline for running Security GenAI Evaluations weekly","number":215254,"url":"https://github.com/elastic/kibana/pull/215254","mergeCommit":{"message":"[Security Assistant] Adds BuildKite pipeline for running Security GenAI Evaluations weekly (#215254)\n\n## Summary\n\nIntroduces a new `security_solution/gen_ai_evals.yml` BuildKite pipeline\nfor automatically running our Assistant and Attack Discovery evaluation\nsuites weekly.\n\n### To Run Locally:\nEnsure you are authenticated with vault for LLM + LangSmith creds:\n\n> See [internal\ndocs](https://github.com/elastic/infra/blob/master/docs/vault/README.md#login-with-your-okta)\nfor setup/login instructions.\n\nFetch Connectors and LangSmith creds:\n\n> [!NOTE]\n> In discussion with @elastic/kibana-operations it was preferred to use\nthe ci-prod secrets vault, so we cannot self-manage the secrets. To test\nthis locally though, you can grab the secrets and follow the\ninstructions in this [paste\nbin](https://p.elstc.co/paste/q7k+zYOc#PN0kasw11u2J0XWC2Ls5PMNWreKzKTpgWA1wtsPzeH+).\n\n```\ncd x-pack/test/security_solution_api_integration\nnode scripts/genai/vault/retrieve_secrets.js \n```\n\n\nNavigate to api integration directory, load the env vars, and start\nserver:\n```\ncd x-pack/test/security_solution_api_integration\nexport KIBANA_SECURITY_TESTING_AI_CONNECTORS=$(base64 -w 0 < scripts/genai/vault/connector_config.json) && export KIBANA_SECURITY_TESTING_LANGSMITH_KEY=$(base64 -w 0 < scripts/genai/vault/langsmith_key.txt)\nyarn genai_evals:server:ess\n```\n\nThen in another terminal, load vars and run the tests:\n```\ncd x-pack/test/security_solution_api_integration\nexport KIBANA_SECURITY_TESTING_AI_CONNECTORS=$(base64 -w 0 < scripts/genai/vault/connector_config.json) && export KIBANA_SECURITY_TESTING_LANGSMITH_KEY=$(base64 -w 0 < scripts/genai/vault/langsmith_key.txt)\nyarn genai_evals🏃ess\n```\n\n### To manually run on BuildKite:\nNavigate to\n[BuildKite](https://buildkite.com/elastic?filter=ftr-security-solution-gen-ai-evaluations)\nand run `ftr-security-solution-gen-ai-evaluations` pipeline.\n\n### To manually run on BuildKite for specific PR:\nIn `.buildkite/ftr_security_stateful_configs.yml`, temporarily move the\n`genai/evaluations/trial_license_complete_tier/configs/ess.config.ts`\nline down to the `enabled` section. Will see if we can do this without\nrequiring a commit. @elastic/kibana-operations is it possible to set a\nbuildkite env var that can be read in FTR tests when a specific GitHub\nlabel is added to the PR? I.e. can I create a `SecurityGenAI:Run Evals`\nlabel that when added will run this suite as part of the build?\n\n> [!NOTE]\n> Currently the connectors secrets only include `gpt-4o` and\n`gpt-4o-mini`. Waiting on finalized list w/ credentials from @jamesspi\nand @peluja1012 and then we can have ops update using the scripts\nincluded in this PR.\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Patryk Kopycinski <patryk.kopycinski@elastic.co>","sha":"e57663a0cf5a1692a5be10413c1d745dd260a24f"}},"sourceBranch":"main","suggestedTargetBranches":["9.0"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/215254","number":215254,"mergeCommit":{"message":"[Security Assistant] Adds BuildKite pipeline for running Security GenAI Evaluations weekly (#215254)\n\n## Summary\n\nIntroduces a new `security_solution/gen_ai_evals.yml` BuildKite pipeline\nfor automatically running our Assistant and Attack Discovery evaluation\nsuites weekly.\n\n### To Run Locally:\nEnsure you are authenticated with vault for LLM + LangSmith creds:\n\n> See [internal\ndocs](https://github.com/elastic/infra/blob/master/docs/vault/README.md#login-with-your-okta)\nfor setup/login instructions.\n\nFetch Connectors and LangSmith creds:\n\n> [!NOTE]\n> In discussion with @elastic/kibana-operations it was preferred to use\nthe ci-prod secrets vault, so we cannot self-manage the secrets. To test\nthis locally though, you can grab the secrets and follow the\ninstructions in this [paste\nbin](https://p.elstc.co/paste/q7k+zYOc#PN0kasw11u2J0XWC2Ls5PMNWreKzKTpgWA1wtsPzeH+).\n\n```\ncd x-pack/test/security_solution_api_integration\nnode scripts/genai/vault/retrieve_secrets.js \n```\n\n\nNavigate to api integration directory, load the env vars, and start\nserver:\n```\ncd x-pack/test/security_solution_api_integration\nexport KIBANA_SECURITY_TESTING_AI_CONNECTORS=$(base64 -w 0 < scripts/genai/vault/connector_config.json) && export KIBANA_SECURITY_TESTING_LANGSMITH_KEY=$(base64 -w 0 < scripts/genai/vault/langsmith_key.txt)\nyarn genai_evals:server:ess\n```\n\nThen in another terminal, load vars and run the tests:\n```\ncd x-pack/test/security_solution_api_integration\nexport KIBANA_SECURITY_TESTING_AI_CONNECTORS=$(base64 -w 0 < scripts/genai/vault/connector_config.json) && export KIBANA_SECURITY_TESTING_LANGSMITH_KEY=$(base64 -w 0 < scripts/genai/vault/langsmith_key.txt)\nyarn genai_evals🏃ess\n```\n\n### To manually run on BuildKite:\nNavigate to\n[BuildKite](https://buildkite.com/elastic?filter=ftr-security-solution-gen-ai-evaluations)\nand run `ftr-security-solution-gen-ai-evaluations` pipeline.\n\n### To manually run on BuildKite for specific PR:\nIn `.buildkite/ftr_security_stateful_configs.yml`, temporarily move the\n`genai/evaluations/trial_license_complete_tier/configs/ess.config.ts`\nline down to the `enabled` section. Will see if we can do this without\nrequiring a commit. @elastic/kibana-operations is it possible to set a\nbuildkite env var that can be read in FTR tests when a specific GitHub\nlabel is added to the PR? I.e. can I create a `SecurityGenAI:Run Evals`\nlabel that when added will run this suite as part of the build?\n\n> [!NOTE]\n> Currently the connectors secrets only include `gpt-4o` and\n`gpt-4o-mini`. Waiting on finalized list w/ credentials from @jamesspi\nand @peluja1012 and then we can have ops update using the scripts\nincluded in this PR.\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Patryk Kopycinski <patryk.kopycinski@elastic.co>","sha":"e57663a0cf5a1692a5be10413c1d745dd260a24f"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/219186","number":219186,"state":"MERGED","mergeCommit":{"sha":"4fb0f9b9918cda00ea3af865e1cd1a71bcde1946","message":"[8.19] [Security Assistant] Adds BuildKite pipeline for running Security GenAI Evaluations weekly (#215254) (#219186)\n\n# Backport\n\nThis will backport the following commits from `main` to `8.19`:\n- [[Security Assistant] Adds BuildKite pipeline for running Security\nGenAI Evaluations weekly\n(#215254)](https://github.com/elastic/kibana/pull/215254)\n\n\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sorenlouv/backport)\n\n\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>"}},{"branch":"9.0","label":"v9.0.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
116 lines
No EOL
3 KiB
Text
116 lines
No EOL
3 KiB
Text
---
|
|
id: kibDevTutorialCI
|
|
slug: /kibana-dev-docs/tutorials/ci
|
|
title: CI
|
|
description: CI
|
|
date: 2023-11-08
|
|
tags: ['kibana', 'onboarding', 'dev', 'ci']
|
|
---
|
|
|
|
## CI
|
|
|
|
Kibana uses BuildKite to run a series of checks against each pull requests and tracked branch. Results are posted in pull requests as comments and from the BuildKite UI
|
|
|
|
### Comments
|
|
|
|
Comments in pull requests can be used to trigger CI operations.
|
|
|
|
#### `buildkite test this`
|
|
|
|
Run test suites and checks.
|
|
|
|
#### `@elasticmachine merge upstream`
|
|
|
|
Merge in the most recent changes from upstream.
|
|
|
|
#### `@elasticmachine run elasticsearch-ci/docs`
|
|
|
|
Build documentation from the root `docs` folder.
|
|
|
|
### Labels
|
|
|
|
Labels can be added to a pull request to run conditional pipelines. Build artifacts will be available on the "Artifacts" tab of the "Build Kibana Distribution and Plugins" step.
|
|
|
|
#### `ci:all-cypress-suites`
|
|
|
|
Some Cypress test suites are only run when code changes are made in certain files, typically files with overlapping test coverage. Adding this label will cause all Cypress tests to run.
|
|
|
|
#### `ci:build-all-platforms`
|
|
|
|
Build Windows, macOS, and Linux archives.
|
|
|
|
#### `ci:build-canvas-shareable-runtime`
|
|
|
|
Build the Canvas shareable runtime and include it in the distribution.
|
|
|
|
#### `ci:build-cdn-assets`
|
|
|
|
Build an archive that can be used to serve Kibana's static assets.
|
|
|
|
#### `ci:build-cloud-image`
|
|
|
|
Build cloud Docker images that can be used for testing deployments on Elastic Cloud.
|
|
|
|
#### `ci:build-cloud-fips-image`
|
|
|
|
Build FIPS cloud Docker images that can be used for testing deployments on Elastic Cloud.
|
|
|
|
#### `ci:build-docker-fips`
|
|
|
|
Build Docker Wolfi image with FIPS enabled.
|
|
|
|
#### `ci:build-os-packages`
|
|
|
|
Build Docker images, and Debian and RPM packages.
|
|
|
|
#### `ci:build-serverless-image`
|
|
|
|
Build serverless Docker images that can be used for testing deployments on Elastic Cloud.
|
|
|
|
#### `ci:build-storybooks`
|
|
|
|
Build and upload storybooks.
|
|
|
|
#### `ci:build-webpack-bundle-analyzer`
|
|
|
|
Build and upload a bundle report generated by `webpack-bundle-analyzer`.
|
|
|
|
#### `ci:cloud-deploy`
|
|
|
|
Create or update a deployment on Elastic Cloud production.
|
|
|
|
#### `ci:cloud-persist-deployment`
|
|
|
|
Prevents an existing deployment from being shutdown due to inactivity.
|
|
|
|
#### `ci:cloud-redeploy`
|
|
|
|
Create a new deployment on Elastic Cloud. Previous deployments linked to a pull request will be shutdown and data will not be preserved.
|
|
|
|
#### `ci:collect-apm`
|
|
|
|
Collect APM metrics, available for viewing on the Kibana CI APM cluster.
|
|
|
|
#### `ci:no-auto-commit`
|
|
|
|
Skip auto-committing changed files.
|
|
|
|
#### `ci:project-deploy-elasticsearch`
|
|
|
|
Create or update a serverless Elasticsearch project on Elastic Cloud QA.
|
|
|
|
#### `ci:project-deploy-observability`
|
|
|
|
Create or update a serverless Observability project on Elastic Cloud QA.
|
|
|
|
#### `ci:project-deploy-security`
|
|
|
|
Create or update a serverless Security project on Elastic Cloud QA.
|
|
|
|
#### `ci:project-persist-deployment`
|
|
|
|
Prevents an existing deployment from being shutdown due to inactivity.
|
|
|
|
#### `ci:security-genai-run-evals`
|
|
|
|
Run evaluations for the GenAI security evaluation suite. |