mirror of
https://github.com/elastic/kibana.git
synced 2025-06-27 18:51:07 -04:00
e57663a0cf
## Summary Introduces a new `security_solution/gen_ai_evals.yml` BuildKite pipeline for automatically running our Assistant and Attack Discovery evaluation suites weekly. ### To Run Locally: Ensure you are authenticated with vault for LLM + LangSmith creds: > See [internal docs](https://github.com/elastic/infra/blob/master/docs/vault/README.md#login-with-your-okta) for setup/login instructions. Fetch Connectors and LangSmith creds: > [!NOTE] > In discussion with @elastic/kibana-operations it was preferred to use the ci-prod secrets vault, so we cannot self-manage the secrets. To test this locally though, you can grab the secrets and follow the instructions in this [paste bin](https://p.elstc.co/paste/q7k+zYOc#PN0kasw11u2J0XWC2Ls5PMNWreKzKTpgWA1wtsPzeH+). ``` cd x-pack/test/security_solution_api_integration node scripts/genai/vault/retrieve_secrets.js ``` Navigate to api integration directory, load the env vars, and start server: ``` cd x-pack/test/security_solution_api_integration export KIBANA_SECURITY_TESTING_AI_CONNECTORS=$(base64 -w 0 < scripts/genai/vault/connector_config.json) && export KIBANA_SECURITY_TESTING_LANGSMITH_KEY=$(base64 -w 0 < scripts/genai/vault/langsmith_key.txt) yarn genai_evals:server:ess ``` Then in another terminal, load vars and run the tests: ``` cd x-pack/test/security_solution_api_integration export KIBANA_SECURITY_TESTING_AI_CONNECTORS=$(base64 -w 0 < scripts/genai/vault/connector_config.json) && export KIBANA_SECURITY_TESTING_LANGSMITH_KEY=$(base64 -w 0 < scripts/genai/vault/langsmith_key.txt) yarn genai_evals🏃ess ``` ### To manually run on BuildKite: Navigate to [BuildKite](https://buildkite.com/elastic?filter=ftr-security-solution-gen-ai-evaluations) and run `ftr-security-solution-gen-ai-evaluations` pipeline. ### To manually run on BuildKite for specific PR: In `.buildkite/ftr_security_stateful_configs.yml`, temporarily move the `genai/evaluations/trial_license_complete_tier/configs/ess.config.ts` line down to the `enabled` section. Will see if we can do this without requiring a commit. @elastic/kibana-operations is it possible to set a buildkite env var that can be read in FTR tests when a specific GitHub label is added to the PR? I.e. can I create a `SecurityGenAI:Run Evals` label that when added will run this suite as part of the build? > [!NOTE] > Currently the connectors secrets only include `gpt-4o` and `gpt-4o-mini`. Waiting on finalized list w/ credentials from @jamesspi and @peluja1012 and then we can have ops update using the scripts included in this PR. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Patryk Kopycinski <patryk.kopycinski@elastic.co>
116 lines
No EOL
3 KiB
Text
116 lines
No EOL
3 KiB
Text
---
|
|
id: kibDevTutorialCI
|
|
slug: /kibana-dev-docs/tutorials/ci
|
|
title: CI
|
|
description: CI
|
|
date: 2023-11-08
|
|
tags: ['kibana', 'onboarding', 'dev', 'ci']
|
|
---
|
|
|
|
## CI
|
|
|
|
Kibana uses BuildKite to run a series of checks against each pull requests and tracked branch. Results are posted in pull requests as comments and from the BuildKite UI
|
|
|
|
### Comments
|
|
|
|
Comments in pull requests can be used to trigger CI operations.
|
|
|
|
#### `buildkite test this`
|
|
|
|
Run test suites and checks.
|
|
|
|
#### `@elasticmachine merge upstream`
|
|
|
|
Merge in the most recent changes from upstream.
|
|
|
|
#### `@elasticmachine run elasticsearch-ci/docs`
|
|
|
|
Build documentation from the root `docs` folder.
|
|
|
|
### Labels
|
|
|
|
Labels can be added to a pull request to run conditional pipelines. Build artifacts will be available on the "Artifacts" tab of the "Build Kibana Distribution and Plugins" step.
|
|
|
|
#### `ci:all-cypress-suites`
|
|
|
|
Some Cypress test suites are only run when code changes are made in certain files, typically files with overlapping test coverage. Adding this label will cause all Cypress tests to run.
|
|
|
|
#### `ci:build-all-platforms`
|
|
|
|
Build Windows, macOS, and Linux archives.
|
|
|
|
#### `ci:build-canvas-shareable-runtime`
|
|
|
|
Build the Canvas shareable runtime and include it in the distribution.
|
|
|
|
#### `ci:build-cdn-assets`
|
|
|
|
Build an archive that can be used to serve Kibana's static assets.
|
|
|
|
#### `ci:build-cloud-image`
|
|
|
|
Build cloud Docker images that can be used for testing deployments on Elastic Cloud.
|
|
|
|
#### `ci:build-cloud-fips-image`
|
|
|
|
Build FIPS cloud Docker images that can be used for testing deployments on Elastic Cloud.
|
|
|
|
#### `ci:build-docker-fips`
|
|
|
|
Build Docker Wolfi image with FIPS enabled.
|
|
|
|
#### `ci:build-os-packages`
|
|
|
|
Build Docker images, and Debian and RPM packages.
|
|
|
|
#### `ci:build-serverless-image`
|
|
|
|
Build serverless Docker images that can be used for testing deployments on Elastic Cloud.
|
|
|
|
#### `ci:build-storybooks`
|
|
|
|
Build and upload storybooks.
|
|
|
|
#### `ci:build-webpack-bundle-analyzer`
|
|
|
|
Build and upload a bundle report generated by `webpack-bundle-analyzer`.
|
|
|
|
#### `ci:cloud-deploy`
|
|
|
|
Create or update a deployment on Elastic Cloud production.
|
|
|
|
#### `ci:cloud-persist-deployment`
|
|
|
|
Prevents an existing deployment from being shutdown due to inactivity.
|
|
|
|
#### `ci:cloud-redeploy`
|
|
|
|
Create a new deployment on Elastic Cloud. Previous deployments linked to a pull request will be shutdown and data will not be preserved.
|
|
|
|
#### `ci:collect-apm`
|
|
|
|
Collect APM metrics, available for viewing on the Kibana CI APM cluster.
|
|
|
|
#### `ci:no-auto-commit`
|
|
|
|
Skip auto-committing changed files.
|
|
|
|
#### `ci:project-deploy-elasticsearch`
|
|
|
|
Create or update a serverless Elasticsearch project on Elastic Cloud QA.
|
|
|
|
#### `ci:project-deploy-observability`
|
|
|
|
Create or update a serverless Observability project on Elastic Cloud QA.
|
|
|
|
#### `ci:project-deploy-security`
|
|
|
|
Create or update a serverless Security project on Elastic Cloud QA.
|
|
|
|
#### `ci:project-persist-deployment`
|
|
|
|
Prevents an existing deployment from being shutdown due to inactivity.
|
|
|
|
#### `ci:security-genai-run-evals`
|
|
|
|
Run evaluations for the GenAI security evaluation suite. |