github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-06-27 18:51:07 -04:00
Code Issues Projects Releases Packages Wiki Activity
kibana/.buildkite
History
Garrett Spong e9a8909fad
[Security Assistant] Simplifies Security Gen AI Evaluation secret management (#219885) 
## Summary

Simplifies secret management for running the Security Gen AI
Evaluations. See updated README.md for full details, but includes:

* Consolidation of multiple vault keys to a single
`KIBANA_SECURITY_GEN_AI_CONFIG` key, which contains all connectors,
langsmith creds and now a way to specify `evaluatorConnectorId`.
* Added `vault` params to both `retrieve_secrets.js` and
`upload_secrets.js` for specifying the vault. Defaults to `sieam-team`
secrets.elastic.co for ease of use by developers.
* Introduces `get_commands.js` script for fetching commands to hand off
to either Kibana Ops for updating, or specifying config overrides when
manually running BuildKite pipelines.
* Deleted `export_env_secrets.js` as it couldn't be used for setting env
vars locally for the dev testing experience.
* Updated `connectors` as per team discussion to include: GPT-4.1,
Claude 3.5/3.7, and Gemini 2.5 Pro. This was a config change made by
Kibana Ops, so no code change present. But you can confirm by running
`retrieve_secrets.js`.

And finally, a much more detailed `README.md` for testing locally, on
PR's and CI, and the process for updating secrets. See full
[README.md](https://github.com/spong/kibana/blob/ci-eval-tweaks/x-pack/test/security_solution_api_integration/test_suites/genai/evaluations/README.md)



Example LangSmith Runs:

* `ES|QL Generation Regression Suite`: [Run
298372](261dcc59-fbe7-4397-a662-ff94042f666c)
* `Alerts RAG Regression (Episodes 1-8)`: [Run
298372](bd5bba1d-97aa-4512-bce7-b09aa943c651)
* `Assistant Eval: Custom Knowledge`: [Run
298372](2d5f7c18-4bf4-4cdb-97a1-16e39a865cab)
* `Eval AD: All Scenarios`: [Run
300138](4690ee16-9df5-416c-8bf0-b62bc2f2aba9/compare?selectedSessions=6d44134b-6492-4f2d-9b28-6d4a82a0e9ae&baseline=undefined)

Note: there is currently a timing bug with Alerts/KB entries being
cleaned up before the server is complete, so you may see poor evals for
`Alerts RAG Regression (Episodes 1-8)` and `Assistant Eval: Custom
Knowledge` until that is fixed. I'll address this in a follow-up PR
since it is unrelated to this change-set.
2025-05-09 11:01:36 -06:00
..
hooks [Ops] Refactor env loading & fix agent-targeting rewrite (#178320) 2024-03-12 16:31:26 +01:00
pipeline-resource-definitions [On-Week] Automate chromium linux builds for reporting in Kibana (#212674) 2025-05-06 22:04:08 +02:00
pipeline-utils [CI] Allow using elastic-images-qa through PR label or env var (#216878) 2025-04-04 10:05:30 +02:00
pipelines [On-Week] Automate chromium linux builds for reporting in Kibana (#212674) 2025-05-06 22:04:08 +02:00
scripts [Security Assistant] Simplifies Security Gen AI Evaluation secret management (#219885) 2025-05-09 11:01:36 -06:00
.mocharc.json [CI] Move kibana-buildkite-library into kibana (#134787) 2022-06-28 13:19:05 -04:00
.npmrc [CI] Move kibana-buildkite-library into kibana (#134787) 2022-06-28 13:19:05 -04:00
disabled_jest_configs.json [Index Management] Fix broken jest tests (#173029) 2023-12-18 17:21:28 +01:00
ftr_base_serverless_configs.yml [FTR] support "deployment agnostic" api-integration tests (#189853) 2024-08-07 10:34:52 -05:00
ftr_chat_serverless_configs.yml [Config stripUnknowns] Skip compatible mode when running in CI (#217536) 2025-04-14 17:52:51 +02:00
ftr_configs_manifests.json [workchat] Initial FTR test setup (#216828) 2025-04-09 10:31:12 +02:00
ftr_oblt_serverless_configs.yml [ftr] split svl common configs with 4 extra groups (#218415) 2025-04-16 11:34:55 -05:00
ftr_oblt_stateful_configs.yml [ska][x-pack] relocate api_integration tests [3] (#219518) 2025-05-05 12:05:25 +02:00
ftr_platform_stateful_configs.yml search: functional test migration for ska (#220382) 2025-05-07 13:37:55 -05:00
ftr_search_serverless_configs.yml [ftr] split svl common configs with 4 extra groups (#218415) 2025-04-16 11:34:55 -05:00
ftr_search_stateful_configs.yml [Search] test: removing stale cypress & functional_enterprise_search references (#220446) 2025-05-08 12:33:09 -05:00
ftr_security_serverless_configs.yml [AI4DSOC] Disable NLPCleanupTask for search_ai_lake tier (#219006) 2025-04-24 14:03:05 -06:00
ftr_security_stateful_configs.yml [ska][x-pack] relocate platform tests [4] (#219691) 2025-05-07 19:13:04 +02:00
package-lock.json [On-Week] Automate chromium linux builds for reporting in Kibana (#212674) 2025-05-06 22:04:08 +02:00
package.json [On-Week] Automate chromium linux builds for reporting in Kibana (#212674) 2025-05-06 22:04:08 +02:00
pull_requests.json [ci] Click to deploy cloud (#205623) 2025-01-23 10:26:03 +01:00
README.md [CI] Move kibana-buildkite-library into kibana (#134787) 2022-06-28 13:19:05 -04:00
scout_ci_config.yml [Scout] add painless lab 2025-04-24 18:19:23 +03:00
tsconfig.json fix(NA): adds pipeline-resource-definitions/fix-location-collection.ts into a tsconfig 2024-04-04 15:16:01 +01:00
tsconfig.test.json [CI] Move kibana-buildkite-library into kibana (#134787) 2022-06-28 13:19:05 -04:00

README.md

Kibana / Buildkite

Directory Structure

  • hooks - special directory used by Buildkite agents for hooks
  • pipeline-utils - Shared TypeScript utils for use in pipeline scripts
  • pipelines - contains pipeline definitions
  • scripts/common - scripts that get sourced by other scripts to set environment variables or import shared functions
  • scripts/lifecycle - general scripts for tasks that run before or after individual steps or the entire build
  • scripts/steps - scripts that define something that will run for a step defined in a pipeline
  • scripts/* - all other scripts are building blocks that make up the tasks in pipelines. They may be run by other scripts, but should not be sourced