mirror of
https://github.com/elastic/kibana.git
synced 2025-04-22 17:04:01 -04:00
5ab8a52187
## Summary Extends the Observability AI Assistant's evaluation framework to create the first set of tests aimed at evaluating the performance of the Investigation App's AI root cause analysis integration. To execute tests, please consult the [README](https://github.com/elastic/kibana/pull/204634/files#diff-4823a154e593051126d3d5822c88d72e89d07f41b8c07a5a69d18281c50b09adR1). Note the prerequisites and the Kibana & Elasticsearch configuration. Further evolution -- This PR is the first MVP of the evaluation framework. A (somewhat light) [meta issue](https://github.com/elastic/kibana/issues/205670) exists for our continued work on this project, and will be added to over time. Test data and fixture architecture -- Logs, metrics, and traces are indexed to [edge-rca](https://studious-disco-k66oojq.pages.github.io/edge-rca/). Observability engineers can [create an oblt-cli cluster](https://studious-disco-k66oojq.pages.github.io/user-guide/cluster-create-ccs/) configured for cross cluster search against edge-rca as the remote cluster. When creating new testing fixtures, engineers will utilize their oblt-cli cluster to create rules against the remote cluster data. Once alerts are triggered in a failure scenario, the engineer can choose to archive the alert data to utilize as a test fixture. Test fixtures are added to the `investigate_app/scripts/load/fixtures` directory for use in tests. When execute tests, the fixtures are loaded into the engineer's oblt-cli cluster, configured for cross cluster search against edge-rca. The local alert fixture and the remote demo data are utilized together to replay root cause analysis and execute the test evaluations. Implementation -- Creates a new directory `scripts`, to house scripts related to setting up and running these tests. Here's what each directory does: ## scripts/evaluate 1. Extends the evaluation script from `observability_ai_assistant_app/scripts/evaluation` by creating a [custom Kibana client](https://github.com/elastic/kibana/pull/204634/files#diff-ae05b2a20168ea08f452297fc1bd59310c69ac3ea4651da1f65cd9fa93bb8fe9R1) with RCA specific methods. The custom client is [passed to the Observability AI Assistant's `runEvaluations`](https://github.com/elastic/kibana/pull/204634/files#diff-0f2d3662c01df8fbe7d1f19704fa071cbd6232fb5f732b313e8ba99012925d0bR14) script an[d invoked instead of the default Kibana Client](https://github.com/elastic/kibana/pull/204634/files#diff-98509a357e86ea5c5931b1b46abc72f76e5304439430358eee845f9ad57f63f1R54). 2. Defines a single, MVP test in `index.spec.ts`. This test find a specific alert fixture designated for that test, creates an investigation for that alert with a specified time range, and calls the root cause analysis api. Once the report is received back from the api, a prompt is created for the evaluation framework with details of the report. The evaluation framework then judges how well the root cause analysis api performed against specified criteria. ## scripts/archive 1. Utilized when creating new test fixtures, this script will easily archive observability alerts data for use as a fixture in a feature test ## scripts/load 1. Loads created testing fixtures before running the test. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Dario Gieselaar <d.gieselaar@gmail.com>
166 lines
4.2 KiB
Text
166 lines
4.2 KiB
Text
.aws-config.json
|
|
.signing-config.json
|
|
.ackrc
|
|
/.es
|
|
/.chromium
|
|
.DS_Store
|
|
.node_binaries
|
|
/.beats
|
|
.native_modules
|
|
node_modules
|
|
!/src/dev/npm/integration_tests/__fixtures__/fixture1/node_modules
|
|
!/src/dev/notice/__fixtures__/node_modules
|
|
!/packages/kbn-import-resolver/src/__fixtures__/node_modules
|
|
!/packages/kbn-import-resolver/src/__fixtures__/packages/box/node_modules
|
|
trash
|
|
/optimize
|
|
/built_assets
|
|
target
|
|
/build
|
|
.jruby
|
|
.idea
|
|
*.iml
|
|
*.log
|
|
types.eslint.config.js
|
|
types.eslint.config.cjs
|
|
__tmp__
|
|
|
|
# Ignore example plugin builds
|
|
/examples/*/build
|
|
/x-pack/examples/*/build
|
|
|
|
# Ignore certain functional test runner artifacts
|
|
/test/*/failure_debug
|
|
/test/*/screenshots/diff
|
|
/test/*/screenshots/failure
|
|
/test/*/screenshots/session
|
|
/test/*/screenshots/visual_regression_gallery.html
|
|
|
|
# Ignore the same artifacts in x-pack
|
|
/x-pack/test/*/failure_debug
|
|
/x-pack/test/*/screenshots/diff
|
|
/x-pack/test/*/screenshots/failure
|
|
/x-pack/test/*/screenshots/session
|
|
/x-pack/test/*/screenshots/visual_regression_gallery.html
|
|
/x-pack/test/functional/apps/*/*/reporting/reports/failure
|
|
|
|
# Ignore the same artifacts in x-pack/test_serverless
|
|
/x-pack/test_serverless/*/failure_debug
|
|
/x-pack/test_serverless/*/screenshots/diff
|
|
/x-pack/test_serverless/*/screenshots/failure
|
|
/x-pack/test_serverless/*/screenshots/session
|
|
/x-pack/test_serverless/*/screenshots/visual_regression_gallery.html
|
|
/x-pack/test_serverless/functional/apps/*/*/reporting/reports/failure
|
|
|
|
/html_docs
|
|
.eslintcache
|
|
/plugins/
|
|
/data
|
|
disabledPlugins
|
|
webpackstats.json
|
|
/config/*
|
|
!/config/kibana.yml
|
|
!/config/README.md
|
|
!/config/serverless.yml
|
|
!/config/serverless.es.yml
|
|
!/config/serverless.oblt.yml
|
|
!/config/serverless.security.yml
|
|
!/config/node.options
|
|
coverage
|
|
!/test/common/fixtures/plugins/coverage
|
|
selenium
|
|
.babel_register_cache.json
|
|
.webpack.babelcache
|
|
*.swp
|
|
*.swo
|
|
*.out
|
|
package-lock.json
|
|
!/.buildkite/package-lock.json
|
|
.yo-rc.json
|
|
.vscode
|
|
*.sublime-*
|
|
npm-debug.log*
|
|
.tern-project
|
|
.nyc_output
|
|
.gradle
|
|
.vagrant
|
|
.envrc
|
|
|
|
## Snyk
|
|
.dccache
|
|
|
|
## @cypress/snapshot from apm plugin
|
|
/snapshots.js
|
|
/apm-diagnostics*.json
|
|
/x-pack/solutions/observability/plugins/apm/ftr_e2e/cypress/e2e/service_map/snapshots/*.actual.png
|
|
/x-pack/solutions/observability/plugins/apm/ftr_e2e/cypress/e2e/service_map/snapshots/*.diff.png
|
|
|
|
# transpiled cypress config
|
|
x-pack/platform/plugins/shared/fleet/cypress.config.d.ts
|
|
x-pack/platform/plugins/shared/fleet/cypress.config.js
|
|
x-pack/platform/plugins/shared/osquery/cypress.config.d.ts
|
|
x-pack/platform/plugins/shared/osquery/cypress.config.js
|
|
x-pack/solutions/search/plugins/enterprise_search/cypress.config.d.ts
|
|
x-pack/solutions/search/plugins/enterprise_search/cypress.config.js
|
|
x-pack/solutions/security/plugins/security_solution/public/management/cypress.config.d.ts
|
|
x-pack/solutions/security/plugins/security_solution/public/management/cypress.config.js
|
|
x-pack/solutions/security/plugins/security_solution/public/management/cypress_endpoint.config.d.ts
|
|
x-pack/solutions/security/plugins/security_solution/public/management/cypress_endpoint.config.js
|
|
|
|
# release notes script output
|
|
report.csv
|
|
report.asciidoc
|
|
|
|
# TS incremental build cache
|
|
*.tsbuildinfo
|
|
|
|
# Automatically generated and user-modifiable
|
|
/tsconfig.refs.json
|
|
*.type_check.json
|
|
|
|
# Yarn local mirror content
|
|
.yarn-local-mirror
|
|
|
|
# Bazel
|
|
.ijwb
|
|
/bazel
|
|
/bazel-*
|
|
.bazelrc.user
|
|
.bazelrc.cache
|
|
|
|
elastic-agent-*
|
|
fleet-server-*
|
|
elastic-agent.yml
|
|
fleet-server.yml
|
|
src/platform/packages/**/package-map.json
|
|
/packages/**/config-paths.json
|
|
/packages/kbn-synthetic-package-map/
|
|
**/.synthetics/
|
|
**/.journeys/
|
|
**/.rca/
|
|
x-pack/test/security_api_integration/plugins/audit_log/audit.log
|
|
|
|
# ignore FTR temp directory
|
|
.ftr
|
|
role_users.json
|
|
|
|
# ignore Scout temp directory
|
|
.scout
|
|
|
|
.devcontainer/.env
|
|
|
|
# Ignore temporary files in oas_docs
|
|
oas_docs/output/kibana.serverless.tmp*.yaml
|
|
oas_docs/output/kibana.tmp*.yaml
|
|
oas_docs/output/kibana.new.yaml
|
|
oas_docs/output/kibana.serverless.new.yaml
|
|
|
|
# Security Solution Playwright
|
|
x-pack/test/security_solution_playwright/test-results/
|
|
x-pack/test/security_solution_playwright/playwright-report/
|
|
x-pack/test/security_solution_playwright/blob-report/
|
|
x-pack/test/security_solution_playwright/playwright/.cache/
|
|
x-pack/test/security_solution_playwright/.auth/
|
|
x-pack/test/security_solution_playwright/.env
|
|
.codeql
|
|
.dependency-graph-log.json
|