mirror of
https://github.com/elastic/kibana.git
synced 2025-04-24 09:48:58 -04:00
Your window into the Elastic Stack
cc66320e97
## Summary It implements the background task to execute the rule migrations and the API to manage them. It also contains a basic implementation of the langGraph agent workflow that will perform the migration using generative AI. > [!NOTE] > This feature needs `siemMigrationsEnabled` experimental flag enabled to work. Otherwise, the new API routes won't be registered, and the `SiemRuleMigrationsService` _setup_ won't be called. So no migration task code can be reached, and no data stream/template will be installed to ES. ### The rule migration task implementation: - Retrieve a batch of N rule migration documents (50 rules initially, we may change that later) with `status: pending`. - Update those documents to `status: processing`. - Execute the migration for each of the N migrations in parallel. - If there is any error update the document with `status: error`. - For each rule migration that finishes we set the result to the storage, and also update `status: finished`. - When all the batch of rules is finished the task will check if there are still migration documents with `status: pending` if so it will process the next batch with a delay (10 seconds initially, we may change that later). - If the task is stopped (via API call or server shut-down), we do a bulk update for all the `status: processing` documents back to `status: pending`. ### Task API - `POST /internal/siem_migrations/rules` (implemented [here](https://github.com/elastic/security-team/issues/10654)) -> Creates the migration on the backend and stores the original rules. It returns the `migration_id` - `GET /internal/siem_migrations/rules/stats` -> Retrieves the stats for all the existing migrations, aggregated by `migration_id`. - `GET /internal/siem_migrations/rules/{migration_id}` -> Retrieves all the migration rule documents of a specific migration. - `PUT /internal/siem_migrations/rules/{migration_id}/start` -> Starts the background task for a specific migration. - `GET /internal/siem_migrations/rules/{migration_id}/stats` -> Retrieves the stats of a specific migration task. The UI will do polling to this endpoint. - `PUT /internal/siem_migrations/rules/{migration_id}/stop` -> Stops the execution of a specific migration running task. When a migration is stopped, the executing task is aborted and all the rules in the batch being processed are moved back to pending, all finished rules will remain stored. When the Kibana server shuts down all the running migrations are stopped automatically. To resume the migration we can call `{migration_id}/start` again and it will take it from the same rules batch it was left. #### Stats (UI polling) response example: ``` { "status": "running", "rules": { "total": 34, "finished": 20, "pending": 4, "processing": 10, "failed": 0 }, "last_updated_at": "2024-10-29T15:04:49.618Z" } ``` ### LLM agent Graph The initial implementation of the agent graph that is executed per rule:  The first node tries to match the original rule with an Elastic prebuilt rule. If it does not succeed, the second node will try to translate the query as a custom rule using the ES|QL knowledge base, this composes previous PoCs: - https://github.com/elastic/kibana/pull/193900 - https://github.com/elastic/kibana/pull/196651 ## Testing locally Enable the flag ``` xpack.securitySolution.enableExperimental: ['siemMigrationsEnabled'] ``` cURL request examples: <details> <summary>Rules migration `create` POST request</summary> ``` curl --location --request POST 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' \ --header 'Content-Type: application/json' \ --data '[ { "id": "f8c325ea-506e-4105-8ccf-da1492e90115", "vendor": "splunk", "title": "Linux Auditd Add User Account Type", "description": "The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.", "query": "sourcetype=\"linux:audit\" type=ADD_USER \n| rename hostname as dest \n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \n| `security_content_ctime(firstTime)` \n| `security_content_ctime(lastTime)`\n| search *", "query_language":"spl", "mitre_attack_ids": [ "T1136" ] }, { "id": "7b87c556-0ca4-47e0-b84c-6cd62a0a3e90", "vendor": "splunk", "title": "Linux Auditd Change File Owner To Root", "description": "The following analytic detects the use of the '\''chown'\'' command to change a file owner to '\''root'\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.", "query": "`linux_auditd` `linux_auditd_normalized_proctitle_process`\r\n| rename host as dest \r\n| where LIKE (process_exec, \"%chown %root%\") \r\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \r\n| `security_content_ctime(firstTime)` \r\n| `security_content_ctime(lastTime)`\r\n| `linux_auditd_change_file_owner_to_root_filter`", "query_language": "spl", "mitre_attack_ids": [ "T1222" ] } ]' ``` </details> <details> <summary>Rules migration `start` task request</summary> - Assuming the connector `azureOpenAiGPT4o` is already created in the local environment. - Using the {{`migration_id`}} from the first POST request response ``` curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/start' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' \ --header 'Content-Type: application/json' \ --data '{ "connectorId": "azureOpenAiGPT4o" }' ``` </details> <details> <summary>Rules migration `stop` task request</summary> - Using the {{`migration_id`}} from the first POST request response. ``` curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/stop' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' ``` </details> <details> <summary>Rules migration task `stats` request</summary> - Using the {{`migration_id`}} from the first POST request response. ``` curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/stats' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' ``` </details> <details> <summary>Rules migration rules documents request</summary> - Using the {{`migration_id`}} from the first POST request response. ``` curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' ``` </details> <details> <summary>Rules migration all stats request</summary> ``` curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/stats' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' ``` </details> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> |
||
|---|---|---|
| .buildkite | ||
| .devcontainer | ||
| .github | ||
| api_docs | ||
| config | ||
| dev_docs | ||
| docs | ||
| examples | ||
| kbn_pm | ||
| legacy_rfcs | ||
| licenses | ||
| oas_docs | ||
| packages | ||
| plugins | ||
| scripts | ||
| src | ||
| test | ||
| typings | ||
| x-pack | ||
| .backportrc.json | ||
| .bazelignore | ||
| .bazeliskversion | ||
| .bazelrc | ||
| .bazelrc.common | ||
| .bazelversion | ||
| .browserslistrc | ||
| .editorconfig | ||
| .eslintignore | ||
| .eslintrc.js | ||
| .gitattributes | ||
| .gitignore | ||
| .i18nrc.json | ||
| .node-version | ||
| .npmrc | ||
| .nvmrc | ||
| .prettierignore | ||
| .prettierrc | ||
| .puppeteerrc | ||
| .stylelintignore | ||
| .stylelintrc | ||
| .telemetryrc.json | ||
| .yarnrc | ||
| BUILD.bazel | ||
| catalog-info.yaml | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| FAQ.md | ||
| fleet_packages.json | ||
| github_checks_reporter.json | ||
| kibana.d.ts | ||
| LICENSE.txt | ||
| NOTICE.txt | ||
| package.json | ||
| preinstall_check.js | ||
| README.md | ||
| renovate.json | ||
| RISK_MATRIX.mdx | ||
| run_fleet_setup_parallel.sh | ||
| SECURITY.md | ||
| sonar-project.properties | ||
| STYLEGUIDE.mdx | ||
| tsconfig.base.json | ||
| tsconfig.browser.json | ||
| tsconfig.browser_bazel.json | ||
| tsconfig.json | ||
| TYPESCRIPT.md | ||
| updatecli-compose.yaml | ||
| versions.json | ||
| WORKSPACE.bazel | ||
| yarn.lock | ||
Kibana
Kibana is your window into the Elastic Stack. Specifically, it's a browser-based analytics and search dashboard for Elasticsearch.
- Getting Started
- Documentation
- Version Compatibility with Elasticsearch
- Questions? Problems? Suggestions?
Getting Started
If you just want to try Kibana out, check out the Elastic Stack Getting Started Page to give it a whirl.
If you're interested in diving a bit deeper and getting a taste of Kibana's capabilities, head over to the Kibana Getting Started Page.
Using a Kibana Release
If you want to use a Kibana release in production, give it a test run, or just play around:
- Download the latest version on the Kibana Download Page.
- Learn more about Kibana's features and capabilities on the Kibana Product Page.
- We also offer a hosted version of Kibana on our Cloud Service.
Building and Running Kibana, and/or Contributing Code
You might want to build Kibana locally to contribute some code, test out the latest features, or try out an open PR:
- CONTRIBUTING.md will help you get Kibana up and running.
- If you would like to contribute code, please follow our STYLEGUIDE.mdx.
- For all other questions, check out the FAQ.md and wiki.
Documentation
Visit Elastic.co for the full Kibana documentation.
For information about building the documentation, see the README in elastic/docs.
Version Compatibility with Elasticsearch
Ideally, you should be running Elasticsearch and Kibana with matching version numbers. If your Elasticsearch has an older version number or a newer major number than Kibana, then Kibana will fail to run. If Elasticsearch has a newer minor or patch number than Kibana, then the Kibana Server will log a warning.
Note: The version numbers below are only examples, meant to illustrate the relationships between different types of version numbers.
| Situation | Example Kibana version | Example ES version | Outcome |
|---|---|---|---|
| Versions are the same. | 7.15.1 | 7.15.1 | 💚 OK |
| ES patch number is newer. | 7.15.0 | 7.15.1 | ⚠️ Logged warning |
| ES minor number is newer. | 7.14.2 | 7.15.0 | ⚠️ Logged warning |
| ES major number is newer. | 7.15.1 | 8.0.0 | 🚫 Fatal error |
| ES patch number is older. | 7.15.1 | 7.15.0 | ⚠️ Logged warning |
| ES minor number is older. | 7.15.1 | 7.14.2 | 🚫 Fatal error |
| ES major number is older. | 8.0.0 | 7.15.1 | 🚫 Fatal error |
Questions? Problems? Suggestions?
- If you've found a bug or want to request a feature, please create a GitHub Issue. Please check to make sure someone else hasn't already created an issue for the same topic.
- Need help using Kibana? Ask away on our Kibana Discuss Forum and a fellow community member or Elastic engineer will be glad to help you out.