Your window into the Elastic Stack
Find a file
Georgii Gorbachev 2431a08d2b
[Security Solution][Detections] Reading last 5 failures from Event Log v1 - raw implementation (#115574)
**Ticket:** https://github.com/elastic/kibana/issues/106469, https://github.com/elastic/kibana/issues/101013

## Summary

TL;DR: New internal endpoint for reading data from Event Log (raw version), legacy status SO under the hood.

With this PR we now read the Failure History (last 5 failures) on the Rule Details page from Event Log. We continue getting the Current Status from the legacy `siem-detection-engine-rule-status` saved objects. Rule Management page also gets data from the legacy saved objects.

- [x] Deprecate existing methods for reading data in `IRuleExecutionLogClient`: `.find()` and `.findBulk()`
- [x] Introduce new methods for reading data in IRuleExecutionLogClient:
  - for reading last N execution events for 1 rule from event log
  - for reading current status and metrics for 1 rule from legacy status SOs
  - for reading current statuses and metrics for N rules from legacy status SOs
- [x] New methods should return data in the legacy status SO format.
- [x] Update all the existing endpoints that depend on `IRuleExecutionLogClient` to use the new methods.
- [x] Implement a new internal endpoint for fetching current status of the rule execution and execution events from Event Log for a given rule.
- [x] The API of the new endpoint should be the same as `rules/_find_statuses` to minimise changes in the app.
- [x] Use the new endpoint on the Rule Details page.

## Near-term plan for technical implementation of the Rule Execution Log (https://github.com/elastic/kibana/issues/101013)

**Stage 1. Reading last 5 failures from Event Log v1 - raw implementation** - ✔️ done in this PR

TL;DR: New internal endpoint for reading data from Event Log (raw version), legacy status SO under the hood.

- Deprecate existing methods for reading data in `IRuleExecutionLogClient`: `.find()` and `.findBulk()`
- Introduce new methods for reading data in IRuleExecutionLogClient:
  - for reading last N execution events for 1 rule from event log
  - for reading current status and metrics for 1 rule from legacy status SOs
  - for reading current statuses and metrics for N rules from legacy status SOs
- New methods should return data in the legacy status SO format.
- Update all the existing endpoints that depend on `IRuleExecutionLogClient` to use the new methods.
- Implement a new internal endpoint for fetching current status of the rule execution and execution events from Event Log for a given rule.
- The API of the new endpoint should be the same as `rules/_find_statuses` to minimise changes in the app.
- Use the new endpoint on the Rule Details page.

**Stage 2: Reading last 5 failures from Event Log v2 - clean implementation**

TL;DR: Clean HTTP API, legacy Rule Status SO under the hood.

🚨🚨🚨 Possible breaking changes in Detections API 🚨🚨🚨

- Design a new data model for the Current Rule Execution Info (the TO-BE new SO type and later the TO-BE data in the rule object itself).
- Design a new data model for the Rule Execution Event (read model to be used on the Rule Details page)
- Think over changes in `IRuleExecutionLogClient` to support the new data model.
- Think over changes in all the endpoints that return any data related to rule monitoring (statuses, metrics, etc). Make sure to check our docs to identify what's documented there regarding rule monitoring.
- Update `IRuleExecutionLogClient` to return data in the new format. 
- Update all the endpoints (including the raw new one) to return data in the new format.
- Update Rule Details page to consume data in the new format.
- Update Rule Management page to consume data in the new format.

**Stage 3: Reading last 5 failures from Event Log v3 - new SO**

TL;DR: Clean HTTP API, new Rule Execution Info SO under the hood.

- Implement a new SO type for storing the current rule execution info. Relation type: 1 rule - 1 current execution info.
- Swap the legacy SO with the new SO in the implementation of `IRuleExecutionLogClient`.

**Stage 4: Cleanup and misc**

- Revisit the problem of deterministic ordering ([comment](https://github.com/elastic/kibana/pull/115574#discussion_r735803087))
- Remove rule execution log's glue code: adapters, feature switch.
- Remove the legacy rule status SO.
- Mark the legacy rule status SO as deleted in Kibana Core.
- Encapsulate the current space id in the instance of IRuleExecutionLogClient. Remove it from parameters of its methods.
- Introduce a Rule Execution Logger scoped to a rule instance. For use in rule executors.
- Add test coverage.

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-11-01 14:40:14 +01:00
.buildkite [ci] Temporarily stop writing to Bazel remote cache (#116866) 2021-10-29 21:06:26 -07:00
.ci Change all references of master branch to main (#116654) 2021-10-29 12:29:48 -04:00
.github add the "UI" team label to APM project issues (#116844) 2021-10-29 16:32:36 -04:00
api_docs [Cases] Fixed typo (#116042) 2021-10-26 23:09:29 +03:00
config [8.0] remove kibana.index config property (#112773) 2021-10-25 14:25:24 -07:00
dev_docs Updates Github link references from master to main (#116789) 2021-10-29 09:53:08 -07:00
docs [Actions][Docs] Rename isLegacy to usesTableApi (#116922) 2021-11-01 14:49:35 +02:00
examples Updates Github link references from master to main (#116789) 2021-10-29 09:53:08 -07:00
legacy_rfcs Updates Github link references from master to main (#116789) 2021-10-29 09:53:08 -07:00
licenses Elastic License 2.0 (#90099) 2021-02-03 18:12:39 -08:00
packages [kbn/optimizer] dll @babel/runtime modules used by entry bundles (#113453) 2021-10-29 18:08:20 -05:00
plugins [dev/cli] ensure plugins/ and all watch source dirs exist (#78973) 2020-09-30 10:20:44 -07:00
scripts [kbn/optimizer] dll @babel/runtime modules used by entry bundles (#113453) 2021-10-29 18:08:20 -05:00
src Remove unused isNewInstance code (#116747) 2021-11-01 12:57:32 +00:00
test retry specific metric suggestion check (#116909) 2021-11-01 13:48:12 +01:00
typings [Fix] Replace Osquery query parser lib (#113425) 2021-09-29 20:07:13 +02:00
vars adds lists plugin to security solution cypress tests execution (#116442) 2021-10-27 20:55:55 +01:00
x-pack [Security Solution][Detections] Reading last 5 failures from Event Log v1 - raw implementation (#115574) 2021-11-01 14:40:14 +01:00
.backportrc.json Change all references of master branch to main (#116654) 2021-10-29 12:29:48 -04:00
.bazelignore chore(NA): stop grouping bazel out symlink folders (#96066) 2021-04-01 14:16:14 -05:00
.bazeliskversion Development and build support for ARM based Apple devices (#114879) 2021-10-19 23:01:52 +01:00
.bazelrc [bazel] Remove Bazel cache for now (#116351) 2021-10-26 12:04:17 -07:00
.bazelrc.common [ci] Disable Bazel cache on CI (#115428) 2021-10-19 22:12:31 +01:00
.bazelversion Development and build support for ARM based Apple devices (#114879) 2021-10-19 23:01:52 +01:00
.browserslistrc [browserslist] remove unnecessary browsers (#89186) 2021-01-25 16:30:18 -07:00
.editorconfig .editorconfig MDX files should follow the same rules as MD (#96942) 2021-04-13 11:40:42 -04:00
.eslintignore [kbn/ui-shared-deps] split into two packages (#110558) 2021-09-16 12:06:46 -07:00
.eslintrc.js [kibanaUtils] Don't import full semver client side (#114986) 2021-10-18 13:20:44 +02:00
.fossa.yml Adds FOSSA CLI configuration file (#70137) 2020-07-02 08:37:37 -07:00
.gitattributes [canvas] Color fixes + Storybook 5 (#34075) 2019-04-02 11:21:51 -05:00
.gitignore [build] Add cloud docker images (#107949) 2021-09-09 20:40:09 -05:00
.i18nrc.json [MetricVis] Move the expression function and renderer to a separate plugin. (#113427) 2021-10-08 08:42:44 +03:00
.node-version Bump node to 16.11.1 (#110684) 2021-10-16 16:21:58 -05:00
.npmrc chore(NA): assure puppeteer_skip_chromium_download is applied across every yarn install situation (#88346) 2021-01-14 18:00:23 +00:00
.nvmrc Bump node to 16.11.1 (#110684) 2021-10-16 16:21:58 -05:00
.prettierignore [dev] Replace sass-lint with stylelint (#86177) 2021-01-15 11:52:29 -06:00
.prettierrc Increase prettier line width to 100 (#20535) 2018-07-09 22:50:37 +02:00
.stylelintignore chore(NA): stop grouping bazel out symlink folders (#96066) 2021-04-01 14:16:14 -05:00
.stylelintrc Amsterdam helpers (#93701) 2021-03-10 10:27:16 -06:00
.telemetryrc.json [Usage collection] Collect non-default kibana configs (#97368) 2021-04-20 11:02:27 -04:00
.yarnrc chore(NA): manage npm dependencies within bazel (#92864) 2021-03-03 12:37:20 -05:00
api-documenter.json Normalize EOL symbol in platform docs (#56021) 2020-01-27 18:42:45 +01:00
BUILD.bazel [build_ts_refs] improve caches, allow building a subset of projects (#107981) 2021-08-10 22:12:45 -07:00
CODE_OF_CONDUCT.md Add CODE_OF_CONDUCT.md (#87439) 2021-02-23 09:01:51 +01:00
CONTRIBUTING.md Update doc slugs to improve analytic tracking, move to appropriate folders (#113630) 2021-10-04 13:36:45 -04:00
FAQ.md propose language changes (#10709) 2017-03-05 12:10:32 -05:00
github_checks_reporter.json implementing github checks - second attempt (#35757) 2019-05-01 16:02:33 -05:00
Jenkinsfile [CI] Disable tracked branch jobs in Jenkins, enable reporting in Buildkite (#112604) 2021-09-21 11:31:15 -04:00
jest.config.integration.js chore(NA): introduce preset for jest-integration tests on @kbn/test (#105144) 2021-07-14 20:53:12 +01:00
jest.config.js [Canvas] Expression tagcloud (#108036) 2021-08-23 06:52:19 -04:00
kibana.d.ts Remove /src/legacy (#95510) 2021-04-06 09:25:36 +02:00
LICENSE.txt Elastic License 2.0 (#90099) 2021-02-03 18:12:39 -08:00
NOTICE.txt [Detections] Adds automatic updating for Prebuilt Security Detection Rules package (#101846) 2021-06-24 15:31:25 -06:00
package.json Upgrade cheerio dependency (1.0.0-rc.91.0.0-rc.10). (#116737) 2021-10-29 20:24:49 +02:00
preinstall_check.js Elastic License 2.0 (#90099) 2021-02-03 18:12:39 -08:00
README.md STYLEGUIDE reference .md => .mdx (#107440) 2021-08-02 14:05:58 -05:00
renovate.json5 [renovate] Renames master to main (#116790) 2021-10-29 09:55:26 -07:00
RISK_MATRIX.mdx Add "Risk Matrix" section to the PR template (#100649) 2021-06-02 14:43:47 +02:00
SECURITY.md Add security policy to the Kibana repository (#85407) 2020-12-10 09:26:00 -05:00
STYLEGUIDE.mdx remove all references to v7 theme (#113570) 2021-10-29 12:28:42 -05:00
tsconfig.base.json Adds missing DOM.Iterable (#115218) 2021-10-18 13:22:49 -04:00
tsconfig.bazel.json [build_ts_refs] improve caches, allow building a subset of projects (#107981) 2021-08-10 22:12:45 -07:00
tsconfig.browser.json Introduce TS incremental builds & move src/test_utils to TS project (#76082) 2020-09-03 14:20:04 +02:00
tsconfig.browser_bazel.json [build_ts_refs] improve caches, allow building a subset of projects (#107981) 2021-08-10 22:12:45 -07:00
tsconfig.json Add interactive setup CLI (#114493) 2021-10-20 22:17:45 +01:00
tsconfig.types.json [build_ts_refs] improve caches, allow building a subset of projects (#107981) 2021-08-10 22:12:45 -07:00
TYPESCRIPT.md Fixed grammar (#74725) 2020-08-11 06:40:22 -04:00
WORKSPACE.bazel Development and build support for ARM based Apple devices (#114879) 2021-10-19 23:01:52 +01:00
yarn.lock Upgrade cheerio dependency (1.0.0-rc.91.0.0-rc.10). (#116737) 2021-10-29 20:24:49 +02:00

Kibana

Kibana is your window into the Elastic Stack. Specifically, it's a browser-based analytics and search dashboard for Elasticsearch.

Getting Started

If you just want to try Kibana out, check out the Elastic Stack Getting Started Page to give it a whirl.

If you're interested in diving a bit deeper and getting a taste of Kibana's capabilities, head over to the Kibana Getting Started Page.

Using a Kibana Release

If you want to use a Kibana release in production, give it a test run, or just play around:

Building and Running Kibana, and/or Contributing Code

You might want to build Kibana locally to contribute some code, test out the latest features, or try out an open PR:

Documentation

Visit Elastic.co for the full Kibana documentation.

For information about building the documentation, see the README in elastic/docs.

Version Compatibility with Elasticsearch

Ideally, you should be running Elasticsearch and Kibana with matching version numbers. If your Elasticsearch has an older version number or a newer major number than Kibana, then Kibana will fail to run. If Elasticsearch has a newer minor or patch number than Kibana, then the Kibana Server will log a warning.

Note: The version numbers below are only examples, meant to illustrate the relationships between different types of version numbers.

Situation Example Kibana version Example ES version Outcome
Versions are the same. 5.1.2 5.1.2 💚 OK
ES patch number is newer. 5.1.2 5.1.5 ⚠️ Logged warning
ES minor number is newer. 5.1.2 5.5.0 ⚠️ Logged warning
ES major number is newer. 5.1.2 6.0.0 🚫 Fatal error
ES patch number is older. 5.1.2 5.1.0 ⚠️ Logged warning
ES minor number is older. 5.1.2 5.0.0 🚫 Fatal error
ES major number is older. 5.1.2 4.0.0 🚫 Fatal error

Questions? Problems? Suggestions?

  • If you've found a bug or want to request a feature, please create a GitHub Issue. Please check to make sure someone else hasn't already created an issue for the same topic.
  • Need help using Kibana? Ask away on our Kibana Discuss Forum and a fellow community member or Elastic engineer will be glad to help you out.