Your window into the Elastic Stack
Find a file
Vitalii Dmyterko b03b2fd477
[Security Solution][Detection Engine] adds ES|QL rule type to Security Detections rules (#165450)
## Summary

- related [epic](https://github.com/elastic/security-team/issues/6196)
- introduces new ES|QL rule type in Technical Preview Stage

- historical POC architecture
[document](https://docs.google.com/document/d/1hcKzNrDEIrmoWwWoqas1YZ-bd8Kk5NRjJNSUaCvSntM/edit#heading=h.gheuu8zcz481)(internal
link). Some of the information there can be outdated, but might be
useful for historical context of some tech decision. In future, detailed
technical documentation will be added

### UI

ES|QL query component introduced in rule edit/creation form

Rule name override supports values returned from ES|QL query
As agreed on Adv. correlation WG, we don't introduce similar possibility
for risk score/severity override at this point

<details>
<summary>How it looks like in UI</summary>
<img width="2082" alt="Screenshot 2023-09-21 at 11 52 59"
src="14c94e36-ca90-496d-a7a5-4a31899d25b6">
<img width="2079" alt="Screenshot 2023-09-21 at 11 53 14"
src="9abd53ec-a0f4-4481-8b1f-4ecccdc5feae">
<img width="2072" alt="Screenshot 2023-09-21 at 12 14 17"
src="58e4f9eb-c15f-4849-bba0-bc1b92e8c945">


</details>


### Context

We introduced concept of Aggregating and Non-aggregating rules for
ES|QL. It depends on, whether STATS..BY command used in query

**Aggregating rule** - is a rule that uses
[stats…by](https://esql.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/esql-stats-by.html)
grouping commands. So, its result can not be matched to a particular
document in ES. This can lead to possibly duplicated alerts, since we
are using document `id` to deduplicate alerts. We are going to introduce
suppression for all rule types in future, that would help to mitigate
this case
```
FROM logs*
| STATS count = COUNT(host.name) BY host.name
| SORT host.name
```

**Non-aggregating rule** - is a rule that does not use
[stats…by](https://esql.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/esql-stats-by.html)
grouping commands. Each row in result can be tracked to a source
document in ES. For this type of rule operator \`[metadata _id, _index,
_version]\` is required to be used after defining index source. This
would allow deduplicate alerts and link them with the source document.

```
FROM logs* [metadata _id, _index, _version]
| WHERE event.id == "test"
| LIMIT 10
```

### Serverless Feature Flag

ES|QL won't be available for Serverless as for 8.11 release, so it will
be hidden by Security experimental feature flag `esqlRulesDisabled`. All
UI changes will be hidden (it's mostly Form creation) and rule type
won't be registered, which prevents rule to be created, returned in
search if it exists or execute.

### Test envs
- Serverless qa, [admin link to
project](https://admin.qa.cld.elstc.co/projects/security/ef79684f92d64f27b69e1b04de86eb1a),
disabled there
- internal
[link](https://elastic.slack.com/archives/C03E8TR26HE/p1693848029955229)
to test env for Stateful


### Rule schema changes

introduces value `esql` to `type` property
introduces value `esql` to `language` property

### Tests coverage
- cypress tests (as per 27/09/2023 added cypress tests for rule
creation/edit/details,bulk_edit))
- functional tests for rule execution(exceptions, overrides, preview and
actual rule execution)
  - functional tests for bulk_edit

 #### Flaky test runner
- [cypress esql
tests](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3233#_),
non failed of added


### Checklist

Delete any items that are not applicable to this PR.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)


### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-09-30 09:45:34 +01:00
.buildkite [Dashboard Navigation] Make links panel available under technical preview (#166896) 2023-09-29 08:25:51 -06:00
.ci Upgrade Node.js to 18.17.1 (#163710) 2023-08-14 12:08:27 -05:00
.github [Security Solution][Detection Engine] adds ES|QL rule type to Security Detections rules (#165450) 2023-09-30 09:45:34 +01:00
api_docs [api-docs] 2023-09-30 Daily api_docs build (#167712) 2023-09-30 00:38:43 -04:00
bundle Update core architecture docs (#164120) 2023-08-21 15:38:24 -04:00
config [Security Solution][Detection Engine] adds ES|QL rule type to Security Detections rules (#165450) 2023-09-30 09:45:34 +01:00
dev_docs [kbn/journeys] fixes to run journeys against ESS cluster (#166923) 2023-09-28 12:06:00 +02:00
docs [Security solution] Rename Generative AI connector to OpenAI (#167677) 2023-09-29 11:54:55 -05:00
examples [Kibana] New "Saved Query Management" privilege to allow saving queries across Kibana (#166937) 2023-09-29 11:52:39 +02:00
kbn_pm chore(NA): exit early with link for docs when using kbn_pm on windows (#139745) 2023-02-10 15:38:29 +00:00
legacy_rfcs rename @elastic/* packages to @kbn/* (#138957) 2022-08-18 08:54:42 -07:00
licenses Elastic License 2.0 (#90099) 2021-02-03 18:12:39 -08:00
packages [Security Solution][Detection Engine] adds ES|QL rule type to Security Detections rules (#165450) 2023-09-30 09:45:34 +01:00
plugins
scripts [Security Solution] Extract OpenAPI codegen to a package (#166269) 2023-09-25 10:51:40 +02:00
src [Dashboard] Store view mode in local storage (#166523) 2023-09-29 19:46:18 -04:00
test [Dashboard] Store view mode in local storage (#166523) 2023-09-29 19:46:18 -04:00
typings Upgrade EUI to v88.3.0 (#166676) 2023-09-20 08:03:02 -05:00
vars [RAM] Alert table all column fix 2 (#161054) 2023-07-07 09:57:22 -06:00
x-pack [Security Solution][Detection Engine] adds ES|QL rule type to Security Detections rules (#165450) 2023-09-30 09:45:34 +01:00
.backportrc.json It adds 8.10 into the .backportrc config file (#164259) 2023-08-21 15:15:30 +01:00
.bazelignore Bazel config maintenance (#135442) 2022-07-05 10:20:26 -05:00
.bazeliskversion chore(NA): upgrade bazelisk into v1.11.0 (#125070) 2022-02-09 20:43:57 +00:00
.bazelrc chore(NA): use new and more performant BuildBuddy servers (#130350) 2022-04-18 02:01:38 +01:00
.bazelrc.common Transpile packages on demand, validate all TS projects (#146212) 2022-12-22 19:00:29 -06:00
.bazelversion chore(NA): revert bazel upgrade for v5.2.0 (#135096) 2022-06-24 03:57:21 +01:00
.browserslistrc [browserslist] remove unnecessary browsers (#89186) 2021-01-25 16:30:18 -07:00
.editorconfig .editorconfig MDX files should follow the same rules as MD (#96942) 2021-04-13 11:40:42 -04:00
.eslintignore [NOTICE.txt] Fix notices for Gainsight and FullStory (#146004) 2023-01-13 14:49:43 +01:00
.eslintrc.js [ML] AIOps: Enable event_generating_elements_should_be_instrumented eslint rule. (#167317) 2023-09-28 11:49:13 +02:00
.gitattributes
.gitignore [APM] Add snapshots to visual regression tests (#165834) 2023-09-11 08:59:16 +01:00
.i18nrc.json [Dashboard Navigation] Make links panel available under technical preview (#166896) 2023-09-29 08:25:51 -06:00
.node-version Upgrade Node.js to 18.17.1 (#163710) 2023-08-14 12:08:27 -05:00
.npmrc chore(NA): assure puppeteer_skip_chromium_download is applied across every yarn install situation (#88346) 2021-01-14 18:00:23 +00:00
.nvmrc Upgrade Node.js to 18.17.1 (#163710) 2023-08-14 12:08:27 -05:00
.prettierignore [dev] Replace sass-lint with stylelint (#86177) 2021-01-15 11:52:29 -06:00
.prettierrc
.stylelintignore chore(NA): stop grouping bazel out symlink folders (#96066) 2021-04-01 14:16:14 -05:00
.stylelintrc Bump stylelint to ^14 (#136693) 2022-07-20 10:11:00 -05:00
.telemetryrc.json [Telemetry] Fix telemetry-tools TS parser for packages (#149819) 2023-01-31 04:09:09 +03:00
.yarnrc chore(NA): manage npm dependencies within bazel (#92864) 2021-03-03 12:37:20 -05:00
BUILD.bazel Transpile packages on demand, validate all TS projects (#146212) 2022-12-22 19:00:29 -06:00
catalog-info.yaml Removing the double quotes from the build.filter (#165500) 2023-09-01 13:58:32 -04:00
CODE_OF_CONDUCT.md Add CODE_OF_CONDUCT.md (#87439) 2021-02-23 09:01:51 +01:00
CONTRIBUTING.md Update doc slugs to improve analytic tracking, move to appropriate folders (#113630) 2021-10-04 13:36:45 -04:00
FAQ.md Fix small typos in the root md files (#134609) 2022-06-23 09:36:11 -05:00
fleet_packages.json [main] Sync bundled packages with Package Storage (#165562) 2023-09-04 02:07:49 -07:00
github_checks_reporter.json
Jenkinsfile [CI] Disable tracked branch jobs in Jenkins, enable reporting in Buildkite (#112604) 2021-09-21 11:31:15 -04:00
kibana.d.ts fix all violations 2022-04-16 01:37:30 -05:00
LICENSE.txt Elastic License 2.0 (#90099) 2021-02-03 18:12:39 -08:00
nav-kibana-dev.docnav.json Developer documentation for designing feature privileges (#166716) 2023-09-27 13:43:55 +02:00
NOTICE.txt [FullStory] Update snippet (#153570) 2023-04-18 04:06:05 -07:00
package.json [serverless] Add Advanced Settings page (#167383) 2023-09-29 08:48:37 -07:00
preinstall_check.js Elastic License 2.0 (#90099) 2021-02-03 18:12:39 -08:00
README.md [README] Update version Compatibility with Elasticsearch (#116040) 2022-01-10 10:31:21 -05:00
renovate.json Upgrade openpgp 5.3.0 to 5.10.1 (#165526) 2023-09-18 08:31:16 -04:00
RISK_MATRIX.mdx Add "Risk Matrix" section to the PR template (#100649) 2021-06-02 14:43:47 +02:00
SECURITY.md Add security policy to the Kibana repository (#85407) 2020-12-10 09:26:00 -05:00
STYLEGUIDE.mdx [styleguide] update path to scss theme (#140742) 2022-09-15 10:41:14 -04:00
tsconfig.base.json [serverless] Add Advanced Settings page (#167383) 2023-09-29 08:48:37 -07:00
tsconfig.browser.json
tsconfig.browser_bazel.json [build_ts_refs] improve caches, allow building a subset of projects (#107981) 2021-08-10 22:12:45 -07:00
tsconfig.json Transpile packages on demand, validate all TS projects (#146212) 2022-12-22 19:00:29 -06:00
TYPESCRIPT.md Fix small typos in the root md files (#134609) 2022-06-23 09:36:11 -05:00
versions.json Update versions.json (#166944) 2023-09-21 11:42:33 -05:00
WORKSPACE.bazel Upgrade Node.js to 18.17.1 (#163710) 2023-08-14 12:08:27 -05:00
yarn.lock [serverless] Add Advanced Settings page (#167383) 2023-09-29 08:48:37 -07:00

Kibana

Kibana is your window into the Elastic Stack. Specifically, it's a browser-based analytics and search dashboard for Elasticsearch.

Getting Started

If you just want to try Kibana out, check out the Elastic Stack Getting Started Page to give it a whirl.

If you're interested in diving a bit deeper and getting a taste of Kibana's capabilities, head over to the Kibana Getting Started Page.

Using a Kibana Release

If you want to use a Kibana release in production, give it a test run, or just play around:

Building and Running Kibana, and/or Contributing Code

You might want to build Kibana locally to contribute some code, test out the latest features, or try out an open PR:

Documentation

Visit Elastic.co for the full Kibana documentation.

For information about building the documentation, see the README in elastic/docs.

Version Compatibility with Elasticsearch

Ideally, you should be running Elasticsearch and Kibana with matching version numbers. If your Elasticsearch has an older version number or a newer major number than Kibana, then Kibana will fail to run. If Elasticsearch has a newer minor or patch number than Kibana, then the Kibana Server will log a warning.

Note: The version numbers below are only examples, meant to illustrate the relationships between different types of version numbers.

Situation Example Kibana version Example ES version Outcome
Versions are the same. 7.15.1 7.15.1 💚 OK
ES patch number is newer. 7.15.0 7.15.1 ⚠️ Logged warning
ES minor number is newer. 7.14.2 7.15.0 ⚠️ Logged warning
ES major number is newer. 7.15.1 8.0.0 🚫 Fatal error
ES patch number is older. 7.15.1 7.15.0 ⚠️ Logged warning
ES minor number is older. 7.15.1 7.14.2 🚫 Fatal error
ES major number is older. 8.0.0 7.15.1 🚫 Fatal error

Questions? Problems? Suggestions?

  • If you've found a bug or want to request a feature, please create a GitHub Issue. Please check to make sure someone else hasn't already created an issue for the same topic.
  • Need help using Kibana? Ask away on our Kibana Discuss Forum and a fellow community member or Elastic engineer will be glad to help you out.