[Security Solution][Detection Engine] adds ES|QL rule type to Security Detections rules (#165450)
## Summary
- related [epic](https://github.com/elastic/security-team/issues/6196)
- introduces new ES|QL rule type in Technical Preview Stage
- historical POC architecture
[document](https://docs.google.com/document/d/1hcKzNrDEIrmoWwWoqas1YZ-bd8Kk5NRjJNSUaCvSntM/edit#heading=h.gheuu8zcz481)(internal
link). Some of the information there can be outdated, but might be
useful for historical context of some tech decision. In future, detailed
technical documentation will be added
### UI
ES|QL query component introduced in rule edit/creation form
Rule name override supports values returned from ES|QL query
As agreed on Adv. correlation WG, we don't introduce similar possibility
for risk score/severity override at this point
<details>
<summary>How it looks like in UI</summary>
<img width="2082" alt="Screenshot 2023-09-21 at 11 52 59"
src="14c94e36-ca90-496d-a7a5-4a31899d25b6">
<img width="2079" alt="Screenshot 2023-09-21 at 11 53 14"
src="9abd53ec-a0f4-4481-8b1f-4ecccdc5feae">
<img width="2072" alt="Screenshot 2023-09-21 at 12 14 17"
src="58e4f9eb-c15f-4849-bba0-bc1b92e8c945">
</details>
### Context
We introduced concept of Aggregating and Non-aggregating rules for
ES|QL. It depends on, whether STATS..BY command used in query
**Aggregating rule** - is a rule that uses
[stats…by](https://esql.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/esql-stats-by.html)
grouping commands. So, its result can not be matched to a particular
document in ES. This can lead to possibly duplicated alerts, since we
are using document `id` to deduplicate alerts. We are going to introduce
suppression for all rule types in future, that would help to mitigate
this case
```
FROM logs*
| STATS count = COUNT(host.name) BY host.name
| SORT host.name
```
**Non-aggregating rule** - is a rule that does not use
[stats…by](https://esql.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/esql-stats-by.html)
grouping commands. Each row in result can be tracked to a source
document in ES. For this type of rule operator \`[metadata _id, _index,
_version]\` is required to be used after defining index source. This
would allow deduplicate alerts and link them with the source document.
```
FROM logs* [metadata _id, _index, _version]
| WHERE event.id == "test"
| LIMIT 10
```
### Serverless Feature Flag
ES|QL won't be available for Serverless as for 8.11 release, so it will
be hidden by Security experimental feature flag `esqlRulesDisabled`. All
UI changes will be hidden (it's mostly Form creation) and rule type
won't be registered, which prevents rule to be created, returned in
search if it exists or execute.
### Test envs
- Serverless qa, [admin link to
project](https://admin.qa.cld.elstc.co/projects/security/ef79684f92d64f27b69e1b04de86eb1a),
disabled there
- internal
[link](https://elastic.slack.com/archives/C03E8TR26HE/p1693848029955229)
to test env for Stateful
### Rule schema changes
introduces value `esql` to `type` property
introduces value `esql` to `language` property
### Tests coverage
- cypress tests (as per 27/09/2023 added cypress tests for rule
creation/edit/details,bulk_edit))
- functional tests for rule execution(exceptions, overrides, preview and
actual rule execution)
- functional tests for bulk_edit
#### Flaky test runner
- [cypress esql
tests](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3233#_),
non failed of added
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
b03b2fd477