[8.8] Update security telemetry filterlist with network fields. (#158166) (#158351)

# Backport

This will backport the following commits from `main` to `8.8`:
- [Update security telemetry filterlist with network fields.
(#158166)](https://github.com/elastic/kibana/pull/158166)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Pete
Hampton","email":"pjhampton@users.noreply.github.com"},"sourceCommit":{"committedDate":"2023-05-23T16:25:23Z","message":"Update
security telemetry filterlist with network fields. (#158166)\n\n##
Summary\r\n\r\nThis PR filters in network-related fields to improve
threat research\r\ncontext, improve telemetry on existing network-based
rules, and allow us\r\nto provide additional malicious indicators with
research publications.\r\n\r\n - Issue backref'd\r\n - OOB artifact
backref'd\r\n - Stakeholders and legal engaged\r\n\r\ncc
@terrancedejesus for context. \r\n\r\n### Checklist\r\n\r\nDelete any
items that are not applicable to this PR.\r\n\r\n- [ ] Any text added
follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [ ] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] Any UI
touched in this PR is usable by keyboard only (learn more\r\nabout
[keyboard accessibility](https://webaim.org/techniques/keyboard/))\r\n-
[ ] Any UI touched in this PR does not create any new axe
failures\r\n(run axe in
browser:\r\n[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),\r\n[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))\r\n-
[ ] If a plugin configuration key changed, check if it needs to
be\r\nallowlisted in the cloud and added to the
[docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n-
[ ] This renders correctly on smaller devices using a
responsive\r\nlayout. (You can test this [in
your\r\nbrowser](https://www.browserstack.com/guide/responsive-testing-on-local-server))\r\n-
[ ] This was checked for
[cross-browser\r\ncompatibility](https://www.elastic.co/support/matrix#matrix_browsers)\r\n\r\n\r\n###
Risk Matrix\r\n\r\nDelete this section if it is not applicable to this
PR.\r\n\r\nBefore closing this PR, invite QA, stakeholders, and other
developers to\r\nidentify risks that should be tested prior to the
change/feature\r\nrelease.\r\n\r\nWhen forming the risk matrix, consider
some of the following examples\r\nand how they may potentially impact
the change:\r\n\r\n| Risk | Probability | Severity | Mitigation/Notes
|\r\n\r\n|---------------------------|-------------|----------|-------------------------|\r\n|
Multiple Spaces&mdash;unexpected behavior in non-default Kibana
Space.\r\n| Low | High | Integration tests will verify that all features
are still\r\nsupported in non-default Kibana Space and when user
switches between\r\nspaces. |\r\n| Multiple nodes&mdash;Elasticsearch
polling might have race conditions\r\nwhen multiple Kibana nodes are
polling for the same tasks. | High | Low\r\n| Tasks are idempotent, so
executing them multiple times will not result\r\nin logical error, but
will degrade performance. To test for this case we\r\nadd plenty of unit
tests around this logic and document manual testing\r\nprocedure. |\r\n|
Code should gracefully handle cases when feature X or plugin Y
are\r\ndisabled. | Medium | High | Unit tests will verify that any
feature flag\r\nor plugin combination still results in our service
operational. |\r\n| [See more potential
risk\r\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
|\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for
breaking API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"8761e5057c7b87d54fcabcc3c7574ed1ce952a08","branchLabelMapping":{"^v8.9.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Feature:Telemetry","release_note:skip","Team:
SecuritySolution","auto-backport","v8.9.0","v8.8.1"],"number":158166,"url":"https://github.com/elastic/kibana/pull/158166","mergeCommit":{"message":"Update
security telemetry filterlist with network fields. (#158166)\n\n##
Summary\r\n\r\nThis PR filters in network-related fields to improve
threat research\r\ncontext, improve telemetry on existing network-based
rules, and allow us\r\nto provide additional malicious indicators with
research publications.\r\n\r\n - Issue backref'd\r\n - OOB artifact
backref'd\r\n - Stakeholders and legal engaged\r\n\r\ncc
@terrancedejesus for context. \r\n\r\n### Checklist\r\n\r\nDelete any
items that are not applicable to this PR.\r\n\r\n- [ ] Any text added
follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [ ] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] Any UI
touched in this PR is usable by keyboard only (learn more\r\nabout
[keyboard accessibility](https://webaim.org/techniques/keyboard/))\r\n-
[ ] Any UI touched in this PR does not create any new axe
failures\r\n(run axe in
browser:\r\n[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),\r\n[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))\r\n-
[ ] If a plugin configuration key changed, check if it needs to
be\r\nallowlisted in the cloud and added to the
[docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n-
[ ] This renders correctly on smaller devices using a
responsive\r\nlayout. (You can test this [in
your\r\nbrowser](https://www.browserstack.com/guide/responsive-testing-on-local-server))\r\n-
[ ] This was checked for
[cross-browser\r\ncompatibility](https://www.elastic.co/support/matrix#matrix_browsers)\r\n\r\n\r\n###
Risk Matrix\r\n\r\nDelete this section if it is not applicable to this
PR.\r\n\r\nBefore closing this PR, invite QA, stakeholders, and other
developers to\r\nidentify risks that should be tested prior to the
change/feature\r\nrelease.\r\n\r\nWhen forming the risk matrix, consider
some of the following examples\r\nand how they may potentially impact
the change:\r\n\r\n| Risk | Probability | Severity | Mitigation/Notes
|\r\n\r\n|---------------------------|-------------|----------|-------------------------|\r\n|
Multiple Spaces&mdash;unexpected behavior in non-default Kibana
Space.\r\n| Low | High | Integration tests will verify that all features
are still\r\nsupported in non-default Kibana Space and when user
switches between\r\nspaces. |\r\n| Multiple nodes&mdash;Elasticsearch
polling might have race conditions\r\nwhen multiple Kibana nodes are
polling for the same tasks. | High | Low\r\n| Tasks are idempotent, so
executing them multiple times will not result\r\nin logical error, but
will degrade performance. To test for this case we\r\nadd plenty of unit
tests around this logic and document manual testing\r\nprocedure. |\r\n|
Code should gracefully handle cases when feature X or plugin Y
are\r\ndisabled. | Medium | High | Unit tests will verify that any
feature flag\r\nor plugin combination still results in our service
operational. |\r\n| [See more potential
risk\r\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
|\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for
breaking API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"8761e5057c7b87d54fcabcc3c7574ed1ce952a08"}},"sourceBranch":"main","suggestedTargetBranches":["8.8"],"targetPullRequestStates":[{"branch":"main","label":"v8.9.0","labelRegex":"^v8.9.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/158166","number":158166,"mergeCommit":{"message":"Update
security telemetry filterlist with network fields. (#158166)\n\n##
Summary\r\n\r\nThis PR filters in network-related fields to improve
threat research\r\ncontext, improve telemetry on existing network-based
rules, and allow us\r\nto provide additional malicious indicators with
research publications.\r\n\r\n - Issue backref'd\r\n - OOB artifact
backref'd\r\n - Stakeholders and legal engaged\r\n\r\ncc
@terrancedejesus for context. \r\n\r\n### Checklist\r\n\r\nDelete any
items that are not applicable to this PR.\r\n\r\n- [ ] Any text added
follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [ ] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] Any UI
touched in this PR is usable by keyboard only (learn more\r\nabout
[keyboard accessibility](https://webaim.org/techniques/keyboard/))\r\n-
[ ] Any UI touched in this PR does not create any new axe
failures\r\n(run axe in
browser:\r\n[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),\r\n[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))\r\n-
[ ] If a plugin configuration key changed, check if it needs to
be\r\nallowlisted in the cloud and added to the
[docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n-
[ ] This renders correctly on smaller devices using a
responsive\r\nlayout. (You can test this [in
your\r\nbrowser](https://www.browserstack.com/guide/responsive-testing-on-local-server))\r\n-
[ ] This was checked for
[cross-browser\r\ncompatibility](https://www.elastic.co/support/matrix#matrix_browsers)\r\n\r\n\r\n###
Risk Matrix\r\n\r\nDelete this section if it is not applicable to this
PR.\r\n\r\nBefore closing this PR, invite QA, stakeholders, and other
developers to\r\nidentify risks that should be tested prior to the
change/feature\r\nrelease.\r\n\r\nWhen forming the risk matrix, consider
some of the following examples\r\nand how they may potentially impact
the change:\r\n\r\n| Risk | Probability | Severity | Mitigation/Notes
|\r\n\r\n|---------------------------|-------------|----------|-------------------------|\r\n|
Multiple Spaces&mdash;unexpected behavior in non-default Kibana
Space.\r\n| Low | High | Integration tests will verify that all features
are still\r\nsupported in non-default Kibana Space and when user
switches between\r\nspaces. |\r\n| Multiple nodes&mdash;Elasticsearch
polling might have race conditions\r\nwhen multiple Kibana nodes are
polling for the same tasks. | High | Low\r\n| Tasks are idempotent, so
executing them multiple times will not result\r\nin logical error, but
will degrade performance. To test for this case we\r\nadd plenty of unit
tests around this logic and document manual testing\r\nprocedure. |\r\n|
Code should gracefully handle cases when feature X or plugin Y
are\r\ndisabled. | Medium | High | Unit tests will verify that any
feature flag\r\nor plugin combination still results in our service
operational. |\r\n| [See more potential
risk\r\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
|\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for
breaking API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"8761e5057c7b87d54fcabcc3c7574ed1ce952a08"}},{"branch":"8.8","label":"v8.8.1","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Pete Hampton <pjhampton@users.noreply.github.com>

x-pack/plugins/security_solution/server/lib/telemetry/filterlists/endpoint_alerts.ts

@@ -51,6 +51,7 @@ const baseAllowlistFields: AllowlistFields = {
 // Allow list for event-related fields, which can also be nested under events[]
 const allowlistBaseEventFields: AllowlistFields = {
   credential_access: true,
+  destination: true,
   dll: {
     name: true,
     path: true,
@@ -96,9 +97,7 @@ const allowlistBaseEventFields: AllowlistFields = {
     parent: baseAllowlistFields,
     ...baseAllowlistFields,
   },
-  network: {
-    direction: true,
-  },
+  network: true,
   registry: {
     data: {
       strings: true,
@@ -108,12 +107,14 @@ const allowlistBaseEventFields: AllowlistFields = {
     path: true,
     value: true,
   },
+  source: true,
   Target: {
     process: {
       parent: baseAllowlistFields,
       ...baseAllowlistFields,
     },
   },
+  url: true,
   user: {
     id: true,
   },

x-pack/plugins/security_solution/server/lib/telemetry/filterlists/prebuilt_rules_alerts.ts

@@ -115,11 +115,7 @@ export const prebuiltRuleAllowlistFields: AllowlistFields = {
       sha256: true,
     },
   },
-  dns: {
-    question: {
-      name: true,
-    },
-  },
+  dns: true,
   event: true,
   group: {
     name: true,

x-pack/plugins/security_solution/server/lib/telemetry/sender.test.ts

@@ -91,6 +91,9 @@ describe('TelemetryEventsSender', () => {
             ruleset: 'Z',
             version: '100',
           },
+          destination: {
+            bytes: 1530,
+          },
           dll: {
             Ext: {
               device: {
@@ -172,6 +175,9 @@ describe('TelemetryEventsSender', () => {
             something_else: 'nope',
           },
           message: 'Malicious Behavior Detection Alert: Regsvr32 with Unusual Arguments',
+          network: {
+            transport: 'tcp',
+          },
           process: {
             name: 'foo.exe',
             nope: 'nope',
@@ -210,6 +216,13 @@ describe('TelemetryEventsSender', () => {
             },
           },
           Responses: '{ "result": 0 }', // >= 7.15
+          source: {
+            geo: {
+              continent_name: 'Europe',
+              country_iso_code: 'EE',
+              country_name: 'Estonia',
+            },
+          },
           Target: {
             process: {
               name: 'bar.exe',
@@ -222,6 +235,12 @@ describe('TelemetryEventsSender', () => {
           threat: {
             ignored_object: true, // this field is not allowlisted
           },
+          url: {
+            domain: 'elastic.co',
+            full: 'https://elastic.co',
+            path: '/',
+            scheme: 'http',
+          },
           Persistence: {
             name: 'foo',
             path: '/foo/bar',
@@ -292,6 +311,9 @@ describe('TelemetryEventsSender', () => {
             ruleset: 'Z',
             version: '100',
           },
+          destination: {
+            bytes: 1530,
+          },
           file: {
             extension: '.exe',
             size: 3,
@@ -352,6 +374,9 @@ describe('TelemetryEventsSender', () => {
             },
           },
           message: 'Malicious Behavior Detection Alert: Regsvr32 with Unusual Arguments',
+          network: {
+            transport: 'tcp',
+          },
           process: {
             name: 'foo.exe',
             working_directory: '/some/usr/dir',
@@ -388,6 +413,13 @@ describe('TelemetryEventsSender', () => {
             },
           },
           Responses: '{ "result": 0 }',
+          source: {
+            geo: {
+              continent_name: 'Europe',
+              country_iso_code: 'EE',
+              country_name: 'Estonia',
+            },
+          },
           Target: {
             process: {
               name: 'bar.exe',
@@ -396,6 +428,12 @@ describe('TelemetryEventsSender', () => {
               },
             },
           },
+          url: {
+            domain: 'elastic.co',
+            full: 'https://elastic.co',
+            path: '/',
+            scheme: 'http',
+          },
           Persistence: {
             name: 'foo',
             path: '/foo/bar',