github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-23 09:19:04 -04:00
Code Issues Projects Releases Packages Wiki Activity

[Security Solution] New Data Quality dashboard Same family category (#167480)

Browse source 
# [Security Solution] New Data Quality dashboard `Same family` category

This PR introduces a new `Same family` category to the [Data Quality dashboard](https://www.elastic.co/guide/en/security/current/data-quality-dash.html).

Fields with the `same family` tag that were previously counted in the `Incompatible fields` category are now counted in the new `Same family` category, per the annotated screenshot below:

![same_family_before_after_annotated](c7c5d496-03e5-4e06-b837-85f2f16ab885)

_Above - Left: previously, fields with the `same family` tag were counted as `Incompatible fields`, Right: in this PR, those fields are counted in the new `Same family` category_

The annotations on the _Right_ side of the screenshot above highlight (in this example):

- The total `Incompatible fields` count was reduced
  - Some patterns, like `auditbeat-*` have significant reductions
- The `Incompatible fields` count was reduced by one field
- The new `Same family` category has one field
- The `agent.type` field moved from the `Incompatible fields` category to the `Same family` category

## Details

### The new `Same family` category

Fields with mappings in the same [family](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html) have exactly the same search behavior as the type specified by ECS, but may have different space usage or performance characteristics.

The color of the `Same family` category is the same as the `Custom fields` category in badges and charts, per the screenshot below:

![same_family_color](259b7e8e-3a52-482e-a419-edf715f8e462)

_Above: The `Same family` tab's badge and chart legend color is the same as the `Custom fields` category_

### The new `Same family` tab

This PR introduces a new `Same family` tab, as shown in the screenshot below:

![same_family_tab](f4da99cf-6fa4-4477-8a85-eec9d85c6787)

_Above: The new `Same family` tab is selected_

In the screenshot above:

- The callout includes a description of fields in the same family (moved from the `Incompatible fields` tab)
- The `constant_keyword` text, yellow in previous versions (when it appered in the `Incompatible fields` tab), is blue
- Only one action, `Copy to clipboard` is available in the `Same family` tab. The remaining text in this _Details_ section is the markdown copied to the clipboard for the example above:

### auditbeat-custom-index-1

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | auditbeat-custom-index-1 | 2 (0.0%) | 3 | `unmanaged` | 13.1KB |

### **Incompatible fields** `3` **Same family** `1` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `10`

#### 1 Same family field mapping

This field is defined by the Elastic Common Schema (ECS), version 8.6.1, but its mapping type doesn't exactly match.

Fields with mappings in the same family have exactly the same search behavior as the type specified by ECS, but may have different space usage or performance characteristics.

#### Same family field mappings - auditbeat-custom-index-1

| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| agent.type | `keyword` | `constant_keyword` `same family` |

## Desk testing

1) Navigate to `Dev Tools` > `Console`

2) Execute the queries below:

<details>
  <summary>Queries to create the `auditbeat-custom-index-1` example in this PR description</summary>

  ```
DELETE auditbeat-custom-index-1

PUT auditbeat-custom-index-1

PUT auditbeat-custom-index-1/_mapping
{
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "agent.type": {
      "type": "constant_keyword"
    },
    "event.category": {
      "type": "constant_keyword"
    }
  }
}

POST auditbeat-custom-index-1/_doc
{
  "@timestamp": "2023-02-06T09:41:49.668Z",
  "host": {
    "name": "foo"
  },
  "event": {
    "category": "an_invalid_category"
  },
  "some.field": "this",
  "source": {
    "port": 90210,
    "ip": "10.1.2.3"
  }
}

POST auditbeat-custom-index-1/_doc
{
  "@timestamp": "2023-02-06T09:42:22.123Z",
  "host": {
    "name": "bar"
  },
  "event": {
    "category": "an_invalid_category"
  },
  "some.field": "space",
  "source": {
    "port": 867,
    "ip": "10.9.8.7"
  }
}
```

</details>

3) Navigate to `Security` > `Dashboards` > `Data Quality`

4) Expand the `auditbeat-custom-index-1` index

**Expected results**

- The sum of the category badge counts, `3 + 1 + 4 + 2`, equals the total number of fields in the `All fields` [`10`] category:
  - `Incompatible fields` [`3`]
  - `Same family` [`1`]
  - `Custom fields` [`4`]
  - `ECS compliant fields` [`2`]
- The `Incompatible fields` callout title is `3 incompatible fields`
- The `Incompatible fields` callout does NOT include a description of same family fields
- The `Incompatible fields` tab displays two fields with incompatible mappings
  - `host.name` (`keyword` vs `text`)
  - `source.ip` (`ip` vs `text`)
- The `Incompatible fields` tab displays one field with incompatible field values
  - `event.category` (`an_invalid_category`)

5) Click the `Summary` tab

**Expected results**

- The `Summary` tab is focused
- The chart legend includes a `Same family` entry, with a count of `1`
- The `Same family` tab's badge color is the same as the `Custom fields` category
- The `Same family` chart legend  color is the same as the `Custom fields` category

6) Click the `Same family` legend item

**Expected results**

- The `Same family` tab is focused
- One field, `agent.type`, is displayed
  - The `constant_keyword` Index mapping type is blue
  - The `same family` badge is yellow

7) Click `Copy to clipboard`

**Expected result**

- The expected markdown is copied to the clipboard:

```
### auditbeat-custom-index-1

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | auditbeat-custom-index-1 | 2 (0.0%) | 3 | `unmanaged` | 12.9KB |


### **Incompatible fields** `3` **Same family** `1` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `10`

#### 1 Same family field mapping

This field is defined by the Elastic Common Schema (ECS), version 8.6.1, but its mapping type doesn't exactly match.

Fields with mappings in the same family have exactly the same search behavior as the type specified by ECS, but may have different space usage or performance characteristics.


#### Same family field mappings - auditbeat-custom-index-1


| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| agent.type | `keyword` | `constant_keyword` `same family` |
```
This commit is contained in:
Andrew Macri 2023-10-02 11:37:40 -06:00 • committed by GitHub
parent 26e61c1413
commit fe5abc9192
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
46 changed files with 1492 additions and 160 deletions
Download patch file Download diff file Expand all files Collapse all files

4
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/compare_fields_table/get_common_table_columns/index.test.tsx
View file

@ -122,8 +122,8 @@ describe('getCommonTableColumns', () => {
);
});
test('it renders the expected type as a warning', () => {
expect(screen.getByTestId('codeWarning')).toHaveTextContent(indexFieldType);
test('it renders the index field with a "success" style', () => {
expect(screen.getByTestId('codeSuccess')).toHaveTextContent(indexFieldType);
});
test('it renders the same family badge', () => {

7
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/compare_fields_table/get_common_table_columns/index.tsx
View file

@ -11,8 +11,9 @@ import React from 'react';
import { SameFamily } from '../../data_quality_panel/same_family';
import { EcsAllowedValues } from '../ecs_allowed_values';
import { getIsInSameFamily } from '../../helpers';
import { IndexInvalidValues } from '../index_invalid_values';
import { CodeDanger, CodeSuccess, CodeWarning } from '../../styles';
import { CodeDanger, CodeSuccess } from '../../styles';
import * as i18n from '../translations';
import type { AllowedValue, EnrichedFieldMetadata, UnallowedValueCount } from '../../types';
@ -45,9 +46,9 @@ export const getCommonTableColumns = (): Array<
name: i18n.INDEX_MAPPING_TYPE_ACTUAL,
render: (_, x) =>
x.type != null && x.indexFieldType !== x.type ? (
x.isInSameFamily ? (
getIsInSameFamily({ ecsExpectedType: x.type, type: x.indexFieldType }) ? (
<div>
<CodeWarning data-test-subj="codeWarning">{x.indexFieldType}</CodeWarning>
<CodeSuccess data-test-subj="codeSuccess">{x.indexFieldType}</CodeSuccess>
<SameFamily />
</div>
) : (

4
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/compare_fields_table/get_incompatible_mappings_table_columns/index.test.tsx
View file

@ -105,8 +105,8 @@ describe('getIncompatibleMappingsTableColumns', () => {
);
});
test('it renders the expected type as a warning', () => {
expect(screen.getByTestId('codeWarning')).toHaveTextContent(indexFieldType);
test('it renders the expected type with a "success" style', () => {
expect(screen.getByTestId('codeSuccess')).toHaveTextContent(indexFieldType);
});
test('it renders the same family badge', () => {

4
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/compare_fields_table/get_incompatible_mappings_table_columns/index.tsx
View file

@ -9,7 +9,7 @@ import type { EuiTableFieldDataColumnType } from '@elastic/eui';
import React from 'react';
import { SameFamily } from '../../data_quality_panel/same_family';
import { CodeDanger, CodeSuccess, CodeWarning } from '../../styles';
import { CodeDanger, CodeSuccess } from '../../styles';
import * as i18n from '../translations';
import type { EnrichedFieldMetadata } from '../../types';
@ -43,7 +43,7 @@ export const getIncompatibleMappingsTableColumns = (): Array<
render: (indexFieldType: string, x) =>
x.isInSameFamily ? (
<div>
<CodeWarning data-test-subj="codeWarning">{indexFieldType}</CodeWarning>
<CodeSuccess data-test-subj="codeSuccess">{indexFieldType}</CodeSuccess>
<SameFamily />
</div>
) : (

1
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/data_quality_summary/summary_actions/check_all/check_index.ts
View file

@ -21,6 +21,7 @@ export const EMPTY_PARTITIONED_FIELD_METADATA: PartitionedFieldMetadata = {
custom: [],
ecsCompliant: [],
incompatible: [],
sameFamily: [],
};
export async function checkIndex({

6
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/index_properties/helpers.test.ts
View file

@ -71,6 +71,7 @@ describe('helpers', () => {
type: 'date',
},
],
sameFamily: [],
};
test('it returns a `PartitionedFieldMetadata` with an `incompatible` `@timestamp` when `mappingsProperties` is undefined', () => {
@ -301,7 +302,7 @@ describe('helpers', () => {
],
hasEcsMetadata: true,
isEcsCompliant: false,
isInSameFamily: true,
isInSameFamily: false,
},
{
dashed_name: 'host-name',
@ -620,7 +621,7 @@ describe('helpers', () => {
],
hasEcsMetadata: true,
isEcsCompliant: false,
isInSameFamily: true,
isInSameFamily: false,
},
{
dashed_name: 'host-name',
@ -657,6 +658,7 @@ describe('helpers', () => {
isInSameFamily: false,
},
],
sameFamily: [],
});
});
});

2
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/index_properties/helpers.ts
View file

@ -23,6 +23,7 @@ export const ALL_TAB_ID = 'allTab';
export const ECS_COMPLIANT_TAB_ID = 'ecsCompliantTab';
export const CUSTOM_TAB_ID = 'customTab';
export const INCOMPATIBLE_TAB_ID = 'incompatibleTab';
export const SAME_FAMILY_TAB_ID = 'sameFamilyTab';
export const SUMMARY_TAB_ID = 'summaryTab';
export const EMPTY_METADATA: PartitionedFieldMetadata = {
@ -30,6 +31,7 @@ export const EMPTY_METADATA: PartitionedFieldMetadata = {
ecsCompliant: [],
custom: [],
incompatible: [],
sameFamily: [],
};
export const getSortedPartitionedFieldMetadata = ({

6
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/index_properties/index.tsx
View file

@ -227,6 +227,11 @@ const IndexPropertiesComponent: React.FC<Props> = ({
? partitionedFieldMetadata.incompatible.length
: undefined;
const indexSameFamily: number | undefined =
error == null && partitionedFieldMetadata != null
? partitionedFieldMetadata.sameFamily.length
: undefined;
if (patternRollup != null) {
const markdownComments =
partitionedFieldMetadata != null
@ -255,6 +260,7 @@ const IndexPropertiesComponent: React.FC<Props> = ({
indexName,
markdownComments,
pattern,
sameFamily: indexSameFamily,
},
},
});

2
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/index_properties/markdown/helpers.test.ts
View file

@ -402,7 +402,7 @@ describe('helpers', () => {
describe('getTabCountsMarkdownComment', () => {
test('it returns a comment with the expected counts', () => {
expect(getTabCountsMarkdownComment(mockPartitionedFieldMetadata)).toBe(
'### **Incompatible fields** `3` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n'
'### **Incompatible fields** `3` **Same family** `0` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n'
);
});
});

2
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/index_properties/markdown/helpers.ts
View file

@ -201,6 +201,8 @@ export const getTabCountsMarkdownComment = (
): string =>
`### **${i18n.INCOMPATIBLE_FIELDS}** ${getCodeFormattedValue(
`${partitionedFieldMetadata.incompatible.length}`
)} **${i18n.SAME_FAMILY}** ${getCodeFormattedValue(
`${partitionedFieldMetadata.sameFamily.length}`
)} **${i18n.CUSTOM_FIELDS}** ${getCodeFormattedValue(
`${partitionedFieldMetadata.custom.length}`
)} **${i18n.ECS_COMPLIANT_FIELDS}** ${getCodeFormattedValue(

81
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/index_properties/translations.ts
View file

@ -276,6 +276,22 @@ export const CUSTOM_CALLOUT = ({ fieldCount, version }: { fieldCount: number; ve
'{fieldCount, plural, =1 {This field is not} other {These fields are not}} defined by the Elastic Common Schema (ECS), version {version}.',
});
export const SAME_FAMILY_CALLOUT = ({
fieldCount,
version,
}: {
fieldCount: number;
version: string;
}) =>
i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.sameFamilyCallout',
{
values: { fieldCount, version },
defaultMessage:
"{fieldCount, plural, =1 {This field is} other {These fields are}} defined by the Elastic Common Schema (ECS), version {version}, but {fieldCount, plural, =1 {its mapping type doesn't} other {their mapping types don't}} exactly match.",
}
);
export const CUSTOM_CALLOUT_TITLE = (fieldCount: number) =>
i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.customCalloutTitle',
@ -286,6 +302,16 @@ export const CUSTOM_CALLOUT_TITLE = (fieldCount: number) =>
}
);
export const SAME_FAMILY_CALLOUT_TITLE = (fieldCount: number) =>
i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.sameFamilyCalloutTitle',
{
values: { fieldCount },
defaultMessage:
'{fieldCount} Same family {fieldCount, plural, =1 {field mapping} other {field mappings}}',
}
);
export const CUSTOM_EMPTY = i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.customEmptyContent',
{
@ -307,13 +333,7 @@ export const INCOMPATIBLE_FIELDS = i18n.translate(
}
);
export const INCOMPATIBLE_CALLOUT = ({
fieldCount,
version,
}: {
fieldCount: number;
version: string;
}) =>
export const INCOMPATIBLE_CALLOUT = (version: string) =>
i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCallout',
{
@ -323,34 +343,27 @@ export const INCOMPATIBLE_CALLOUT = ({
}
);
export const INCOMPATIBLE_FIELDS_WITH = i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCallout.incompatibleFieldsWithLabel',
export const FIELDS_WITH_MAPPINGS_SAME_FAMILY = i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCallout.fieldsWithMappingsSameFamilyLabel',
{
defaultMessage:
'Incompatible fields with mappings in the same family have exactly the same search behavior but may have different space usage or performance characteristics.',
'Fields with mappings in the same family have exactly the same search behavior as the type specified by ECS, but may have different space usage or performance characteristics.',
}
);
export const WHEN_AN_INCOMPATIBLE_FIELD = i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCallout.whenAnIncompatibleFieldLabel',
export const WHEN_A_FIELD_IS_INCOMPATIBLE = i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCallout.whenAFieldIsIncompatibleLabel',
{
defaultMessage: 'When an incompatible field is not in the same family:',
defaultMessage: 'When a field is incompatible:',
}
);
export const INCOMPATIBLE_CALLOUT_TITLE = ({
fieldCount,
fieldsInSameFamily,
}: {
fieldCount: number;
fieldsInSameFamily: number;
}) =>
export const INCOMPATIBLE_CALLOUT_TITLE = (fieldCount: number) =>
i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCalloutTitle',
{
values: { fieldCount, fieldsInSameFamily },
defaultMessage:
'{fieldCount} incompatible {fieldCount, plural, =1 {field} other {fields}}, {fieldsInSameFamily} {fieldsInSameFamily, plural, =1 {field} other {fields}} with {fieldsInSameFamily, plural, =1 {a mapping} other {mappings}} in the same family',
values: { fieldCount },
defaultMessage: '{fieldCount} incompatible {fieldCount, plural, =1 {field} other {fields}}',
}
);
@ -391,6 +404,21 @@ export const OTHER_APP_CAPABILITIES_WORK_PROPERLY = i18n.translate(
}
);
export const SAME_FAMILY_EMPTY = i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.sameFamilyEmptyContent',
{
defaultMessage:
'All of the field mappings and document values in this index are compliant with the Elastic Common Schema (ECS).',
}
);
export const SAME_FAMILY_EMPTY_TITLE = i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.sameFamilyEmptyTitle',
{
defaultMessage: 'All field mappings and values are ECS compliant',
}
);
export const PAGES_DISPLAY_EVENTS = i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.pagesDisplayEventsMessage',
{
@ -428,6 +456,13 @@ export const ECS_IS_A_PERMISSIVE_SCHEMA = i18n.translate(
}
);
export const SAME_FAMILY = i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.sameFamilyTab',
{
defaultMessage: 'Same family',
}
);
export const SOMETIMES_INDICES_CREATED_BY_OLDER = i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.sometimesIndicesCreatedByOlderDescription',
{

1
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/pattern/helpers.test.ts
View file

@ -273,6 +273,7 @@ describe('helpers', () => {
'\n#### Incompatible field mappings - auditbeat-custom-index-1\n\n\n| Field | ECS mapping type (expected) | Index mapping type (actual) | \n|-------|-----------------------------|-----------------------------|\n| host.name | `keyword` | `text` |\n| source.ip | `ip` | `text` |\n\n#### Incompatible field values - auditbeat-custom-index-1\n\n\n| Field | ECS values (expected) | Document values (actual) | \n|-------|-----------------------|--------------------------|\n| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `an_invalid_category` (2),\n`theory` (1) |\n\n',
],
pattern: 'auditbeat-*',
sameFamily: 0,
},
};
const isILMAvailable = true;

25
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/stat_label/translations.ts
View file

@ -102,6 +102,23 @@ export const INDICES = i18n.translate(
}
);
export const SAME_FAMILY = i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.statLabels.sameFamilyLabel',
{
defaultMessage: 'Same family',
}
);
export const SAME_FAMILY_PATTERN_TOOL_TIP = (pattern: string) =>
i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.statLabels.sameFamilyPatternToolTip',
{
values: { pattern },
defaultMessage:
'The total count of fields in the same family as the type specified by ECS, in indices matching the {pattern} pattern',
}
);
export const SIZE = i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.statLabels.sizeLabel',
{
@ -166,6 +183,14 @@ export const TOTAL_INDICES_TOOL_TIP = i18n.translate(
}
);
export const TOTAL_SAME_FAMILY_TOOL_TIP = i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.statLabels.totalSameFamilyToolTip',
{
defaultMessage:
'The total count of fields in the same family as the ECS type, in all indices that were checked',
}
);
export const TOTAL_SIZE_TOOL_TIP = i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.statLabels.totalSizeToolTip',
{

28
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/tabs/callouts/incompatible_callout/index.test.tsx
View file

@ -11,10 +11,8 @@ import React from 'react';
import {
DETECTION_ENGINE_RULES_MAY_NOT_MATCH,
INCOMPATIBLE_FIELDS_WITH,
MAPPINGS_THAT_CONFLICT_WITH_ECS,
PAGES_MAY_NOT_DISPLAY_EVENTS,
WHEN_AN_INCOMPATIBLE_FIELD,
} from '../../../index_properties/translations';
import {
eventCategory,
@ -28,7 +26,7 @@ import { IncompatibleCallout } from '.';
const content = 'Is your name Michael?';
const eventCategoryWithWIldcard: EnrichedFieldMetadata = {
const eventCategoryWithWildcard: EnrichedFieldMetadata = {
...eventCategory, // `event.category` is a `keyword` per the ECS spec
indexFieldType: 'wildcard', // this index has a mapping of `wildcard` instead of `keyword`
isInSameFamily: true, // `wildcard` and `keyword` are in the same family
@ -41,10 +39,10 @@ describe('IncompatibleCallout', () => {
<TestProviders>
<IncompatibleCallout
enrichedFieldMetadata={[
eventCategoryWithWIldcard, // isEcsCompliant: false, indexInvalidValues.length: 0, isInSameFamily: true, `wildcard` and `keyword` are in the same family
eventCategoryWithUnallowedValues, // isEcsCompliant: false, indexInvalidValues.length: 2, isInSameFamily: true, `keyword` and `keyword` are in the same family
hostNameWithTextMapping, // isEcsCompliant: false, indexInvalidValues.length: 0, isInSameFamily: false, `keyword` and `text` are not in the family
sourceIpWithTextMapping, // isEcsCompliant: false, indexInvalidValues.length: 0, isInSameFamily: false, `ip` is not a member of any families
eventCategoryWithWildcard, // `wildcard` and `keyword`
eventCategoryWithUnallowedValues, // `keyword` and `keyword`
hostNameWithTextMapping, // `keyword` and `text`
sourceIpWithTextMapping, // `ip` is not a member of any families
]}
>
<div data-test-subj="children">{content}</div>
@ -54,9 +52,7 @@ describe('IncompatibleCallout', () => {
});
test('it renders a title with the expected incompatible and family counts', () => {
expect(screen.getByTestId('title')).toHaveTextContent(
'4 incompatible fields, 1 field with a mapping in the same family'
);
expect(screen.getByTestId('title')).toHaveTextContent('4 incompatible fields');
});
test('it includes the ECS version in the main content', () => {
@ -65,18 +61,6 @@ describe('IncompatibleCallout', () => {
);
});
test('it notes that mappings in the same family have the same search, but different space and performance characteristics', () => {
expect(screen.getByTestId('incompatibleFIeldsWith')).toHaveTextContent(
INCOMPATIBLE_FIELDS_WITH
);
});
test('it introduces the consequences of having incompatible fields not in the same family', () => {
expect(screen.getByTestId('whenAnIncompatableField')).toHaveTextContent(
WHEN_AN_INCOMPATIBLE_FIELD
);
});
test('it warns rules may not match', () => {
expect(screen.getByTestId('rulesMayNotMatch')).toHaveTextContent(
DETECTION_ENGINE_RULES_MAY_NOT_MATCH

30
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/tabs/callouts/incompatible_callout/index.tsx
View file

@ -7,10 +7,9 @@
import { EcsVersion } from '@kbn/ecs';
import { EuiCallOut, EuiSpacer, EuiText } from '@elastic/eui';
import { EuiCallOut, EuiSpacer } from '@elastic/eui';
import React, { useMemo } from 'react';
import { getIncompatiableFieldsInSameFamilyCount } from './helpers';
import * as i18n from '../../../index_properties/translations';
import { CalloutItem } from '../../styles';
import type { EnrichedFieldMetadata } from '../../../../types';
@ -22,36 +21,15 @@ interface Props {
const IncompatibleCalloutComponent: React.FC<Props> = ({ children, enrichedFieldMetadata }) => {
const fieldCount = enrichedFieldMetadata.length;
const fieldsInSameFamily = getIncompatiableFieldsInSameFamilyCount(enrichedFieldMetadata);
const title = useMemo(
() => (
<span data-test-subj="title">
{i18n.INCOMPATIBLE_CALLOUT_TITLE({ fieldCount, fieldsInSameFamily })}
</span>
),
[fieldCount, fieldsInSameFamily]
() => <span data-test-subj="title">{i18n.INCOMPATIBLE_CALLOUT_TITLE(fieldCount)}</span>,
[fieldCount]
);
return (
<EuiCallOut color="danger" data-test-subj="incompatibleCallout" size="s" title={title}>
<div data-test-subj="fieldsAreIncompatible">
{i18n.INCOMPATIBLE_CALLOUT({
fieldCount: enrichedFieldMetadata.length,
version: EcsVersion,
})}
</div>
<EuiSpacer size="s" />
<div>
<EuiText data-test-subj="incompatibleFIeldsWith" size="xs">
{i18n.INCOMPATIBLE_FIELDS_WITH}
</EuiText>
</div>
<div data-test-subj="fieldsAreIncompatible">{i18n.INCOMPATIBLE_CALLOUT(EcsVersion)}</div>
<EuiSpacer size="xs" />
<div>
<EuiText data-test-subj="whenAnIncompatableField" size="xs">
{i18n.WHEN_AN_INCOMPATIBLE_FIELD}
</EuiText>
</div>
<CalloutItem data-test-subj="rulesMayNotMatch">
{i18n.DETECTION_ENGINE_RULES_MAY_NOT_MATCH}
</CalloutItem>

51
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/tabs/callouts/same_family_callout/index.test.tsx Normal file
View file

@ -0,0 +1,51 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { EcsVersion } from '@kbn/ecs';
import { render, screen } from '@testing-library/react';
import React from 'react';
import { FIELDS_WITH_MAPPINGS_SAME_FAMILY } from '../../../index_properties/translations';
import { TestProviders } from '../../../../mock/test_providers/test_providers';
import { SameFamilyCallout } from '.';
import { mockPartitionedFieldMetadataWithSameFamily } from '../../../../mock/partitioned_field_metadata/mock_partitioned_field_metadata_with_same_family';
const content = 'you are reviewing a pull request';
describe('SameFamilyCallout', () => {
beforeEach(() => {
render(
<TestProviders>
<SameFamilyCallout
enrichedFieldMetadata={mockPartitionedFieldMetadataWithSameFamily.sameFamily}
>
<div data-test-subj="children">{content}</div>
</SameFamilyCallout>
</TestProviders>
);
});
test('it renders a title with the expected count of same family field mappings', () => {
expect(screen.getByTestId('title')).toHaveTextContent('1 Same family field mapping');
});
test('it includes the ECS version in the main content', () => {
expect(screen.getByTestId('fieldsDefinedByEcs')).toHaveTextContent(
`This field is defined by the Elastic Common Schema (ECS), version ${EcsVersion}, but its mapping type doesn't exactly match.`
);
});
test('it notes fields with mappings have the same behavior, but may have different space usage or performance characteristics', () => {
expect(screen.getByTestId('fieldsWithMappingsSameFamily')).toHaveTextContent(
FIELDS_WITH_MAPPINGS_SAME_FAMILY
);
});
test('it renders the children', () => {
expect(screen.getByTestId('children')).toHaveTextContent(content);
});
});

49
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/tabs/callouts/same_family_callout/index.tsx Normal file
View file

@ -0,0 +1,49 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { EcsVersion } from '@kbn/ecs';
import { EuiCallOut, EuiSpacer, EuiText } from '@elastic/eui';
import React, { useMemo } from 'react';
import * as i18n from '../../../index_properties/translations';
import type { EnrichedFieldMetadata } from '../../../../types';
interface Props {
children?: React.ReactNode;
enrichedFieldMetadata: EnrichedFieldMetadata[];
}
const SameFamilyCalloutComponent: React.FC<Props> = ({ children, enrichedFieldMetadata }) => {
const title = useMemo(
() => (
<span data-test-subj="title">
{i18n.SAME_FAMILY_CALLOUT_TITLE(enrichedFieldMetadata.length)}
</span>
),
[enrichedFieldMetadata.length]
);
return (
<EuiCallOut color="primary" size="s" title={title}>
<div data-test-subj="fieldsDefinedByEcs">
{i18n.SAME_FAMILY_CALLOUT({
fieldCount: enrichedFieldMetadata.length,
version: EcsVersion,
})}
</div>
<EuiSpacer size="s" />
<div>
<EuiText data-test-subj="fieldsWithMappingsSameFamily" size="xs">
{i18n.FIELDS_WITH_MAPPINGS_SAME_FAMILY}
</EuiText>
</div>
{children}
</EuiCallOut>
);
};
export const SameFamilyCallout = React.memo(SameFamilyCalloutComponent);

4
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/tabs/custom_tab/helpers.test.ts
View file

@ -87,7 +87,7 @@ ${ECS_IS_A_PERMISSIVE_SCHEMA}
).toEqual([
'### auditbeat-custom-index-1\n',
'| Result | Index | Docs | Incompatible fields | ILM Phase | Size |\n|--------|-------|------|---------------------|-----------|------|\n| ❌ | auditbeat-custom-index-1 | 4 (0.0%) | 3 | `unmanaged` | 27.7KB |\n\n',
'### **Incompatible fields** `3` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
'### **Incompatible fields** `3` **Same family** `0` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
`#### 4 Custom field mappings\n\nThese fields are not defined by the Elastic Common Schema (ECS), version ${EcsVersion}.\n\nECS is a permissive schema. If your events have additional data that cannot be mapped to ECS, you can simply add them to your events, using custom field names.\n`,
'#### Custom fields - auditbeat-custom-index-1\n\n\n| Field | Index mapping type | \n|-------|--------------------|\n| host.name.keyword | `keyword` | `--` |\n| some.field | `text` | `--` |\n| some.field.keyword | `keyword` | `--` |\n| source.ip.keyword | `keyword` | `--` |\n',
]);
@ -109,7 +109,7 @@ ${ECS_IS_A_PERMISSIVE_SCHEMA}
).toEqual([
'### auditbeat-custom-index-1\n',
'| Result | Index | Docs | Incompatible fields | Size |\n|--------|-------|------|---------------------|------|\n| ❌ | auditbeat-custom-index-1 | 4 (0.0%) | 3 | 27.7KB |\n\n',
'### **Incompatible fields** `3` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
'### **Incompatible fields** `3` **Same family** `0` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
`#### 4 Custom field mappings\n\nThese fields are not defined by the Elastic Common Schema (ECS), version ${EcsVersion}.\n\nECS is a permissive schema. If your events have additional data that cannot be mapped to ECS, you can simply add them to your events, using custom field names.\n`,
'#### Custom fields - auditbeat-custom-index-1\n\n\n| Field | Index mapping type | \n|-------|--------------------|\n| host.name.keyword | `keyword` | `--` |\n| some.field | `text` | `--` |\n| some.field.keyword | `keyword` | `--` |\n| source.ip.keyword | `keyword` | `--` |\n',
]);

4
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/tabs/helpers.test.tsx
View file

@ -95,6 +95,10 @@ describe('helpers', () => {
id: 'incompatibleTab',
name: 'Incompatible fields',
},
{
id: 'sameFamilyTab',
name: 'Same family',
},
{
id: 'customTab',
name: 'Custom fields',

28
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/tabs/helpers.tsx
View file

@ -24,20 +24,22 @@ import { AllTab } from './all_tab';
import { CustomTab } from './custom_tab';
import { getCustomColor } from './custom_tab/helpers';
import { EcsCompliantTab } from './ecs_compliant_tab';
import { getSizeInBytes } from '../../helpers';
import { IncompatibleTab } from './incompatible_tab';
import { getIncompatibleColor } from './incompatible_tab/helpers';
import { getIncompatibleColor, getSameFamilyColor } from './incompatible_tab/helpers';
import {
ALL_TAB_ID,
ECS_COMPLIANT_TAB_ID,
INCOMPATIBLE_TAB_ID,
SAME_FAMILY_TAB_ID,
SUMMARY_TAB_ID,
} from '../index_properties/helpers';
import { getMarkdownComment } from '../index_properties/markdown/helpers';
import { getFillColor } from './summary_tab/helpers';
import * as i18n from '../index_properties/translations';
import { SameFamilyTab } from './same_family_tab';
import { SummaryTab } from './summary_tab';
import { getFillColor } from './summary_tab/helpers';
import type { EnrichedFieldMetadata, IlmPhase, PartitionedFieldMetadata } from '../../types';
import { getSizeInBytes } from '../../helpers';
export const getMissingTimestampComment = (): string =>
getMarkdownComment({
@ -157,6 +159,26 @@ export const getTabs = ({
id: INCOMPATIBLE_TAB_ID,
name: i18n.INCOMPATIBLE_FIELDS,
},
{
append: (
<EuiBadge color={getSameFamilyColor()}>{partitionedFieldMetadata.sameFamily.length}</EuiBadge>
),
content: (
<SameFamilyTab
addSuccessToast={addSuccessToast}
docsCount={docsCount}
formatBytes={formatBytes}
formatNumber={formatNumber}
ilmPhase={ilmPhase}
indexName={indexName}
partitionedFieldMetadata={partitionedFieldMetadata}
patternDocsCount={patternDocsCount}
sizeInBytes={getSizeInBytes({ indexName, stats })}
/>
),
id: SAME_FAMILY_TAB_ID,
name: i18n.SAME_FAMILY,
},
{
append: (
<EuiBadge color={getCustomColor(partitionedFieldMetadata)}>

29
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/tabs/incompatible_tab/helpers.test.ts
View file

@ -18,15 +18,14 @@ import {
getIncompatibleMappingsFields,
getIncompatibleValues,
getIncompatibleValuesFields,
getSameFamilyColor,
showInvalidCallout,
} from './helpers';
import { EMPTY_STAT } from '../../../helpers';
import {
DETECTION_ENGINE_RULES_MAY_NOT_MATCH,
INCOMPATIBLE_FIELDS_WITH,
MAPPINGS_THAT_CONFLICT_WITH_ECS,
PAGES_MAY_NOT_DISPLAY_EVENTS,
WHEN_AN_INCOMPATIBLE_FIELD,
} from '../../index_properties/translations';
import { mockPartitionedFieldMetadata } from '../../../mock/partitioned_field_metadata/mock_partitioned_field_metadata';
import { PartitionedFieldMetadata } from '../../../types';
@ -34,18 +33,10 @@ import { PartitionedFieldMetadata } from '../../../types';
describe('helpers', () => {
describe('getIncompatibleFieldsMarkdownComment', () => {
test('it returns the expected counts and ECS version', () => {
expect(
getIncompatibleFieldsMarkdownComment({
fieldsInSameFamily: 7,
incompatible: 11,
})
).toEqual(`#### 11 incompatible fields, 7 fields with mappings in the same family
expect(getIncompatibleFieldsMarkdownComment(11)).toEqual(`#### 11 incompatible fields
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version ${EcsVersion}.
${INCOMPATIBLE_FIELDS_WITH}
${WHEN_AN_INCOMPATIBLE_FIELD}
${DETECTION_ENGINE_RULES_MAY_NOT_MATCH}
${PAGES_MAY_NOT_DISPLAY_EVENTS}
${MAPPINGS_THAT_CONFLICT_WITH_ECS}
@ -69,6 +60,12 @@ ${MAPPINGS_THAT_CONFLICT_WITH_ECS}
});
});
describe('getSameFamilyColor', () => {
test('it returns the expected color', () => {
expect(getSameFamilyColor()).toEqual(euiThemeVars.euiColorLightShade);
});
});
describe('getIncompatibleMappings', () => {
test('it (only) returns the mappings where type !== indexFieldType', () => {
expect(getIncompatibleMappings(mockPartitionedFieldMetadata.incompatible)).toEqual([
@ -284,7 +281,7 @@ ${MAPPINGS_THAT_CONFLICT_WITH_ECS}
],
hasEcsMetadata: true,
isEcsCompliant: false,
isInSameFamily: true,
isInSameFamily: false,
},
]);
});
@ -358,8 +355,8 @@ ${MAPPINGS_THAT_CONFLICT_WITH_ECS}
).toEqual([
'### auditbeat-custom-index-1\n',
'| Result | Index | Docs | Incompatible fields | ILM Phase | Size |\n|--------|-------|------|---------------------|-----------|------|\n| ❌ | auditbeat-custom-index-1 | 4 (0.0%) | 3 | `unmanaged` | 27.7KB |\n\n',
'### **Incompatible fields** `3` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
`#### 3 incompatible fields, 0 fields with mappings in the same family\n\nFields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version ${EcsVersion}.\n\n${INCOMPATIBLE_FIELDS_WITH}\n\n${WHEN_AN_INCOMPATIBLE_FIELD}\n${DETECTION_ENGINE_RULES_MAY_NOT_MATCH}\n${PAGES_MAY_NOT_DISPLAY_EVENTS}\n${MAPPINGS_THAT_CONFLICT_WITH_ECS}\n`,
'### **Incompatible fields** `3` **Same family** `0` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
`#### 3 incompatible fields\n\nFields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version ${EcsVersion}.\n\n${DETECTION_ENGINE_RULES_MAY_NOT_MATCH}\n${PAGES_MAY_NOT_DISPLAY_EVENTS}\n${MAPPINGS_THAT_CONFLICT_WITH_ECS}\n`,
'\n#### Incompatible field mappings - auditbeat-custom-index-1\n\n\n| Field | ECS mapping type (expected) | Index mapping type (actual) | \n|-------|-----------------------------|-----------------------------|\n| host.name | `keyword` | `text` |\n| source.ip | `ip` | `text` |\n\n#### Incompatible field values - auditbeat-custom-index-1\n\n\n| Field | ECS values (expected) | Document values (actual) | \n|-------|-----------------------|--------------------------|\n| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `an_invalid_category` (2), `theory` (1) |\n\n',
]);
});
@ -385,7 +382,7 @@ ${MAPPINGS_THAT_CONFLICT_WITH_ECS}
).toEqual([
'### auditbeat-custom-index-1\n',
'| Result | Index | Docs | Incompatible fields | ILM Phase | Size |\n|--------|-------|------|---------------------|-----------|------|\n| ✅ | auditbeat-custom-index-1 | 4 (0.0%) | 0 | `unmanaged` | 27.7KB |\n\n',
'### **Incompatible fields** `0` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
'### **Incompatible fields** `0` **Same family** `0` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
'\n\n\n',
]);
});
@ -411,7 +408,7 @@ ${MAPPINGS_THAT_CONFLICT_WITH_ECS}
).toEqual([
'### auditbeat-custom-index-1\n',
'| Result | Index | Docs | Incompatible fields | Size |\n|--------|-------|------|---------------------|------|\n| ✅ | auditbeat-custom-index-1 | 4 (0.0%) | 0 | 27.7KB |\n\n',
'### **Incompatible fields** `0` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
'### **Incompatible fields** `0` **Same family** `0` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
'\n\n\n',
]);
});

37
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/tabs/incompatible_tab/helpers.ts
View file

@ -7,7 +7,6 @@
import { EcsVersion } from '@kbn/ecs';
import { getIncompatiableFieldsInSameFamilyCount } from '../callouts/incompatible_callout/helpers';
import {
getSummaryMarkdownComment,
getIncompatibleMappingsMarkdownTableRows,
@ -32,28 +31,17 @@ import {
DOCUMENT_VALUES_ACTUAL,
ECS_VALUES_EXPECTED,
} from '../../../compare_fields_table/translations';
import { getIsInSameFamily } from '../../../helpers';
export const getIncompatibleFieldsMarkdownComment = ({
fieldsInSameFamily,
incompatible,
}: {
fieldsInSameFamily: number;
incompatible: number;
}): string =>
export const getIncompatibleFieldsMarkdownComment = (incompatible: number): string =>
getMarkdownComment({
suggestedAction: `${i18n.INCOMPATIBLE_CALLOUT({
fieldCount: incompatible,
version: EcsVersion,
})}
suggestedAction: `${i18n.INCOMPATIBLE_CALLOUT(EcsVersion)}
${i18n.INCOMPATIBLE_FIELDS_WITH}
${i18n.WHEN_AN_INCOMPATIBLE_FIELD}
${i18n.DETECTION_ENGINE_RULES_MAY_NOT_MATCH}
${i18n.PAGES_MAY_NOT_DISPLAY_EVENTS}
${i18n.MAPPINGS_THAT_CONFLICT_WITH_ECS}
`,
title: i18n.INCOMPATIBLE_CALLOUT_TITLE({ fieldCount: incompatible, fieldsInSameFamily }),
title: i18n.INCOMPATIBLE_CALLOUT_TITLE(incompatible),
});
export const showInvalidCallout = (enrichedFieldMetadata: EnrichedFieldMetadata[]): boolean =>
@ -61,10 +49,17 @@ export const showInvalidCallout = (enrichedFieldMetadata: EnrichedFieldMetadata[
export const getIncompatibleColor = (): string => getFillColor('incompatible');
export const getSameFamilyColor = (): string => getFillColor('same-family');
export const getIncompatibleMappings = (
enrichedFieldMetadata: EnrichedFieldMetadata[]
): EnrichedFieldMetadata[] =>
enrichedFieldMetadata.filter((x) => !x.isEcsCompliant && x.type !== x.indexFieldType);
enrichedFieldMetadata.filter(
(x) =>
!x.isEcsCompliant &&
x.type !== x.indexFieldType &&
!getIsInSameFamily({ ecsExpectedType: x.type, type: x.indexFieldType })
);
export const getIncompatibleMappingsFields = (
enrichedFieldMetadata: EnrichedFieldMetadata[]
@ -151,16 +146,10 @@ export const getAllIncompatibleMarkdownComments = ({
}): string[] => {
const incompatibleMappings = getIncompatibleMappings(partitionedFieldMetadata.incompatible);
const incompatibleValues = getIncompatibleValues(partitionedFieldMetadata.incompatible);
const fieldsInSameFamily = getIncompatiableFieldsInSameFamilyCount(
partitionedFieldMetadata.incompatible
);
const incompatibleFieldsMarkdownComment =
partitionedFieldMetadata.incompatible.length > 0
? getIncompatibleFieldsMarkdownComment({
fieldsInSameFamily,
incompatible: partitionedFieldMetadata.incompatible.length,
})
? getIncompatibleFieldsMarkdownComment(partitionedFieldMetadata.incompatible.length)
: '';
return [

126
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/tabs/same_family_tab/helpers.test.ts Normal file
View file

@ -0,0 +1,126 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import numeral from '@elastic/numeral';
import { EcsVersion } from '@kbn/ecs';
import {
getAllSameFamilyMarkdownComments,
getSameFamilyMarkdownComment,
getSameFamilyMarkdownTablesComment,
} from './helpers';
import { EMPTY_STAT } from '../../../helpers';
import { mockPartitionedFieldMetadata } from '../../../mock/partitioned_field_metadata/mock_partitioned_field_metadata';
import { mockPartitionedFieldMetadataWithSameFamily } from '../../../mock/partitioned_field_metadata/mock_partitioned_field_metadata_with_same_family';
describe('helpers', () => {
describe('getSameFamilyMarkdownComment', () => {
test('it returns the expected counts and ECS version', () => {
expect(getSameFamilyMarkdownComment(7)).toEqual(`#### 7 Same family field mappings
These fields are defined by the Elastic Common Schema (ECS), version ${EcsVersion}, but their mapping types don't exactly match.
Fields with mappings in the same family have exactly the same search behavior as the type specified by ECS, but may have different space usage or performance characteristics.
`);
});
});
describe('getSameFamilyMarkdownTablesComment', () => {
test('it returns the expected comment when the index has same family mappings', () => {
const expected = `\n#### Same family field mappings - auditbeat-custom-index-1\n\n\n| Field | ECS mapping type (expected) | Index mapping type (actual) | \n|-------|-----------------------------|-----------------------------|\n| agent.type | \`keyword\` | \`constant_keyword\` \`same family\` |\n\n`;
expect(
getSameFamilyMarkdownTablesComment({
sameFamilyMappings: [
{
dashed_name: 'agent-type',
description:
'Type of the agent.\nThe agent type always stays the same and should be given by the agent used. In case of Filebeat the agent would always be Filebeat also if two Filebeat instances are run on the same machine.',
example: 'filebeat',
flat_name: 'agent.type',
ignore_above: 1024,
level: 'core',
name: 'type',
normalize: [],
short: 'Type of the agent.',
type: 'keyword',
indexFieldName: 'agent.type',
indexFieldType: 'constant_keyword',
indexInvalidValues: [],
hasEcsMetadata: true,
isEcsCompliant: false,
isInSameFamily: true,
},
],
indexName: 'auditbeat-custom-index-1',
})
).toEqual(expected);
});
test('it returns the expected comment when the index does NOT have same family mappings', () => {
expect(
getSameFamilyMarkdownTablesComment({
sameFamilyMappings: [],
indexName: 'auditbeat-custom-index-1',
})
).toEqual('\n\n');
});
});
describe('getAllSameFamilyMarkdownComments', () => {
const defaultBytesFormat = '0,0.[0]b';
const formatBytes = (value: number | undefined) =>
value != null ? numeral(value).format(defaultBytesFormat) : EMPTY_STAT;
const defaultNumberFormat = '0,0.[000]';
const formatNumber = (value: number | undefined) =>
value != null ? numeral(value).format(defaultNumberFormat) : EMPTY_STAT;
test('it returns the expected collection of comments', () => {
expect(
getAllSameFamilyMarkdownComments({
docsCount: 4,
formatBytes,
formatNumber,
ilmPhase: 'unmanaged',
indexName: 'auditbeat-custom-index-1',
isILMAvailable: true,
partitionedFieldMetadata: mockPartitionedFieldMetadataWithSameFamily,
patternDocsCount: 57410,
sizeInBytes: 28413,
})
).toEqual([
'### auditbeat-custom-index-1\n',
'| Result | Index | Docs | Incompatible fields | ILM Phase | Size |\n|--------|-------|------|---------------------|-----------|------|\n| ❌ | auditbeat-custom-index-1 | 4 (0.0%) | 3 | `unmanaged` | 27.7KB |\n\n',
'### **Incompatible fields** `3` **Same family** `1` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `10`\n',
`#### 1 Same family field mapping\n\nThis field is defined by the Elastic Common Schema (ECS), version ${EcsVersion}, but its mapping type doesn't exactly match.\n\nFields with mappings in the same family have exactly the same search behavior as the type specified by ECS, but may have different space usage or performance characteristics.\n`,
'\n#### Same family field mappings - auditbeat-custom-index-1\n\n\n| Field | ECS mapping type (expected) | Index mapping type (actual) | \n|-------|-----------------------------|-----------------------------|\n| agent.type | `keyword` | `constant_keyword` `same family` |\n\n',
]);
});
test('it returns the expected comment when `sameFamily` is empty', () => {
expect(
getAllSameFamilyMarkdownComments({
docsCount: 4,
formatBytes,
formatNumber,
ilmPhase: 'unmanaged',
indexName: 'auditbeat-custom-index-1',
isILMAvailable: true,
partitionedFieldMetadata: mockPartitionedFieldMetadata,
patternDocsCount: 57410,
sizeInBytes: 28413,
})
).toEqual([
'### auditbeat-custom-index-1\n',
'| Result | Index | Docs | Incompatible fields | ILM Phase | Size |\n|--------|-------|------|---------------------|-----------|------|\n| ❌ | auditbeat-custom-index-1 | 4 (0.0%) | 3 | `unmanaged` | 27.7KB |\n\n',
'### **Incompatible fields** `3` **Same family** `0` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
'\n\n',
]);
});
});
});

111
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/tabs/same_family_tab/helpers.ts Normal file
View file

@ -0,0 +1,111 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { EcsVersion } from '@kbn/ecs';
import {
FIELD,
ECS_MAPPING_TYPE_EXPECTED,
INDEX_MAPPING_TYPE_ACTUAL,
} from '../../../compare_fields_table/translations';
import {
getSummaryMarkdownComment,
getIncompatibleMappingsMarkdownTableRows,
getMarkdownComment,
getMarkdownTable,
getSummaryTableMarkdownComment,
getTabCountsMarkdownComment,
} from '../../index_properties/markdown/helpers';
import * as i18n from '../../index_properties/translations';
import { SAME_FAMILY_FIELD_MAPPINGS_TABLE_TITLE } from './translations';
import type { EnrichedFieldMetadata, IlmPhase, PartitionedFieldMetadata } from '../../../types';
export const getSameFamilyMarkdownComment = (fieldsInSameFamily: number): string =>
getMarkdownComment({
suggestedAction: `${i18n.SAME_FAMILY_CALLOUT({
fieldCount: fieldsInSameFamily,
version: EcsVersion,
})}
${i18n.FIELDS_WITH_MAPPINGS_SAME_FAMILY}
`,
title: i18n.SAME_FAMILY_CALLOUT_TITLE(fieldsInSameFamily),
});
export const getSameFamilyMappings = (
enrichedFieldMetadata: EnrichedFieldMetadata[]
): EnrichedFieldMetadata[] => enrichedFieldMetadata.filter((x) => x.isInSameFamily);
export const getSameFamilyMarkdownTablesComment = ({
sameFamilyMappings,
indexName,
}: {
sameFamilyMappings: EnrichedFieldMetadata[];
indexName: string;
}): string => `
${
sameFamilyMappings.length > 0
? getMarkdownTable({
enrichedFieldMetadata: sameFamilyMappings,
getMarkdownTableRows: getIncompatibleMappingsMarkdownTableRows,
headerNames: [FIELD, ECS_MAPPING_TYPE_EXPECTED, INDEX_MAPPING_TYPE_ACTUAL],
title: SAME_FAMILY_FIELD_MAPPINGS_TABLE_TITLE(indexName),
})
: ''
}
`;
export const getAllSameFamilyMarkdownComments = ({
docsCount,
formatBytes,
formatNumber,
ilmPhase,
indexName,
isILMAvailable,
partitionedFieldMetadata,
patternDocsCount,
sizeInBytes,
}: {
docsCount: number;
formatBytes: (value: number | undefined) => string;
formatNumber: (value: number | undefined) => string;
ilmPhase: IlmPhase | undefined;
indexName: string;
isILMAvailable: boolean;
partitionedFieldMetadata: PartitionedFieldMetadata;
patternDocsCount: number;
sizeInBytes: number | undefined;
}): string[] => {
const sameFamilyMappings = getSameFamilyMappings(partitionedFieldMetadata.sameFamily);
const fieldsInSameFamily = partitionedFieldMetadata.sameFamily.length;
const incompatibleFieldsMarkdownComment =
partitionedFieldMetadata.sameFamily.length > 0
? getSameFamilyMarkdownComment(fieldsInSameFamily)
: '';
return [
getSummaryMarkdownComment(indexName),
getSummaryTableMarkdownComment({
docsCount,
formatBytes,
formatNumber,
ilmPhase,
indexName,
isILMAvailable,
partitionedFieldMetadata,
patternDocsCount,
sizeInBytes,
}),
getTabCountsMarkdownComment(partitionedFieldMetadata),
incompatibleFieldsMarkdownComment,
getSameFamilyMarkdownTablesComment({
sameFamilyMappings,
indexName,
}),
].filter((x) => x !== '');
};

111
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/tabs/same_family_tab/index.tsx Normal file
View file

@ -0,0 +1,111 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { copyToClipboard, EuiButtonEmpty, EuiSpacer } from '@elastic/eui';
import React, { useCallback, useMemo } from 'react';
import { SameFamilyCallout } from '../callouts/same_family_callout';
import { CompareFieldsTable } from '../../../compare_fields_table';
import { getIncompatibleMappingsTableColumns } from '../../../compare_fields_table/get_incompatible_mappings_table_columns';
import { useDataQualityContext } from '../../data_quality_context';
import { getAllSameFamilyMarkdownComments, getSameFamilyMappings } from './helpers';
import * as i18n from '../../index_properties/translations';
import { SAME_FAMILY_FIELD_MAPPINGS_TABLE_TITLE } from './translations';
import { COPIED_RESULTS_TOAST_TITLE } from '../../../translations';
import type { IlmPhase, PartitionedFieldMetadata } from '../../../types';
interface Props {
addSuccessToast: (toast: { title: string }) => void;
docsCount: number;
formatBytes: (value: number | undefined) => string;
formatNumber: (value: number | undefined) => string;
ilmPhase: IlmPhase | undefined;
indexName: string;
partitionedFieldMetadata: PartitionedFieldMetadata;
patternDocsCount: number;
sizeInBytes: number | undefined;
}
const SameFamilyTabComponent: React.FC<Props> = ({
addSuccessToast,
docsCount,
formatBytes,
formatNumber,
ilmPhase,
indexName,
partitionedFieldMetadata,
patternDocsCount,
sizeInBytes,
}) => {
const sameFamilyMappings = useMemo(
() => getSameFamilyMappings(partitionedFieldMetadata.sameFamily),
[partitionedFieldMetadata.sameFamily]
);
const { isILMAvailable } = useDataQualityContext();
const markdownComments: string[] = useMemo(
() =>
getAllSameFamilyMarkdownComments({
docsCount,
formatBytes,
formatNumber,
ilmPhase,
indexName,
isILMAvailable,
partitionedFieldMetadata,
patternDocsCount,
sizeInBytes,
}),
[
docsCount,
formatBytes,
formatNumber,
ilmPhase,
indexName,
isILMAvailable,
partitionedFieldMetadata,
patternDocsCount,
sizeInBytes,
]
);
const onCopy = useCallback(() => {
copyToClipboard(markdownComments.join('\n'));
addSuccessToast({
title: COPIED_RESULTS_TOAST_TITLE,
});
}, [addSuccessToast, markdownComments]);
return (
<div data-test-subj="sameFamilyTab">
<SameFamilyCallout enrichedFieldMetadata={partitionedFieldMetadata.sameFamily}>
<EuiButtonEmpty aria-label={i18n.COPY_TO_CLIPBOARD} flush="both" onClick={onCopy}>
{i18n.COPY_TO_CLIPBOARD}
</EuiButtonEmpty>
</SameFamilyCallout>
<>
{sameFamilyMappings.length > 0 && (
<>
<EuiSpacer />
<CompareFieldsTable
enrichedFieldMetadata={sameFamilyMappings}
getTableColumns={getIncompatibleMappingsTableColumns}
title={SAME_FAMILY_FIELD_MAPPINGS_TABLE_TITLE(indexName)}
/>
</>
)}
</>
</div>
);
};
SameFamilyTabComponent.displayName = 'SameFamilyTabComponent';
export const SameFamilyTab = React.memo(SameFamilyTabComponent);

17
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/tabs/same_family_tab/translations.ts Normal file
View file

@ -0,0 +1,17 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { i18n } from '@kbn/i18n';
export const SAME_FAMILY_FIELD_MAPPINGS_TABLE_TITLE = (indexName: string) =>
i18n.translate(
'securitySolutionPackages.ecsDataQualityDashboard.sameFamilyTab.sameFamilyFieldMappingsTableTitle',
{
values: { indexName },
defaultMessage: 'Same family field mappings - {indexName}',
}
);

23
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/tabs/summary_tab/helpers.test.ts
View file

@ -17,6 +17,7 @@ import {
CUSTOM_TAB_ID,
ECS_COMPLIANT_TAB_ID,
INCOMPATIBLE_TAB_ID,
SAME_FAMILY_TAB_ID,
} from '../../index_properties/helpers';
import {
CUSTOM_FIELDS,
@ -24,6 +25,7 @@ import {
INCOMPATIBLE_FIELDS,
UNKNOWN,
} from '../../index_properties/translations';
import { SAME_FAMILY } from '../../stat_label/translations';
import {
CategoryId,
getFillColor,
@ -63,6 +65,10 @@ describe('helpers', () => {
categoryId: 'ecs-compliant',
expectedColor: euiThemeVars.euiColorSuccess,
},
{
categoryId: 'same-family',
expectedColor: euiThemeVars.euiColorLightShade,
},
{
categoryId: invalid,
expectedColor: euiThemeVars.euiColorGhost,
@ -95,6 +101,10 @@ describe('helpers', () => {
categoryId: 'ecs-compliant',
expectedLabel: ECS_COMPLIANT_FIELDS,
},
{
categoryId: 'same-family',
expectedLabel: SAME_FAMILY,
},
{
categoryId: invalid,
expectedLabel: UNKNOWN,
@ -125,6 +135,10 @@ describe('helpers', () => {
groupByField: 'ecs-compliant',
expectedTabId: ECS_COMPLIANT_TAB_ID,
},
{
groupByField: 'same-family',
expectedTabId: SAME_FAMILY_TAB_ID,
},
{
groupByField: 'some-other-group',
expectedTabId: ALL_TAB_ID,
@ -165,8 +179,8 @@ describe('helpers', () => {
).toEqual([
'### auditbeat-custom-index-1\n',
'| Result | Index | Docs | Incompatible fields | ILM Phase | Size |\n|--------|-------|------|---------------------|-----------|------|\n| ❌ | auditbeat-custom-index-1 | 4 (0.0%) | 3 | `unmanaged` | 27.7KB |\n\n',
'### **Incompatible fields** `3` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
`#### 3 incompatible fields, 0 fields with mappings in the same family\n\nFields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version ${EcsVersion}.\n\nIncompatible fields with mappings in the same family have exactly the same search behavior but may have different space usage or performance characteristics.\n\nWhen an incompatible field is not in the same family:\n❌ Detection engine rules referencing these fields may not match them correctly\n❌ Pages may not display some events or fields due to unexpected field mappings or values\n❌ Mappings or field values that don't comply with ECS are not supported\n`,
'### **Incompatible fields** `3` **Same family** `0` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
`#### 3 incompatible fields\n\nFields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version ${EcsVersion}.\n\n❌ Detection engine rules referencing these fields may not match them correctly\n❌ Pages may not display some events or fields due to unexpected field mappings or values\n❌ Mappings or field values that don't comply with ECS are not supported\n`,
'\n#### Incompatible field mappings - auditbeat-custom-index-1\n\n\n| Field | ECS mapping type (expected) | Index mapping type (actual) | \n|-------|-----------------------------|-----------------------------|\n| host.name | `keyword` | `text` |\n| source.ip | `ip` | `text` |\n\n#### Incompatible field values - auditbeat-custom-index-1\n\n\n| Field | ECS values (expected) | Document values (actual) | \n|-------|-----------------------|--------------------------|\n| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `an_invalid_category` (2), `theory` (1) |\n\n',
]);
});
@ -211,6 +225,7 @@ describe('helpers', () => {
type: 'date',
},
],
sameFamily: [],
};
expect(
@ -229,8 +244,8 @@ describe('helpers', () => {
).toEqual([
'### auditbeat-custom-empty-index-1\n',
'| Result | Index | Docs | Incompatible fields | ILM Phase | Size |\n|--------|-------|------|---------------------|-----------|------|\n| ❌ | auditbeat-custom-empty-index-1 | 0 (0.0%) | 1 | `unmanaged` | 247B |\n\n',
'### **Incompatible fields** `1` **Custom fields** `0` **ECS compliant fields** `0` **All fields** `0`\n',
`#### 1 incompatible field, 0 fields with mappings in the same family\n\nFields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version ${EcsVersion}.\n\nIncompatible fields with mappings in the same family have exactly the same search behavior but may have different space usage or performance characteristics.\n\nWhen an incompatible field is not in the same family:\n❌ Detection engine rules referencing these fields may not match them correctly\n❌ Pages may not display some events or fields due to unexpected field mappings or values\n❌ Mappings or field values that don't comply with ECS are not supported\n`,
'### **Incompatible fields** `1` **Same family** `0` **Custom fields** `0` **ECS compliant fields** `0` **All fields** `0`\n',
`#### 1 incompatible field\n\nFields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version ${EcsVersion}.\n\n❌ Detection engine rules referencing these fields may not match them correctly\n❌ Pages may not display some events or fields due to unexpected field mappings or values\n❌ Mappings or field values that don't comply with ECS are not supported\n`,
'\n#### Incompatible field mappings - auditbeat-custom-empty-index-1\n\n\n| Field | ECS mapping type (expected) | Index mapping type (actual) | \n|-------|-----------------------------|-----------------------------|\n| @timestamp | `date` | `-` |\n\n\n',
'#### Missing an @timestamp (date) field mapping for this index\n\nConsider adding an @timestamp (date) field mapping to this index, as required by the Elastic Common Schema (ECS), because:\n\n❌ Detection engine rules referencing these fields may not match them correctly\n❌ Pages may not display some events or fields due to unexpected field mappings or values\n',
]);

9
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/tabs/summary_tab/helpers.ts
View file

@ -13,6 +13,7 @@ import {
ECS_COMPLIANT_TAB_ID,
CUSTOM_TAB_ID,
INCOMPATIBLE_TAB_ID,
SAME_FAMILY_TAB_ID,
} from '../../index_properties/helpers';
import {
getAllIncompatibleMarkdownComments,
@ -21,7 +22,7 @@ import {
import * as i18n from '../../index_properties/translations';
import type { IlmPhase, PartitionedFieldMetadata } from '../../../types';
export type CategoryId = 'incompatible' | 'custom' | 'ecs-compliant';
export type CategoryId = 'incompatible' | 'custom' | 'ecs-compliant' | 'same-family';
interface SummaryData {
categoryId: CategoryId;
@ -40,6 +41,8 @@ export const getFillColor = (categoryId: CategoryId | string): string => {
switch (categoryId) {
case 'incompatible':
return euiThemeVars.euiColorDanger;
case 'same-family':
return euiThemeVars.euiColorLightShade;
case 'custom':
return euiThemeVars.euiColorLightShade;
case 'ecs-compliant':
@ -53,6 +56,8 @@ export const getNodeLabel = (categoryId: CategoryId): string => {
switch (categoryId) {
case 'incompatible':
return i18n.INCOMPATIBLE_FIELDS;
case 'same-family':
return i18n.SAME_FAMILY;
case 'custom':
return i18n.CUSTOM_FIELDS;
case 'ecs-compliant':
@ -66,6 +71,8 @@ export const getTabId = (groupByField: string): string => {
switch (groupByField) {
case 'incompatible':
return INCOMPATIBLE_TAB_ID;
case 'same-family':
return SAME_FAMILY_TAB_ID;
case 'custom':
return CUSTOM_TAB_ID;
case 'ecs-compliant':

20
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/ecs_summary_donut_chart/chart_legend/index.tsx
View file

@ -13,9 +13,13 @@ import {
ECS_COMPLIANT_TAB_ID,
CUSTOM_TAB_ID,
INCOMPATIBLE_TAB_ID,
SAME_FAMILY_TAB_ID,
} from '../../data_quality_panel/index_properties/helpers';
import { getCustomColor } from '../../data_quality_panel/tabs/custom_tab/helpers';
import { getIncompatibleColor } from '../../data_quality_panel/tabs/incompatible_tab/helpers';
import {
getIncompatibleColor,
getSameFamilyColor,
} from '../../data_quality_panel/tabs/incompatible_tab/helpers';
import type { PartitionedFieldMetadata } from '../../types';
import * as i18n from '../../data_quality_panel/index_properties/translations';
import { LegendContainer } from '../../data_quality_panel/tabs/styles';
@ -33,6 +37,11 @@ const ChartLegendComponent: React.FC<Props> = ({ partitionedFieldMetadata, setSe
[setSelectedTabId]
);
const showSameFamilyTab = useCallback(
() => setSelectedTabId(SAME_FAMILY_TAB_ID),
[setSelectedTabId]
);
const showCustomTab = useCallback(() => setSelectedTabId(CUSTOM_TAB_ID), [setSelectedTabId]);
const showEcsCompliantTab = useCallback(
@ -51,6 +60,15 @@ const ChartLegendComponent: React.FC<Props> = ({ partitionedFieldMetadata, setSe
/>
)}
{partitionedFieldMetadata.sameFamily.length > 0 && (
<ChartLegendItem
color={getSameFamilyColor()}
count={partitionedFieldMetadata.sameFamily.length}
onClick={showSameFamilyTab}
text={i18n.SAME_FAMILY}
/>
)}
{partitionedFieldMetadata.custom.length > 0 && (
<ChartLegendItem
color={getCustomColor(partitionedFieldMetadata)}

1
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/ecs_summary_donut_chart/helpers.test.ts
View file

@ -21,6 +21,7 @@ describe('helpers', () => {
custom: [],
ecsCompliant: [],
incompatible: [],
sameFamily: [],
};
expect(allMetadataIsEmpty(allIsEmpty)).toBe(true);

120
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/helpers.test.ts
View file

@ -6,6 +6,7 @@
*/
import { IlmExplainLifecycleLifecycleExplain } from '@elastic/elasticsearch/lib/api/types';
import { euiThemeVars } from '@kbn/ui-theme';
import { EcsFlat } from '@kbn/ecs';
import { omit } from 'lodash/fp';
@ -25,10 +26,12 @@ import {
getMissingTimestampFieldMetadata,
getPartitionedFieldMetadata,
getPartitionedFieldMetadataStats,
getSameFamilyStatColor,
getSizeInBytes,
getTotalDocsCount,
getTotalPatternIncompatible,
getTotalPatternIndicesChecked,
getTotalPatternSameFamily,
getTotalSizeInBytes,
hasValidTimestampMapping,
isMappingCompatible,
@ -78,6 +81,109 @@ import {
const ecsMetadata: Record<string, EcsMetadata> = EcsFlat as unknown as Record<string, EcsMetadata>;
describe('helpers', () => {
describe('getTotalPatternSameFamily', () => {
const baseResult: DataQualityCheckResult = {
docsCount: 4,
error: null,
ilmPhase: 'unmanaged',
incompatible: 3,
indexName: 'auditbeat-custom-index-1',
markdownComments: [
'### auditbeat-custom-index-1\n',
'| Result | Index | Docs | Incompatible fields | ILM Phase |\n|--------|-------|------|---------------------|-----------|\n| ❌ | auditbeat-custom-index-1 | 4 (0.0%) | 3 | `unmanaged` |\n\n',
'### **Incompatible fields** `3` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
"#### 3 incompatible fields, 0 fields with mappings in the same family\n\nFields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.1.\n\nIncompatible fields with mappings in the same family have exactly the same search behavior but may have different space usage or performance characteristics.\n\nWhen an incompatible field is not in the same family:\n❌ Detection engine rules referencing these fields may not match them correctly\n❌ Pages may not display some events or fields due to unexpected field mappings or values\n❌ Mappings or field values that don't comply with ECS are not supported\n",
'\n#### Incompatible field mappings - auditbeat-custom-index-1\n\n\n| Field | ECS mapping type (expected) | Index mapping type (actual) | \n|-------|-----------------------------|-----------------------------|\n| host.name | `keyword` | `text` |\n| source.ip | `ip` | `text` |\n\n#### Incompatible field values - auditbeat-custom-index-1\n\n\n| Field | ECS values (expected) | Document values (actual) | \n|-------|-----------------------|--------------------------|\n| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `an_invalid_category` (2),\n`theory` (1) |\n\n',
],
pattern: 'auditbeat-*',
sameFamily: 0,
};
it('returns undefined when results is undefined', () => {
expect(getTotalPatternSameFamily(undefined)).toBeUndefined();
});
it('returns 0 when results is an empty object', () => {
expect(getTotalPatternSameFamily({})).toBe(0);
});
it('should sum sameFamily values and return the total', () => {
const results: Record<string, DataQualityCheckResult> = {
a: {
...baseResult,
indexName: 'a',
markdownComments: [],
pattern: 'pattern',
sameFamily: 2,
},
b: {
...baseResult,
indexName: 'b',
markdownComments: [],
pattern: 'pattern',
sameFamily: 3,
},
c: { ...baseResult, indexName: 'c', markdownComments: [], pattern: 'pattern' },
};
expect(getTotalPatternSameFamily(results)).toBe(5);
});
it('handles a mix of defined and undefined sameFamily values', () => {
const results: Record<string, DataQualityCheckResult> = {
a: {
...baseResult,
indexName: 'a',
markdownComments: [],
pattern: 'pattern',
sameFamily: 1,
},
b: {
...baseResult,
indexName: 'b',
markdownComments: [],
pattern: 'pattern',
sameFamily: undefined,
},
c: {
...baseResult,
indexName: 'c',
markdownComments: [],
pattern: 'pattern',
sameFamily: 2,
},
};
expect(getTotalPatternSameFamily(results)).toBe(3);
});
});
describe('getSameFamilyStatColor', () => {
it('returns the expected color when sameFamily is greater than zero', () => {
const result = getSameFamilyStatColor(1);
expect(result).toEqual(euiThemeVars.euiColorLightShade);
});
it('returns undefined when sameFamily is 0', () => {
const result = getSameFamilyStatColor(0);
expect(result).toBeUndefined();
});
it('returns undefined when sameFamily is negative', () => {
const result = getSameFamilyStatColor(-1);
expect(result).toBeUndefined();
});
it('returns undefined when sameFamily is undefined', () => {
const result = getSameFamilyStatColor(undefined);
expect(result).toBeUndefined();
});
});
describe('getIndexNames', () => {
const isILMAvailable = true;
const ilmPhases = ['hot', 'warm', 'unmanaged'];
@ -477,7 +583,7 @@ describe('helpers', () => {
indexInvalidValues: [], // empty array, because the index does not contain any invalid values
hasEcsMetadata: true,
isEcsCompliant: true, // because the index has the expected mapping type, and no unallowed values
isInSameFamily: true, // `keyword` and `keyword` are in the same family
isInSameFamily: false,
};
test('it returns the happy path result when the index has no mapping conflicts, and no unallowed values', () => {
@ -670,6 +776,7 @@ describe('helpers', () => {
hostNameWithTextMapping,
sourceIpWithTextMapping,
],
sameFamily: [],
};
expect(getPartitionedFieldMetadata(enrichedFieldMetadata)).toEqual(expected);
@ -697,6 +804,7 @@ describe('helpers', () => {
hostNameWithTextMapping,
sourceIpWithTextMapping,
],
sameFamily: [],
};
expect(getPartitionedFieldMetadataStats(partitionedFieldMetadata)).toEqual({
@ -704,6 +812,7 @@ describe('helpers', () => {
custom: 4,
ecsCompliant: 2,
incompatible: 3,
sameFamily: 0,
});
});
});
@ -1078,6 +1187,7 @@ describe('helpers', () => {
indexName: '.ds-packetbeat-8.5.3-2023.02.04-000001',
markdownComments: ['foo', 'bar', 'baz'],
pattern: 'packetbeat-*',
sameFamily: 0,
},
'.ds-packetbeat-8.6.1-2023.02.04-000001': {
docsCount: 1628343,
@ -1087,6 +1197,7 @@ describe('helpers', () => {
indexName: '.ds-packetbeat-8.6.1-2023.02.04-000001',
markdownComments: ['foo', 'bar', 'baz'],
pattern: 'packetbeat-*',
sameFamily: 0,
},
};
@ -1103,6 +1214,7 @@ describe('helpers', () => {
indexName: '.ds-auditbeat-8.6.1-2023.02.07-000001',
markdownComments: ['foo', 'bar', 'baz'],
pattern: 'auditbeat-*',
sameFamily: 0,
},
'auditbeat-custom-index-1': {
docsCount: 4,
@ -1112,6 +1224,7 @@ describe('helpers', () => {
indexName: 'auditbeat-custom-index-1',
markdownComments: ['foo', 'bar', 'baz'],
pattern: 'auditbeat-*',
sameFamily: 0,
},
'auditbeat-custom-empty-index-1': {
docsCount: 0,
@ -1121,6 +1234,7 @@ describe('helpers', () => {
indexName: 'auditbeat-custom-empty-index-1',
markdownComments: ['foo', 'bar', 'baz'],
pattern: 'auditbeat-*',
sameFamily: 0,
},
};
@ -1137,6 +1251,7 @@ describe('helpers', () => {
indexName: '.ds-auditbeat-8.6.1-2023.02.07-000001',
markdownComments: ['foo', 'bar', 'baz'],
pattern: 'auditbeat-*',
sameFamily: 0,
},
'auditbeat-custom-index-1': {
docsCount: 4,
@ -1146,6 +1261,7 @@ describe('helpers', () => {
indexName: 'auditbeat-custom-index-1',
markdownComments: ['foo', 'bar', 'baz'],
pattern: 'auditbeat-*',
sameFamily: 0,
},
'auditbeat-custom-empty-index-1': {
docsCount: 0,
@ -1155,6 +1271,7 @@ describe('helpers', () => {
indexName: 'auditbeat-custom-empty-index-1',
markdownComments: ['foo', 'bar', 'baz'],
pattern: 'auditbeat-*',
sameFamily: 0,
},
};
@ -1219,6 +1336,7 @@ describe('helpers', () => {
indexName: '.ds-packetbeat-8.5.3-2023.02.04-000001',
markdownComments: ['foo', 'bar', 'baz'],
pattern: 'packetbeat-*',
sameFamily: 0,
};
expect(getErrorSummary(resultWithError)).toEqual({

30
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/helpers.ts
View file

@ -180,7 +180,11 @@ export const getEnrichedFieldMetadata = ({
const ecsExpectedType = ecsMetadata[field].type;
const isEcsCompliant =
isMappingCompatible({ ecsExpectedType, type }) && indexInvalidValues.length === 0;
const isInSameFamily = getIsInSameFamily({ ecsExpectedType, type });
const isInSameFamily =
!isMappingCompatible({ ecsExpectedType, type }) &&
indexInvalidValues.length === 0 &&
getIsInSameFamily({ ecsExpectedType, type });
return {
...ecsMetadata[field],
@ -223,26 +227,31 @@ export const getPartitionedFieldMetadata = (
ecsCompliant: x.isEcsCompliant ? [...acc.ecsCompliant, x] : acc.ecsCompliant,
custom: !x.hasEcsMetadata ? [...acc.custom, x] : acc.custom,
incompatible:
x.hasEcsMetadata && !x.isEcsCompliant ? [...acc.incompatible, x] : acc.incompatible,
x.hasEcsMetadata && !x.isEcsCompliant && !x.isInSameFamily
? [...acc.incompatible, x]
: acc.incompatible,
sameFamily: x.isInSameFamily ? [...acc.sameFamily, x] : acc.sameFamily,
}),
{
all: [],
ecsCompliant: [],
custom: [],
incompatible: [],
sameFamily: [],
}
);
export const getPartitionedFieldMetadataStats = (
partitionedFieldMetadata: PartitionedFieldMetadata
): PartitionedFieldMetadataStats => {
const { all, ecsCompliant, custom, incompatible } = partitionedFieldMetadata;
const { all, ecsCompliant, custom, incompatible, sameFamily } = partitionedFieldMetadata;
return {
all: all.length,
ecsCompliant: ecsCompliant.length,
custom: custom.length,
incompatible: incompatible.length,
sameFamily: sameFamily.length,
};
};
@ -369,9 +378,24 @@ export const getTotalPatternIndicesChecked = (patternRollup: PatternRollup | und
}
};
export const getTotalPatternSameFamily = (
results: Record<string, DataQualityCheckResult> | undefined
): number | undefined => {
if (results == null) {
return undefined;
}
const allResults = Object.values(results);
return allResults.reduce<number>((acc, { sameFamily }) => acc + (sameFamily ?? 0), 0);
};
export const getIncompatibleStatColor = (incompatible: number | undefined): string | undefined =>
incompatible != null && incompatible > 0 ? getFillColor('incompatible') : undefined;
export const getSameFamilyStatColor = (sameFamily: number | undefined): string | undefined =>
sameFamily != null && sameFamily > 0 ? getFillColor('same-family') : undefined;
export const getErrorSummary = ({
error,
indexName,

2
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/mock/data_quality_check_result/mock_index.tsx
View file

@ -22,6 +22,7 @@ export const mockDataQualityCheckResult: Record<string, DataQualityCheckResult>
'\n#### Incompatible field mappings - auditbeat-custom-index-1\n\n\n| Field | ECS mapping type (expected) | Index mapping type (actual) | \n|-------|-----------------------------|-----------------------------|\n| host.name | `keyword` | `text` |\n| source.ip | `ip` | `text` |\n\n#### Incompatible field values - auditbeat-custom-index-1\n\n\n| Field | ECS values (expected) | Document values (actual) | \n|-------|-----------------------|--------------------------|\n| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `an_invalid_category` (2),\n`theory` (1) |\n\n',
],
pattern: 'auditbeat-*',
sameFamily: 0,
},
'auditbeat-7.9.3-2023.02.13-000001': {
docsCount: 2438,
@ -37,5 +38,6 @@ export const mockDataQualityCheckResult: Record<string, DataQualityCheckResult>
'\n#### Incompatible field mappings - auditbeat-7.9.3-2023.02.13-000001\n\n\n| Field | ECS mapping type (expected) | Index mapping type (actual) | \n|-------|-----------------------------|-----------------------------|\n| error.message | `match_only_text` | `text` `same family` |\n| error.stack_trace | `wildcard` | `keyword` `same family` |\n| http.request.body.content | `wildcard` | `keyword` `same family` |\n| http.response.body.content | `wildcard` | `keyword` `same family` |\n| message | `match_only_text` | `text` `same family` |\n| process.command_line | `wildcard` | `keyword` `same family` |\n| process.parent.command_line | `wildcard` | `keyword` `same family` |\n| registry.data.strings | `wildcard` | `keyword` `same family` |\n| url.full | `wildcard` | `keyword` `same family` |\n| url.original | `wildcard` | `keyword` `same family` |\n| url.path | `wildcard` | `keyword` `same family` |\n\n#### Incompatible field values - auditbeat-7.9.3-2023.02.13-000001\n\n\n| Field | ECS values (expected) | Document values (actual) | \n|-------|-----------------------|--------------------------|\n| event.kind | `alert`, `enrichment`, `event`, `metric`, `state`, `pipeline_error`, `signal` | `error` (7) |\n\n',
],
pattern: 'auditbeat-*',
sameFamily: 0,
},
};

2
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/mock/enriched_field_metadata/mock_enriched_field_metadata.ts
View file

@ -163,7 +163,7 @@ export const eventCategory: EnrichedFieldMetadata = {
indexInvalidValues: [],
hasEcsMetadata: true,
isEcsCompliant: true,
isInSameFamily: true, // `keyword` and `keyword` are in the same family
isInSameFamily: false,
};
export const eventCategoryWithUnallowedValues: EnrichedFieldMetadata = {

15
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/mock/partitioned_field_metadata/mock_partitioned_field_metadata.tsx
View file

@ -7,6 +7,10 @@
import { PartitionedFieldMetadata } from '../../types';
/**
* Note: this mock does NOT have any `sameFamily` entries.
* See `mockPartitionedFieldMetadataWithSameFamily` for an example of a mock with `sameFamily` entries.
*/
export const mockPartitionedFieldMetadata: PartitionedFieldMetadata = {
all: [
{
@ -175,7 +179,7 @@ export const mockPartitionedFieldMetadata: PartitionedFieldMetadata = {
],
hasEcsMetadata: true,
isEcsCompliant: false, // this index has unallowed values
isInSameFamily: true, // keyword and keyword are in the same family
isInSameFamily: false, // keyword and keyword
},
{
dashed_name: 'host-name',
@ -258,7 +262,7 @@ export const mockPartitionedFieldMetadata: PartitionedFieldMetadata = {
indexInvalidValues: [],
hasEcsMetadata: true,
isEcsCompliant: true, // indexFieldType === type, and no indexInvalidValues
isInSameFamily: false, // `long` is not a member of any families
isInSameFamily: false,
},
],
ecsCompliant: [
@ -279,7 +283,7 @@ export const mockPartitionedFieldMetadata: PartitionedFieldMetadata = {
indexInvalidValues: [],
hasEcsMetadata: true,
isEcsCompliant: true, // indexFieldType === type, and no indexInvalidValues
isInSameFamily: false, // `date` is not a member of any families
isInSameFamily: false,
},
{
dashed_name: 'source-port',
@ -296,7 +300,7 @@ export const mockPartitionedFieldMetadata: PartitionedFieldMetadata = {
indexInvalidValues: [],
hasEcsMetadata: true,
isEcsCompliant: true, // indexFieldType === type, and no indexInvalidValues
isInSameFamily: false, // `long` is not a member of any families
isInSameFamily: false,
},
],
custom: [
@ -481,7 +485,7 @@ export const mockPartitionedFieldMetadata: PartitionedFieldMetadata = {
],
hasEcsMetadata: true,
isEcsCompliant: false, // indexFieldType === type, but there are indexInvalidValues
isInSameFamily: true, // `keyword` and `keyword` are in the same family
isInSameFamily: false,
},
{
dashed_name: 'host-name',
@ -518,4 +522,5 @@ export const mockPartitionedFieldMetadata: PartitionedFieldMetadata = {
isInSameFamily: false, // `ip` is not a member of any families
},
],
sameFamily: [],
};

566
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/mock/partitioned_field_metadata/mock_partitioned_field_metadata_with_same_family.ts Normal file
View file

@ -0,0 +1,566 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { PartitionedFieldMetadata } from '../../types';
/** This mock `PartitionedFieldMetadata` has a sameFamily[] field */
export const mockPartitionedFieldMetadataWithSameFamily: PartitionedFieldMetadata = {
all: [
{
dashed_name: 'timestamp',
description:
'Date/time when the event originated.\nThis is the date/time extracted from the event, typically representing when the event was generated by the source.\nIf the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline.\nRequired field for all events.',
example: '2016-05-23T08:05:34.853Z',
flat_name: '@timestamp',
level: 'core',
name: '@timestamp',
normalize: [],
required: true,
short: 'Date/time when the event originated.',
type: 'date',
indexFieldName: '@timestamp',
indexFieldType: 'date',
indexInvalidValues: [],
hasEcsMetadata: true,
isEcsCompliant: true,
isInSameFamily: false,
},
{
dashed_name: 'agent-type',
description:
'Type of the agent.\nThe agent type always stays the same and should be given by the agent used. In case of Filebeat the agent would always be Filebeat also if two Filebeat instances are run on the same machine.',
example: 'filebeat',
flat_name: 'agent.type',
ignore_above: 1024,
level: 'core',
name: 'type',
normalize: [],
short: 'Type of the agent.',
type: 'keyword',
indexFieldName: 'agent.type',
indexFieldType: 'constant_keyword',
indexInvalidValues: [],
hasEcsMetadata: true,
isEcsCompliant: false,
isInSameFamily: true,
},
{
allowed_values: [
{
description:
'Events in this category are related to the challenge and response process in which credentials are supplied and verified to allow the creation of a session. Common sources for these logs are Windows event logs and ssh logs. Visualize and analyze events in this category to look for failed logins, and other authentication-related activity.',
expected_event_types: ['start', 'end', 'info'],
name: 'authentication',
},
{
description:
'Events in the configuration category have to deal with creating, modifying, or deleting the settings or parameters of an application, process, or system.\nExample sources include security policy change logs, configuration auditing logging, and system integrity monitoring.',
expected_event_types: ['access', 'change', 'creation', 'deletion', 'info'],
name: 'configuration',
},
{
description:
'The database category denotes events and metrics relating to a data storage and retrieval system. Note that use of this category is not limited to relational database systems. Examples include event logs from MS SQL, MySQL, Elasticsearch, MongoDB, etc. Use this category to visualize and analyze database activity such as accesses and changes.',
expected_event_types: ['access', 'change', 'info', 'error'],
name: 'database',
},
{
description:
'Events in the driver category have to do with operating system device drivers and similar software entities such as Windows drivers, kernel extensions, kernel modules, etc.\nUse events and metrics in this category to visualize and analyze driver-related activity and status on hosts.',
expected_event_types: ['change', 'end', 'info', 'start'],
name: 'driver',
},
{
description:
'This category is used for events relating to email messages, email attachments, and email network or protocol activity.\nEmails events can be produced by email security gateways, mail transfer agents, email cloud service providers, or mail server monitoring applications.',
expected_event_types: ['info'],
name: 'email',
},
{
description:
'Relating to a set of information that has been created on, or has existed on a filesystem. Use this category of events to visualize and analyze the creation, access, and deletions of files. Events in this category can come from both host-based and network-based sources. An example source of a network-based detection of a file transfer would be the Zeek file.log.',
expected_event_types: ['change', 'creation', 'deletion', 'info'],
name: 'file',
},
{
description:
'Use this category to visualize and analyze information such as host inventory or host lifecycle events.\nMost of the events in this category can usually be observed from the outside, such as from a hypervisor or a control plane\'s point of view. Some can also be seen from within, such as "start" or "end".\nNote that this category is for information about hosts themselves; it is not meant to capture activity "happening on a host".',
expected_event_types: ['access', 'change', 'end', 'info', 'start'],
name: 'host',
},
{
description:
'Identity and access management (IAM) events relating to users, groups, and administration. Use this category to visualize and analyze IAM-related logs and data from active directory, LDAP, Okta, Duo, and other IAM systems.',
expected_event_types: [
'admin',
'change',
'creation',
'deletion',
'group',
'info',
'user',
],
name: 'iam',
},
{
description:
'Relating to intrusion detections from IDS/IPS systems and functions, both network and host-based. Use this category to visualize and analyze intrusion detection alerts from systems such as Snort, Suricata, and Palo Alto threat detections.',
expected_event_types: ['allowed', 'denied', 'info'],
name: 'intrusion_detection',
},
{
description:
'Malware detection events and alerts. Use this category to visualize and analyze malware detections from EDR/EPP systems such as Elastic Endpoint Security, Symantec Endpoint Protection, Crowdstrike, and network IDS/IPS systems such as Suricata, or other sources of malware-related events such as Palo Alto Networks threat logs and Wildfire logs.',
expected_event_types: ['info'],
name: 'malware',
},
{
description:
'Relating to all network activity, including network connection lifecycle, network traffic, and essentially any event that includes an IP address. Many events containing decoded network protocol transactions fit into this category. Use events in this category to visualize or analyze counts of network ports, protocols, addresses, geolocation information, etc.',
expected_event_types: [
'access',
'allowed',
'connection',
'denied',
'end',
'info',
'protocol',
'start',
],
name: 'network',
},
{
description:
'Relating to software packages installed on hosts. Use this category to visualize and analyze inventory of software installed on various hosts, or to determine host vulnerability in the absence of vulnerability scan data.',
expected_event_types: ['access', 'change', 'deletion', 'info', 'installation', 'start'],
name: 'package',
},
{
description:
'Use this category of events to visualize and analyze process-specific information such as lifecycle events or process ancestry.',
expected_event_types: ['access', 'change', 'end', 'info', 'start'],
name: 'process',
},
{
description:
'Having to do with settings and assets stored in the Windows registry. Use this category to visualize and analyze activity such as registry access and modifications.',
expected_event_types: ['access', 'change', 'creation', 'deletion'],
name: 'registry',
},
{
description:
'The session category is applied to events and metrics regarding logical persistent connections to hosts and services. Use this category to visualize and analyze interactive or automated persistent connections between assets. Data for this category may come from Windows Event logs, SSH logs, or stateless sessions such as HTTP cookie-based sessions, etc.',
expected_event_types: ['start', 'end', 'info'],
name: 'session',
},
{
description:
"Use this category to visualize and analyze events describing threat actors' targets, motives, or behaviors.",
expected_event_types: ['indicator'],
name: 'threat',
},
{
description:
'Relating to vulnerability scan results. Use this category to analyze vulnerabilities detected by Tenable, Qualys, internal scanners, and other vulnerability management sources.',
expected_event_types: ['info'],
name: 'vulnerability',
},
{
description:
'Relating to web server access. Use this category to create a dashboard of web server/proxy activity from apache, IIS, nginx web servers, etc. Note: events from network observers such as Zeek http log may also be included in this category.',
expected_event_types: ['access', 'error', 'info'],
name: 'web',
},
],
dashed_name: 'event-category',
description:
'This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy.\n`event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory.\nThis field is an array. This will allow proper categorization of some events that fall in multiple categories.',
example: 'authentication',
flat_name: 'event.category',
ignore_above: 1024,
level: 'core',
name: 'category',
normalize: ['array'],
short: 'Event category. The second categorization field in the hierarchy.',
type: 'keyword',
indexFieldName: 'event.category',
indexFieldType: 'constant_keyword',
indexInvalidValues: [
{
count: 2,
fieldName: 'an_invalid_category',
},
],
hasEcsMetadata: true,
isEcsCompliant: false, // because it has invalid values
isInSameFamily: false,
},
{
dashed_name: 'host-name',
description:
'Name of the host.\nIt can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.',
flat_name: 'host.name',
ignore_above: 1024,
level: 'core',
name: 'name',
normalize: [],
short: 'Name of the host.',
type: 'keyword',
indexFieldName: 'host.name',
indexFieldType: 'text',
indexInvalidValues: [],
hasEcsMetadata: true,
isEcsCompliant: false, // text is not keyword
isInSameFamily: false,
},
{
indexFieldName: 'host.name.keyword',
indexFieldType: 'keyword',
indexInvalidValues: [],
hasEcsMetadata: false, // unknown field
isEcsCompliant: false,
isInSameFamily: false,
},
{
indexFieldName: 'some.field',
indexFieldType: 'text',
indexInvalidValues: [],
hasEcsMetadata: false,
isEcsCompliant: false, // unknown field
isInSameFamily: false,
},
{
indexFieldName: 'some.field.keyword',
indexFieldType: 'keyword',
indexInvalidValues: [],
hasEcsMetadata: false, // unknown field
isEcsCompliant: false,
isInSameFamily: false,
},
{
dashed_name: 'source-ip',
description: 'IP address of the source (IPv4 or IPv6).',
flat_name: 'source.ip',
level: 'core',
name: 'ip',
normalize: [],
short: 'IP address of the source.',
type: 'ip',
indexFieldName: 'source.ip',
indexFieldType: 'text',
indexInvalidValues: [],
hasEcsMetadata: true,
isEcsCompliant: false, // text is not ip
isInSameFamily: false,
},
{
indexFieldName: 'source.ip.keyword',
indexFieldType: 'keyword',
indexInvalidValues: [],
hasEcsMetadata: false, // unknown field
isEcsCompliant: false,
isInSameFamily: false,
},
{
dashed_name: 'source-port',
description: 'Port of the source.',
flat_name: 'source.port',
format: 'string',
level: 'core',
name: 'port',
normalize: [],
short: 'Port of the source.',
type: 'long',
indexFieldName: 'source.port',
indexFieldType: 'long',
indexInvalidValues: [],
hasEcsMetadata: true,
isEcsCompliant: true,
isInSameFamily: false,
},
],
ecsCompliant: [
{
dashed_name: 'timestamp',
description:
'Date/time when the event originated.\nThis is the date/time extracted from the event, typically representing when the event was generated by the source.\nIf the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline.\nRequired field for all events.',
example: '2016-05-23T08:05:34.853Z',
flat_name: '@timestamp',
level: 'core',
name: '@timestamp',
normalize: [],
required: true,
short: 'Date/time when the event originated.',
type: 'date',
indexFieldName: '@timestamp',
indexFieldType: 'date',
indexInvalidValues: [],
hasEcsMetadata: true,
isEcsCompliant: true,
isInSameFamily: false,
},
{
dashed_name: 'source-port',
description: 'Port of the source.',
flat_name: 'source.port',
format: 'string',
level: 'core',
name: 'port',
normalize: [],
short: 'Port of the source.',
type: 'long',
indexFieldName: 'source.port',
indexFieldType: 'long',
indexInvalidValues: [],
hasEcsMetadata: true,
isEcsCompliant: true,
isInSameFamily: false,
},
],
custom: [
{
indexFieldName: 'host.name.keyword',
indexFieldType: 'keyword',
indexInvalidValues: [],
hasEcsMetadata: false,
isEcsCompliant: false,
isInSameFamily: false,
},
{
indexFieldName: 'some.field',
indexFieldType: 'text',
indexInvalidValues: [],
hasEcsMetadata: false,
isEcsCompliant: false,
isInSameFamily: false,
},
{
indexFieldName: 'some.field.keyword',
indexFieldType: 'keyword',
indexInvalidValues: [],
hasEcsMetadata: false,
isEcsCompliant: false,
isInSameFamily: false,
},
{
indexFieldName: 'source.ip.keyword',
indexFieldType: 'keyword',
indexInvalidValues: [],
hasEcsMetadata: false,
isEcsCompliant: false,
isInSameFamily: false,
},
],
incompatible: [
{
allowed_values: [
{
description:
'Events in this category are related to the challenge and response process in which credentials are supplied and verified to allow the creation of a session. Common sources for these logs are Windows event logs and ssh logs. Visualize and analyze events in this category to look for failed logins, and other authentication-related activity.',
expected_event_types: ['start', 'end', 'info'],
name: 'authentication',
},
{
description:
'Events in the configuration category have to deal with creating, modifying, or deleting the settings or parameters of an application, process, or system.\nExample sources include security policy change logs, configuration auditing logging, and system integrity monitoring.',
expected_event_types: ['access', 'change', 'creation', 'deletion', 'info'],
name: 'configuration',
},
{
description:
'The database category denotes events and metrics relating to a data storage and retrieval system. Note that use of this category is not limited to relational database systems. Examples include event logs from MS SQL, MySQL, Elasticsearch, MongoDB, etc. Use this category to visualize and analyze database activity such as accesses and changes.',
expected_event_types: ['access', 'change', 'info', 'error'],
name: 'database',
},
{
description:
'Events in the driver category have to do with operating system device drivers and similar software entities such as Windows drivers, kernel extensions, kernel modules, etc.\nUse events and metrics in this category to visualize and analyze driver-related activity and status on hosts.',
expected_event_types: ['change', 'end', 'info', 'start'],
name: 'driver',
},
{
description:
'This category is used for events relating to email messages, email attachments, and email network or protocol activity.\nEmails events can be produced by email security gateways, mail transfer agents, email cloud service providers, or mail server monitoring applications.',
expected_event_types: ['info'],
name: 'email',
},
{
description:
'Relating to a set of information that has been created on, or has existed on a filesystem. Use this category of events to visualize and analyze the creation, access, and deletions of files. Events in this category can come from both host-based and network-based sources. An example source of a network-based detection of a file transfer would be the Zeek file.log.',
expected_event_types: ['change', 'creation', 'deletion', 'info'],
name: 'file',
},
{
description:
'Use this category to visualize and analyze information such as host inventory or host lifecycle events.\nMost of the events in this category can usually be observed from the outside, such as from a hypervisor or a control plane\'s point of view. Some can also be seen from within, such as "start" or "end".\nNote that this category is for information about hosts themselves; it is not meant to capture activity "happening on a host".',
expected_event_types: ['access', 'change', 'end', 'info', 'start'],
name: 'host',
},
{
description:
'Identity and access management (IAM) events relating to users, groups, and administration. Use this category to visualize and analyze IAM-related logs and data from active directory, LDAP, Okta, Duo, and other IAM systems.',
expected_event_types: [
'admin',
'change',
'creation',
'deletion',
'group',
'info',
'user',
],
name: 'iam',
},
{
description:
'Relating to intrusion detections from IDS/IPS systems and functions, both network and host-based. Use this category to visualize and analyze intrusion detection alerts from systems such as Snort, Suricata, and Palo Alto threat detections.',
expected_event_types: ['allowed', 'denied', 'info'],
name: 'intrusion_detection',
},
{
description:
'Malware detection events and alerts. Use this category to visualize and analyze malware detections from EDR/EPP systems such as Elastic Endpoint Security, Symantec Endpoint Protection, Crowdstrike, and network IDS/IPS systems such as Suricata, or other sources of malware-related events such as Palo Alto Networks threat logs and Wildfire logs.',
expected_event_types: ['info'],
name: 'malware',
},
{
description:
'Relating to all network activity, including network connection lifecycle, network traffic, and essentially any event that includes an IP address. Many events containing decoded network protocol transactions fit into this category. Use events in this category to visualize or analyze counts of network ports, protocols, addresses, geolocation information, etc.',
expected_event_types: [
'access',
'allowed',
'connection',
'denied',
'end',
'info',
'protocol',
'start',
],
name: 'network',
},
{
description:
'Relating to software packages installed on hosts. Use this category to visualize and analyze inventory of software installed on various hosts, or to determine host vulnerability in the absence of vulnerability scan data.',
expected_event_types: ['access', 'change', 'deletion', 'info', 'installation', 'start'],
name: 'package',
},
{
description:
'Use this category of events to visualize and analyze process-specific information such as lifecycle events or process ancestry.',
expected_event_types: ['access', 'change', 'end', 'info', 'start'],
name: 'process',
},
{
description:
'Having to do with settings and assets stored in the Windows registry. Use this category to visualize and analyze activity such as registry access and modifications.',
expected_event_types: ['access', 'change', 'creation', 'deletion'],
name: 'registry',
},
{
description:
'The session category is applied to events and metrics regarding logical persistent connections to hosts and services. Use this category to visualize and analyze interactive or automated persistent connections between assets. Data for this category may come from Windows Event logs, SSH logs, or stateless sessions such as HTTP cookie-based sessions, etc.',
expected_event_types: ['start', 'end', 'info'],
name: 'session',
},
{
description:
"Use this category to visualize and analyze events describing threat actors' targets, motives, or behaviors.",
expected_event_types: ['indicator'],
name: 'threat',
},
{
description:
'Relating to vulnerability scan results. Use this category to analyze vulnerabilities detected by Tenable, Qualys, internal scanners, and other vulnerability management sources.',
expected_event_types: ['info'],
name: 'vulnerability',
},
{
description:
'Relating to web server access. Use this category to create a dashboard of web server/proxy activity from apache, IIS, nginx web servers, etc. Note: events from network observers such as Zeek http log may also be included in this category.',
expected_event_types: ['access', 'error', 'info'],
name: 'web',
},
],
dashed_name: 'event-category',
description:
'This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy.\n`event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory.\nThis field is an array. This will allow proper categorization of some events that fall in multiple categories.',
example: 'authentication',
flat_name: 'event.category',
ignore_above: 1024,
level: 'core',
name: 'category',
normalize: ['array'],
short: 'Event category. The second categorization field in the hierarchy.',
type: 'keyword',
indexFieldName: 'event.category',
indexFieldType: 'constant_keyword',
indexInvalidValues: [
{
count: 2,
fieldName: 'an_invalid_category',
},
],
hasEcsMetadata: true,
isEcsCompliant: false, // has invalid values
isInSameFamily: false,
},
{
dashed_name: 'host-name',
description:
'Name of the host.\nIt can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.',
flat_name: 'host.name',
ignore_above: 1024,
level: 'core',
name: 'name',
normalize: [],
short: 'Name of the host.',
type: 'keyword',
indexFieldName: 'host.name',
indexFieldType: 'text',
indexInvalidValues: [],
hasEcsMetadata: true,
isEcsCompliant: false, // text is not keyword
isInSameFamily: false,
},
{
dashed_name: 'source-ip',
description: 'IP address of the source (IPv4 or IPv6).',
flat_name: 'source.ip',
level: 'core',
name: 'ip',
normalize: [],
short: 'IP address of the source.',
type: 'ip',
indexFieldName: 'source.ip',
indexFieldType: 'text',
indexInvalidValues: [],
hasEcsMetadata: true,
isEcsCompliant: false, // text is not ip
isInSameFamily: false,
},
],
sameFamily: [
{
dashed_name: 'agent-type',
description:
'Type of the agent.\nThe agent type always stays the same and should be given by the agent used. In case of Filebeat the agent would always be Filebeat also if two Filebeat instances are run on the same machine.',
example: 'filebeat',
flat_name: 'agent.type',
ignore_above: 1024,
level: 'core',
name: 'type',
normalize: [],
short: 'Type of the agent.',
type: 'keyword',
indexFieldName: 'agent.type',
indexFieldType: 'constant_keyword',
indexInvalidValues: [],
hasEcsMetadata: true,
isEcsCompliant: false, // types are not an exact match
isInSameFamily: true, // types are in the same family
},
],
};

1
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/mock/pattern_rollup/mock_alerts_pattern_rollup.ts
View file

@ -82,6 +82,7 @@ export const alertIndexWithAllResults: PatternRollup = {
indexName: '.internal.alerts-security.alerts-default-000001',
markdownComments: ['foo', 'bar', 'baz'],
pattern: '.alerts-security.alerts-default',
sameFamily: 0,
},
},
sizeInBytes: 29717961631,

3
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/mock/pattern_rollup/mock_auditbeat_pattern_rollup.ts
View file

@ -151,6 +151,7 @@ export const auditbeatWithAllResults: PatternRollup = {
indexName: '.ds-auditbeat-8.6.1-2023.02.07-000001',
markdownComments: ['foo', 'bar', 'baz'],
pattern: 'auditbeat-*',
sameFamily: 0,
},
'auditbeat-custom-index-1': {
docsCount: 4,
@ -160,6 +161,7 @@ export const auditbeatWithAllResults: PatternRollup = {
indexName: 'auditbeat-custom-index-1',
markdownComments: ['foo', 'bar', 'baz'],
pattern: 'auditbeat-*',
sameFamily: 0,
},
'auditbeat-custom-empty-index-1': {
docsCount: 0,
@ -169,6 +171,7 @@ export const auditbeatWithAllResults: PatternRollup = {
indexName: 'auditbeat-custom-empty-index-1',
markdownComments: ['foo', 'bar', 'baz'],
pattern: 'auditbeat-*',
sameFamily: 0,
},
},
sizeInBytes: 18820446,

2
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/mock/pattern_rollup/mock_packetbeat_pattern_rollup.ts
View file

@ -129,6 +129,7 @@ export const packetbeatWithSomeErrors: PatternRollup = {
indexName: '.ds-packetbeat-8.5.3-2023.02.04-000001',
markdownComments: ['foo', 'bar', 'baz'],
pattern: 'packetbeat-*',
sameFamily: undefined,
},
'.ds-packetbeat-8.6.1-2023.02.04-000001': {
docsCount: 1628343,
@ -138,6 +139,7 @@ export const packetbeatWithSomeErrors: PatternRollup = {
indexName: '.ds-packetbeat-8.6.1-2023.02.04-000001',
markdownComments: ['foo', 'bar', 'baz'],
pattern: 'packetbeat-*',
sameFamily: 0,
},
},
sizeInBytes: 1096520898,

3
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/types.ts
View file

@ -53,6 +53,7 @@ export interface PartitionedFieldMetadata {
custom: EnrichedFieldMetadata[];
ecsCompliant: EnrichedFieldMetadata[];
incompatible: EnrichedFieldMetadata[];
sameFamily: EnrichedFieldMetadata[];
}
export interface PartitionedFieldMetadataStats {
@ -60,6 +61,7 @@ export interface PartitionedFieldMetadataStats {
custom: number;
ecsCompliant: number;
incompatible: number;
sameFamily: number;
}
export interface UnallowedValueRequestItem {
@ -103,6 +105,7 @@ export interface DataQualityCheckResult {
incompatible: number | undefined;
indexName: string;
markdownComments: string[];
sameFamily: number | undefined;
pattern: string;
}

57
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/use_results_rollup/helpers.test.ts
View file

@ -12,6 +12,7 @@ import {
getTotalIncompatible,
getTotalIndices,
getTotalIndicesChecked,
getTotalSameFamily,
onPatternRollupUpdated,
updateResultOnCheckCompleted,
} from './helpers';
@ -21,10 +22,11 @@ import {
packetbeatNoResults,
packetbeatWithSomeErrors,
} from '../mock/pattern_rollup/mock_packetbeat_pattern_rollup';
import { PatternRollup } from '../types';
import { DataQualityCheckResult, PatternRollup } from '../types';
import { EMPTY_STAT } from '../helpers';
import { IndicesStatsIndicesStats } from '@elastic/elasticsearch/lib/api/types';
import { mockPartitionedFieldMetadata } from '../mock/partitioned_field_metadata/mock_partitioned_field_metadata';
import { alertIndexWithAllResults } from '../mock/pattern_rollup/mock_alerts_pattern_rollup';
const defaultBytesFormat = '0,0.[0]b';
const formatBytes = (value: number | undefined) =>
@ -50,6 +52,44 @@ describe('helpers', () => {
global.fetch = originalFetch;
});
describe('getTotalSameFamily', () => {
const defaultDataQualityCheckResult: DataQualityCheckResult = {
docsCount: 26093,
error: null,
ilmPhase: 'hot',
incompatible: 0,
indexName: '.internal.alerts-security.alerts-default-000001',
markdownComments: ['foo', 'bar', 'baz'],
pattern: '.alerts-security.alerts-default',
sameFamily: 7,
};
const alertIndexWithSameFamily: PatternRollup = {
...alertIndexWithAllResults,
results: {
'.internal.alerts-security.alerts-default-000001': {
...defaultDataQualityCheckResult,
},
},
};
const withSameFamily: Record<string, PatternRollup> = {
'.internal.alerts-security.alerts-default-000001': alertIndexWithSameFamily,
};
test('it returns the expected count when patternRollups has sameFamily', () => {
expect(getTotalSameFamily(withSameFamily)).toEqual(7);
});
test('it returns undefined when patternRollups is empty', () => {
expect(getTotalSameFamily({})).toBeUndefined();
});
test('it returns zero when none of the rollups have same family', () => {
expect(getTotalSameFamily(patternRollups)).toEqual(0);
});
});
describe('getTotalIndices', () => {
test('it returns the expected total when ALL `PatternRollup`s have an `indices`', () => {
expect(getTotalIndices(patternRollups)).toEqual(5);
@ -230,11 +270,12 @@ describe('helpers', () => {
markdownComments: [
'### .ds-packetbeat-8.6.1-2023.02.04-000001\n',
'| Result | Index | Docs | Incompatible fields | ILM Phase | Size |\n|--------|-------|------|---------------------|-----------|------|\n| ❌ | .ds-packetbeat-8.6.1-2023.02.04-000001 | 1,628,343 (50.0%) | 3 | `hot` | 697.7MB |\n\n',
'### **Incompatible fields** `3` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
"#### 3 incompatible fields, 0 fields with mappings in the same family\n\nFields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.1.\n\nIncompatible fields with mappings in the same family have exactly the same search behavior but may have different space usage or performance characteristics.\n\nWhen an incompatible field is not in the same family:\n❌ Detection engine rules referencing these fields may not match them correctly\n❌ Pages may not display some events or fields due to unexpected field mappings or values\n❌ Mappings or field values that don't comply with ECS are not supported\n",
'### **Incompatible fields** `3` **Same family** `0` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
"#### 3 incompatible fields\n\nFields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.1.\n\n❌ Detection engine rules referencing these fields may not match them correctly\n❌ Pages may not display some events or fields due to unexpected field mappings or values\n❌ Mappings or field values that don't comply with ECS are not supported\n",
'\n#### Incompatible field mappings - .ds-packetbeat-8.6.1-2023.02.04-000001\n\n\n| Field | ECS mapping type (expected) | Index mapping type (actual) | \n|-------|-----------------------------|-----------------------------|\n| host.name | `keyword` | `text` |\n| source.ip | `ip` | `text` |\n\n#### Incompatible field values - .ds-packetbeat-8.6.1-2023.02.04-000001\n\n\n| Field | ECS values (expected) | Document values (actual) | \n|-------|-----------------------|--------------------------|\n| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `an_invalid_category` (2), `theory` (1) |\n\n',
],
pattern: 'packetbeat-*',
sameFamily: 0,
},
},
sizeInBytes: 1464758182,
@ -330,11 +371,12 @@ describe('helpers', () => {
markdownComments: [
'### .ds-packetbeat-8.6.1-2023.02.04-000001\n',
'| Result | Index | Docs | Incompatible fields | ILM Phase | Size |\n|--------|-------|------|---------------------|-----------|------|\n| ❌ | .ds-packetbeat-8.6.1-2023.02.04-000001 | 1,628,343 () | 3 | `hot` | 697.7MB |\n\n',
'### **Incompatible fields** `3` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
"#### 3 incompatible fields, 0 fields with mappings in the same family\n\nFields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.1.\n\nIncompatible fields with mappings in the same family have exactly the same search behavior but may have different space usage or performance characteristics.\n\nWhen an incompatible field is not in the same family:\n❌ Detection engine rules referencing these fields may not match them correctly\n❌ Pages may not display some events or fields due to unexpected field mappings or values\n❌ Mappings or field values that don't comply with ECS are not supported\n",
'### **Incompatible fields** `3` **Same family** `0` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
"#### 3 incompatible fields\n\nFields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.1.\n\n❌ Detection engine rules referencing these fields may not match them correctly\n❌ Pages may not display some events or fields due to unexpected field mappings or values\n❌ Mappings or field values that don't comply with ECS are not supported\n",
'\n#### Incompatible field mappings - .ds-packetbeat-8.6.1-2023.02.04-000001\n\n\n| Field | ECS mapping type (expected) | Index mapping type (actual) | \n|-------|-----------------------------|-----------------------------|\n| host.name | `keyword` | `text` |\n| source.ip | `ip` | `text` |\n\n#### Incompatible field values - .ds-packetbeat-8.6.1-2023.02.04-000001\n\n\n| Field | ECS values (expected) | Document values (actual) | \n|-------|-----------------------|--------------------------|\n| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `an_invalid_category` (2), `theory` (1) |\n\n',
],
pattern: 'packetbeat-*',
sameFamily: 0,
},
},
sizeInBytes: 1464758182,
@ -478,11 +520,12 @@ describe('helpers', () => {
markdownComments: [
'### .ds-packetbeat-8.6.1-2023.02.04-000001\n',
'| Result | Index | Docs | Incompatible fields | ILM Phase | Size |\n|--------|-------|------|---------------------|-----------|------|\n| ❌ | .ds-packetbeat-8.6.1-2023.02.04-000001 | 1,628,343 (50.0%) | 3 | -- | 697.7MB |\n\n',
'### **Incompatible fields** `3` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
"#### 3 incompatible fields, 0 fields with mappings in the same family\n\nFields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.1.\n\nIncompatible fields with mappings in the same family have exactly the same search behavior but may have different space usage or performance characteristics.\n\nWhen an incompatible field is not in the same family:\n❌ Detection engine rules referencing these fields may not match them correctly\n❌ Pages may not display some events or fields due to unexpected field mappings or values\n❌ Mappings or field values that don't comply with ECS are not supported\n",
'### **Incompatible fields** `3` **Same family** `0` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n',
"#### 3 incompatible fields\n\nFields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.1.\n\n❌ Detection engine rules referencing these fields may not match them correctly\n❌ Pages may not display some events or fields due to unexpected field mappings or values\n❌ Mappings or field values that don't comply with ECS are not supported\n",
'\n#### Incompatible field mappings - .ds-packetbeat-8.6.1-2023.02.04-000001\n\n\n| Field | ECS mapping type (expected) | Index mapping type (actual) | \n|-------|-----------------------------|-----------------------------|\n| host.name | `keyword` | `text` |\n| source.ip | `ip` | `text` |\n\n#### Incompatible field values - .ds-packetbeat-8.6.1-2023.02.04-000001\n\n\n| Field | ECS values (expected) | Document values (actual) | \n|-------|-----------------------|--------------------------|\n| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `an_invalid_category` (2), `theory` (1) |\n\n',
],
pattern: 'packetbeat-*',
sameFamily: 0,
},
},
sizeInBytes: 1464758182,

15
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/use_results_rollup/helpers.ts
View file

@ -12,6 +12,7 @@ import {
getSizeInBytes,
getTotalPatternIncompatible,
getTotalPatternIndicesChecked,
getTotalPatternSameFamily,
} from '../helpers';
import type { IlmPhase, PartitionedFieldMetadata, PatternRollup } from '../types';
@ -65,6 +66,18 @@ export const getTotalIncompatible = (
: undefined;
};
export const getTotalSameFamily = (
patternRollups: Record<string, PatternRollup>
): number | undefined => {
const allRollups = Object.values(patternRollups);
const anyRollupsHaveResults = allRollups.some(({ results }) => results != null);
// only return the total when at least one `PatternRollup` has results:
return anyRollupsHaveResults
? allRollups.reduce((acc, { results }) => acc + (getTotalPatternSameFamily(results) ?? 0), 0)
: undefined;
};
export const getTotalIndicesChecked = (patternRollups: Record<string, PatternRollup>): number => {
const allRollups = Object.values(patternRollups);
@ -137,6 +150,7 @@ export const updateResultOnCheckCompleted = ({
: [];
const incompatible = partitionedFieldMetadata?.incompatible.length;
const sameFamily = partitionedFieldMetadata?.sameFamily.length;
return {
...patternRollups,
@ -152,6 +166,7 @@ export const updateResultOnCheckCompleted = ({
indexName,
markdownComments,
pattern,
sameFamily,
},
},
},

4
x-pack/packages/security-solution/ecs_data_quality_dashboard/impl/data_quality/use_results_rollup/index.tsx
View file

@ -13,6 +13,7 @@ import {
getTotalIncompatible,
getTotalIndices,
getTotalIndicesChecked,
getTotalSameFamily,
getTotalSizeInBytes,
onPatternRollupUpdated,
updateResultOnCheckCompleted,
@ -39,6 +40,7 @@ interface UseResultsRollup {
totalIncompatible: number | undefined;
totalIndices: number | undefined;
totalIndicesChecked: number | undefined;
totalSameFamily: number | undefined;
totalSizeInBytes: number | undefined;
updatePatternIndexNames: ({
indexNames,
@ -67,6 +69,7 @@ export const useResultsRollup = ({ ilmPhases, patterns }: Props): UseResultsRoll
() => getTotalIndicesChecked(patternRollups),
[patternRollups]
);
const totalSameFamily = useMemo(() => getTotalSameFamily(patternRollups), [patternRollups]);
const totalSizeInBytes = useMemo(() => getTotalSizeInBytes(patternRollups), [patternRollups]);
const updatePatternIndexNames = useCallback(
@ -175,6 +178,7 @@ export const useResultsRollup = ({ ilmPhases, patterns }: Props): UseResultsRoll
totalIncompatible,
totalIndices,
totalIndicesChecked,
totalSameFamily,
totalSizeInBytes,
updatePatternIndexNames,
updatePatternRollup,

3
x-pack/plugins/translations/translations/fr-FR.json
View file

@ -4978,7 +4978,6 @@
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.ecsCompliantCallout": "{fieldCount, plural, =1 {Le type de mapping d'index et les valeurs de document pour ce champ sont conformes} one {Le type de mapping d'index et les valeurs de document pour ce champ sont conformes} many {Les types de mapping d'index et les valeurs de document pour ce champ sont conformes} other {Les types de mapping d'index et les valeurs de document pour ce champ sont conformes}} avec la version {version} d'Elastic Common Schema (ECS)",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.ecsCompliantCalloutTitle": "{fieldCount} {fieldCount, plural, =1 {champ} one {champs} many {champs} other {champs}} conforme(s) à ECS",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCallout": "Les champs sont incompatibles avec ECS lorsque les mappings d'index, ou les valeurs des champs de l'index, ne sont pas conformes à la version {version} d'Elastic Common Schema (ECS).",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCalloutTitle": "{fieldCount} {fieldCount, plural, =1 {champ} one {champs} many {champs} other {champs}} incompatible(s), {fieldsInSameFamily} {fieldsInSameFamily, plural, =1 {champ} one {champs} many {champs} other {champs}} avec {fieldsInSameFamily, plural, =1 {un mapping} one {un mapping} many {des mappings} other {des mappings}} dans la même famille",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.summaryMarkdownDescription": "L'index \"{indexName}\" contient des [mappings]({mappingUrl}) ou des valeurs de champs différents des [définitions]({ecsFieldReferenceUrl}) de la version \"{version}\" d'[Elastic Common Schema]({ecsReferenceUrl}) (ECS).",
"securitySolutionPackages.ecsDataQualityDashboard.statLabels.customIndexToolTip": "Décompte des mappings d'index personnalisés dans l'index {indexName}",
"securitySolutionPackages.ecsDataQualityDashboard.statLabels.customPatternToolTip": "Nombre total de mappings d'index personnalisés, dans les index correspondant au modèle {pattern}",
@ -5077,8 +5076,6 @@
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.ecsCompliantMappingsAreFullySupportedMessage": "✅ Les mappings et valeurs de champs conformes à ECS sont totalement pris en charge",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.ecsIsAPermissiveSchemaMessage": "ECS est un schéma permissif. Si vos événements ont des données supplémentaires qui ne peuvent pas être mappées à ECS, vous pouvez tout simplement les ajouter à vos événements à l’aide de noms de champs personnalisés.",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.ecsVersionMarkdownComment": "Version Elastic Common Schema (ECS)",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCallout.incompatibleFieldsWithLabel": "Les champs incompatibles avec des mappings dans la même famille ont exactement le même comportement de recherche, mais ils peuvent avoir une utilisation de l’espace différente ou différentes caractéristiques de performances.",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCallout.whenAnIncompatibleFieldLabel": "Lorsqu'un champ non compatible ne fait pas partie de la même famille :",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleEmptyContent": "Tous les mappings de champs et valeurs de documents de cet index sont conformes à Elastic Common Schema (ECS).",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleEmptyTitle": "Tous les mappings et valeurs de champs sont conformes à ECS",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleFieldsTab": "Champs incompatibles",

3
x-pack/plugins/translations/translations/ja-JP.json
View file

@ -4994,7 +4994,6 @@
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.ecsCompliantCallout": "{fieldCount, plural, =1 {このフィールドのインデックスマッピングタイプとドキュメント値は} other {これらのフィールドのインデックスマッピングタイプとドキュメント値は}}Elastic Common Schema（ECS）バージョン{version}に準拠しています",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.ecsCompliantCalloutTitle": "{fieldCount}個のECS互換フィールド{fieldCount, plural, =1 {フィールド} other {フィールド}}",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCallout": "インデックスのマッピングやインデックスのフィールドの値がElastic Common Schema（ECS）、バージョン{version}に準拠していない場合、フィールドはECSと非互換となります。",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCalloutTitle": "{fieldCount}個の対応していない{fieldCount, plural, =1 {フィールド} other {フィールド}}、{fieldsInSameFamily} {fieldsInSameFamily, plural, =1 {フィールド} other {フィールド}}と同じファミリーの{fieldsInSameFamily, plural, =1 {マッピング} other {マッピング}}",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.summaryMarkdownDescription": "`{indexName}`インデックスは[マッピング]({mappingUrl})またはフィールド値が[Elastic Common Schema]({ecsReferenceUrl})（ECS）、バージョン`{version}`の[定義]({ecsFieldReferenceUrl})と異なっています。",
"securitySolutionPackages.ecsDataQualityDashboard.statLabels.customIndexToolTip": "{indexName}インデックスのカスタムフィールドマッピングの件数",
"securitySolutionPackages.ecsDataQualityDashboard.statLabels.customPatternToolTip": "{pattern}パターンと一致するインデックスのカスタムフィールドマッピングの合計件数",
@ -5093,8 +5092,6 @@
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.ecsCompliantMappingsAreFullySupportedMessage": "✅ ECS互換マッピングおよびフィールド値が完全にサポートされている",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.ecsIsAPermissiveSchemaMessage": "ECSは柔軟なスキーマです。ECSにマッピングできない追加のデータがイベントにある場合は、カスタムフィールド名を使用して、それをイベントに追加できます。",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.ecsVersionMarkdownComment": "Elastic Common Schema（ECS）バージョン",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCallout.incompatibleFieldsWithLabel": "同じファミリーにマッピングがある互換性がないフィールドの検索動作はまったく同じですが、スペースの使用量やパフォーマンス特性は異なる場合があります。",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCallout.whenAnIncompatibleFieldLabel": "互換性がないフィールドが同じファミリーにないとき：",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleEmptyContent": "このインデックスのすべてのフィールドマッピングとドキュメント値がElastic Common Schema（ECS）と互換性があります。",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleEmptyTitle": "すべてのフィールドマッピングと値がECSと互換性があります",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleFieldsTab": "非互換フィールド",

3
x-pack/plugins/translations/translations/zh-CN.json
View file

@ -4993,7 +4993,6 @@
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.ecsCompliantCallout": "{fieldCount, plural, =1 {此字段的索引映射类型和文档值遵循} other {这些字段的索引映射类型和文档值遵循}} Elastic Common Schema (ECS) 版本 {version}",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.ecsCompliantCalloutTitle": "{fieldCount} 个符合 ECS 规范的{fieldCount, plural, =1 {字段} other {字段}}",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCallout": "索引映射或索引中字段的值未遵循 Elastic Common Schema (ECS) 版本 {version} 时，字段将与 ECS 不兼容。",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCalloutTitle": "{fieldCount} 个不兼容{fieldCount, plural, =1 {字段} other {字段}}，同一系列中有 {fieldsInSameFamily} 个{fieldsInSameFamily, plural, =1 {字段} other {字段}}具有{fieldsInSameFamily, plural, =1 {映射} other {映射}}",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.summaryMarkdownDescription": "`{indexName}` 索引具有与 [Elastic Common Schema]({ecsReferenceUrl}) (ECS) 版本 `{version}` [定义]({ecsFieldReferenceUrl}) 不同的[映射]({mappingUrl}) 或字段值。",
"securitySolutionPackages.ecsDataQualityDashboard.statLabels.customIndexToolTip": "{indexName} 索引中定制字段映射的计数",
"securitySolutionPackages.ecsDataQualityDashboard.statLabels.customPatternToolTip": "与 {pattern} 模式匹配的索引中定制字段映射的总计数",
@ -5092,8 +5091,6 @@
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.ecsCompliantMappingsAreFullySupportedMessage": "✅ 完全支持符合 ECS 规范的映射和字段值",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.ecsIsAPermissiveSchemaMessage": "ECS 是一种允许使用的架构。如果您的事件具有其他无法映射为 ECS 的数据，您可以使用定制字段名称直接将其添加到事件中。",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.ecsVersionMarkdownComment": "Elastic Common Schema (ECS) 版本",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCallout.incompatibleFieldsWithLabel": "同一系列中包含映射的不兼容字段具有完全相同的搜索行为，但工作区使用情况和性能特征可能会有所不同。",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCallout.whenAnIncompatibleFieldLabel": "当不兼容字段不在同一系列中时：",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleEmptyContent": "此索引中的所有字段映射和文档值均符合 Elastic Common Schema (ECS) 规范。",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleEmptyTitle": "所有字段映射和值均符合 ECS 规范",
"securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleFieldsTab": "不兼容的字段",