github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-24 17:59:23 -04:00
Code Issues Projects Releases Packages Wiki Activity

[Security Solution][CTI] Update legacy CTI signals to latest ECS threat fields (#107988) (#108628)

Browse source 
* WIP: Adding integration test

* Replace threat.indicator mappings with threat.enrichments mappings

The nested threat.indicator mappings were experimental, and replaced by
threat.enrichmentsin ECS 1.10. While these fields are also experimental,
they fix the conflict between CTI data's normal threat.indicator
mappings.

* Add threat.enrichments mappings to our signals template mappings

event.* is no longer nested within here; it was determined that event
fields were not relevant to enrichment. All relevant ECS fieldsets
(file, pe, etc) are now nested under threat.enrichments.

* Update snapshot with newest threat.enrichments mappings

This test is a snapshot of the actual mappings applied by our templates. Looks good to me!

* Update ECS types to match latest

We now have two threat fields we care about for CTI, for legacy and
official ECS.

* Add a basic test for behavior of legacy enriched signals.

They're still queryable by threat.indicator, meaning that any existing
dashboards will still work.

* WIP: First pass at a data migration for CTI signals

* Defines reindex script to move things around
* Adds integration tests to make sure the migration and new mappings
  work
* Need to test a few more things and verify corner cases
* Need to extract some helpers from tests

* Bump our template version to ensure devs roll over

Marshall bumped to 55, giving us 10 versions for 7.14.x updates.
However, devs would not otherwise roll over and get my mapping updates
without destroying their signals index and rebuilding (which is also not
the same thing, exactly), so this trades having one higher signals
version for a more streamlined dev workflow.

* More robust guard against data migration

We only attempt to migrate legacy enrichments if the document:

* is a signal from an indicator match rule
* has a `threat.indicator` field
* does not have a `threat.enrichments` field

* Minor reorder of operations to make logic clearer

* Add more assertions around our signals data migration

Tests a few more pieces of the resulting document, giving more
confidence that it's the correct transformation (and mappings).

This also modifies/anonymizes the data that was originally generated on
a work machine.

* Remove outdated note

This was for when these tests were driven via the UI; the API is more
responsive and now synchronization is currently needed here, beyond the
200 responses.

* Fix typo in comment

These fields are in ECS 1.11.

* Update snapshot test

We bumped the version previously, causing this test to become outdated.

* Update ECS typings in timelines plugin

These were copied from the security_solution plugin. I updated those,
but neglected to update these.

Until there's a better mechanism for deduplication here, I'm going to
kick the can and update both for now.

* Update enrichments logic to read/write from threat.enrichments

* indicator match rule logic
  * we now simply copy from the specified indicator path, and place that
    in `threat.enrichments.indicator`
* event enrichment API logic
  * We were previously returning fields from `indicator.*`, we now
    include the `indicator.*` suffix in order to be more consistent with
    the sibling `matched.*` fields
* row renderer logic
  * removal of dataset
  * updates relevant to API changes above

* Fix logical error in generating links from indicator fields

We want to link the reference field, not a `first_seen` field.

* Always include the indicator prefix in first-party indicator fields

Prior to this change we would display e.g. `threatintel.indicator.foo`
for investigation enrichment fields. Now that the structure has changed
slightly and we return both `indicator.*` and `matched.*` fields for
existing enrichents, we want to display investigation enrichment
similarly.

* Update indicator match rule integration tests

Now that we've updated our enrichment logic, we need to update our
enrichment tests.

* Remove unused translation

* Update example row renderer data for enriched alerts

* Update parallel CTI constants to get our CTI row renderer working

We were not requesting the necessary fields for our row renderer, since
these constants (specifically CTI_ROW_RENDERER_FIELDS) now exist in both
security_solution and the timelines plugin. I had updated the former,
but only the latter is actually used.

* Update CTI enrichment UI tests

* Update prepackaged threat timeline template with new threat fields

Also bumps the timelineTemplateVersion.

* Update Indicator Match rule tests

These needed three things:

* Update to timeline template (see previous commit)
* Changing expectations from `threat.indicator` to `threat.enrichments`
* Update row renderer expectation to exclude dataset

* Update mock data with newest CTI enrichment fields

* Fix assertion on our threat details

These fields are prefixed with `indicator` now because:

1. This data pertains to the indicator, not the match per se
2. The actual field is prefixed with indicator (or, it at least
   specifies an indicator in the case of a custom threat index (via
   threat_indicator_path))

* Update test data and tests for our field parsing helpers

* Update more event-parsing tests

Ths one involved updating a mock in another package.

* Modify our helper function to support old filebeat indicators

When we query indicators for enrichment matches, the current expectation
is that we'll be querying 7.14 filebeat modules, which have an indicator
path of 'threatintel.indicator'. The only place that matters on the UI
is on the threat intel panel, where these indicators come back with such
a prefix.

This change has one behavior: it brings back the `provider` field on the
Alert summary tab for queried enrichments from filebeat modules.

* Update variable and method names to be more consistent with internal terminology

Indicators come from a CTI index. Enrichments are the application of
indicator data to other documents, and contain both indicator fields and
matched context.

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Co-authored-by: Ryland Herrick <ryalnd@gmail.com>
This commit is contained in:
Kibana Machine 2021-08-14 14:25:57 -04:00 • committed by GitHub
parent 81837aa742
commit 3e3ca30aff
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
42 changed files with 3478 additions and 1048 deletions
Download patch file Download diff file Expand all files Collapse all files

34
packages/kbn-securitysolution-t-grid/src/mock/mock_event_details.ts
View file

@ -39,12 +39,12 @@ export const eventHit = {
'/var/lib/jenkins/workspace/Beats_beats_PR-22624/.gvm/versions/go1.14.7.linux.amd64/bin/go',
],
'source.geo.location': [{ coordinates: [118.7778, 32.0617], type: 'Point' }],
'threat.indicator': [
'threat.enrichments': [
{
'matched.field': ['matched_field', 'other_matched_field'],
first_seen: ['2021-02-22T17:29:25.195Z'],
provider: ['yourself'],
type: ['custom'],
'indicator.first_seen': ['2021-02-22T17:29:25.195Z'],
'indicator.provider': ['yourself'],
'indicator.type': ['custom'],
'matched.atomic': ['matched_atomic'],
lazer: [
{
@ -57,9 +57,9 @@ export const eventHit = {
},
{
'matched.field': ['matched_field_2'],
first_seen: ['2021-02-22T17:29:25.195Z'],
provider: ['other_you'],
type: ['custom'],
'indicator.first_seen': ['2021-02-22T17:29:25.195Z'],
'indicator.provider': ['other_you'],
'indicator.type': ['custom'],
'matched.atomic': ['matched_atomic_2'],
lazer: [
{
@ -259,70 +259,70 @@ export const eventDetailsFormattedFields = [
},
{
category: 'threat',
field: 'threat.indicator.matched.field',
field: 'threat.enrichments.matched.field',
values: ['matched_field', 'other_matched_field', 'matched_field_2'],
originalValue: ['matched_field', 'other_matched_field', 'matched_field_2'],
isObjectArray: false,
},
{
category: 'threat',
field: 'threat.indicator.first_seen',
field: 'threat.enrichments.indicator.first_seen',
values: ['2021-02-22T17:29:25.195Z'],
originalValue: ['2021-02-22T17:29:25.195Z'],
isObjectArray: false,
},
{
category: 'threat',
field: 'threat.indicator.provider',
field: 'threat.enrichments.indicator.provider',
values: ['yourself', 'other_you'],
originalValue: ['yourself', 'other_you'],
isObjectArray: false,
},
{
category: 'threat',
field: 'threat.indicator.type',
field: 'threat.enrichments.indicator.type',
values: ['custom'],
originalValue: ['custom'],
isObjectArray: false,
},
{
category: 'threat',
field: 'threat.indicator.matched.atomic',
field: 'threat.enrichments.matched.atomic',
values: ['matched_atomic', 'matched_atomic_2'],
originalValue: ['matched_atomic', 'matched_atomic_2'],
isObjectArray: false,
},
{
category: 'threat',
field: 'threat.indicator.lazer.great.field',
field: 'threat.enrichments.lazer.great.field',
values: ['grrrrr', 'grrrrr_2'],
originalValue: ['grrrrr', 'grrrrr_2'],
isObjectArray: false,
},
{
category: 'threat',
field: 'threat.indicator.lazer.great.field.wowoe.fooooo',
field: 'threat.enrichments.lazer.great.field.wowoe.fooooo',
values: ['grrrrr'],
originalValue: ['grrrrr'],
isObjectArray: false,
},
{
category: 'threat',
field: 'threat.indicator.lazer.great.field.astring',
field: 'threat.enrichments.lazer.great.field.astring',
values: ['cool'],
originalValue: ['cool'],
isObjectArray: false,
},
{
category: 'threat',
field: 'threat.indicator.lazer.great.field.aNumber',
field: 'threat.enrichments.lazer.great.field.aNumber',
values: ['1'],
originalValue: ['1'],
isObjectArray: false,
},
{
category: 'threat',
field: 'threat.indicator.lazer.great.field.neat',
field: 'threat.enrichments.lazer.great.field.neat',
values: ['true'],
originalValue: ['true'],
isObjectArray: false,

4
x-pack/plugins/security_solution/common/constants.ts
View file

@ -59,9 +59,9 @@ export const SAVED_OBJECTS_MANAGEMENT_FEATURE_ID = 'Saved Objects Management';
export const DEFAULT_SPACE_ID = 'default';
// Document path where threat indicator fields are expected. Fields are used
// to enrich signals, and are copied to threat.indicator.
// to enrich signals, and are copied to threat.enrichments.
export const DEFAULT_INDICATOR_SOURCE_PATH = 'threatintel.indicator';
export const INDICATOR_DESTINATION_PATH = 'threat.indicator';
export const ENRICHMENT_DESTINATION_PATH = 'threat.enrichments';
export const DEFAULT_THREAT_INDEX_KEY = 'securitySolution:defaultThreatIndex';
export const DEFAULT_THREAT_INDEX_VALUE = ['filebeat-*'];

28
x-pack/plugins/security_solution/common/cti/constants.ts
View file

@ -5,7 +5,7 @@
* 2.0.
*/
import { INDICATOR_DESTINATION_PATH } from '../constants';
import { ENRICHMENT_DESTINATION_PATH } from '../constants';
export const MATCHED_ATOMIC = 'matched.atomic';
export const MATCHED_FIELD = 'matched.field';
@ -13,28 +13,26 @@ export const MATCHED_ID = 'matched.id';
export const MATCHED_TYPE = 'matched.type';
export const INDICATOR_MATCH_SUBFIELDS = [MATCHED_ATOMIC, MATCHED_FIELD, MATCHED_TYPE];
export const INDICATOR_MATCHED_ATOMIC = `${INDICATOR_DESTINATION_PATH}.${MATCHED_ATOMIC}`;
export const INDICATOR_MATCHED_FIELD = `${INDICATOR_DESTINATION_PATH}.${MATCHED_FIELD}`;
export const INDICATOR_MATCHED_TYPE = `${INDICATOR_DESTINATION_PATH}.${MATCHED_TYPE}`;
export const INDICATOR_MATCHED_ATOMIC = `${ENRICHMENT_DESTINATION_PATH}.${MATCHED_ATOMIC}`;
export const INDICATOR_MATCHED_FIELD = `${ENRICHMENT_DESTINATION_PATH}.${MATCHED_FIELD}`;
export const INDICATOR_MATCHED_TYPE = `${ENRICHMENT_DESTINATION_PATH}.${MATCHED_TYPE}`;
export const EVENT_DATASET = 'event.dataset';
export const EVENT_REFERENCE = 'event.reference';
export const EVENT_URL = 'event.url';
export const PROVIDER = 'provider';
export const FIRSTSEEN = 'first_seen';
export const INDICATOR_DATASET = `${INDICATOR_DESTINATION_PATH}.${EVENT_DATASET}`;
export const INDICATOR_EVENT_URL = `${INDICATOR_DESTINATION_PATH}.${EVENT_URL}`;
export const INDICATOR_FIRSTSEEN = `${INDICATOR_DESTINATION_PATH}.${FIRSTSEEN}`;
export const INDICATOR_LASTSEEN = `${INDICATOR_DESTINATION_PATH}.last_seen`;
export const INDICATOR_PROVIDER = `${INDICATOR_DESTINATION_PATH}.${PROVIDER}`;
export const INDICATOR_REFERENCE = `${INDICATOR_DESTINATION_PATH}.${EVENT_REFERENCE}`;
export const FIRST_SEEN = 'indicator.first_seen';
export const LAST_SEEN = 'indicator.last_seen';
export const PROVIDER = 'indicator.provider';
export const REFERENCE = 'indicator.reference';
export const INDICATOR_FIRSTSEEN = `${ENRICHMENT_DESTINATION_PATH}.${FIRST_SEEN}`;
export const INDICATOR_LASTSEEN = `${ENRICHMENT_DESTINATION_PATH}.${LAST_SEEN}`;
export const INDICATOR_PROVIDER = `${ENRICHMENT_DESTINATION_PATH}.${PROVIDER}`;
export const INDICATOR_REFERENCE = `${ENRICHMENT_DESTINATION_PATH}.${REFERENCE}`;
export const CTI_ROW_RENDERER_FIELDS = [
INDICATOR_MATCHED_ATOMIC,
INDICATOR_MATCHED_FIELD,
INDICATOR_MATCHED_TYPE,
INDICATOR_DATASET,
INDICATOR_REFERENCE,
INDICATOR_PROVIDER,
];

21
x-pack/plugins/security_solution/common/ecs/threat/index.ts
View file

@ -6,6 +6,7 @@
*/
import { EventEcs } from '../event';
import { UrlEcs } from '../url';
interface ThreatMatchEcs {
atomic?: string[];
@ -15,13 +16,27 @@ interface ThreatMatchEcs {
type?: string[];
}
export interface ThreatIndicatorEcs {
export interface LegacyThreatIndicatorEcs {
domain?: string[];
matched?: ThreatMatchEcs;
event?: EventEcs & { reference?: string[] };
provider?: string[];
type?: string[];
}
export interface ThreatEcs {
indicator: ThreatIndicatorEcs[];
export interface ThreatIndicatorEcs {
url?: UrlEcs;
provider?: string[];
reference?: string[];
type?: string[];
}
export interface ThreatEnrichmentEcs {
indicator?: ThreatIndicatorEcs;
matched?: ThreatMatchEcs;
}
export interface ThreatEcs {
indicator?: LegacyThreatIndicatorEcs[];
enrichments?: ThreatEnrichmentEcs[];
}

38
x-pack/plugins/security_solution/cypress/integration/detection_alerts/cti_enrichments.spec.ts
View file

@ -56,13 +56,13 @@ describe('CTI Enrichment', () => {
it('Displays enrichment matched.* fields on the timeline', () => {
const expectedFields = {
'threat.indicator.matched.atomic': getNewThreatIndicatorRule().atomic,
'threat.indicator.matched.type': 'indicator_match_rule',
'threat.indicator.matched.field': getNewThreatIndicatorRule().indicatorMappingField,
'threat.enrichments.matched.atomic': getNewThreatIndicatorRule().atomic,
'threat.enrichments.matched.type': 'indicator_match_rule',
'threat.enrichments.matched.field': getNewThreatIndicatorRule().indicatorMappingField,
};
const fields = Object.keys(expectedFields) as Array<keyof typeof expectedFields>;
addsFieldsToTimeline('threat.indicator.matched', fields);
addsFieldsToTimeline('threat.enrichments.matched', fields);
fields.forEach((field) => {
cy.get(TIMELINE_FIELD(field)).should('have.text', expectedFields[field]);
@ -75,7 +75,7 @@ describe('CTI Enrichment', () => {
{
line: 3,
text:
' "indicator": "{\\"first_seen\\":\\"2021-03-10T08:02:14.000Z\\",\\"file\\":{\\"size\\":80280,\\"pe\\":{},\\"type\\":\\"elf\\",\\"hash\\":{\\"sha256\\":\\"a04ac6d98ad989312783d4fe3456c53730b212c79a426fb215708b6c6daa3de3\\",\\"tlsh\\":\\"6D7312E017B517CC1371A8353BED205E9128223972AE35302E97528DF957703BAB2DBE\\",\\"ssdeep\\":\\"1536:87vbq1lGAXSEYQjbChaAU2yU23M51DjZgSQAvcYkFtZTjzBht5:8D+CAXFYQChaAUk5ljnQssL\\",\\"md5\\":\\"9b6c3518a91d23ed77504b5416bfb5b3\\"}},\\"type\\":\\"file\\",\\"event\\":{\\"reference\\":\\"https://urlhaus-api.abuse.ch/v1/download/a04ac6d98ad989312783d4fe3456c53730b212c79a426fb215708b6c6daa3de3/\\",\\"ingested\\":\\"2021-03-10T14:51:09.809069Z\\",\\"created\\":\\"2021-03-10T14:51:07.663Z\\",\\"kind\\":\\"enrichment\\",\\"module\\":\\"threatintel\\",\\"category\\":\\"threat\\",\\"type\\":\\"indicator\\",\\"dataset\\":\\"threatintel.abusemalware\\"},\\"matched\\":{\\"atomic\\":\\"a04ac6d98ad989312783d4fe3456c53730b212c79a426fb215708b6c6daa3de3\\",\\"field\\":\\"myhash.mysha256\\",\\"id\\":\\"84cf452c1e0375c3d4412cb550bd1783358468a3b3b777da4829d72c7d6fb74f\\",\\"index\\":\\"filebeat-7.12.0-2021.03.10-000001\\",\\"type\\":\\"indicator_match_rule\\"}}"',
' "enrichments": "{\\"indicator\\":{\\"first_seen\\":\\"2021-03-10T08:02:14.000Z\\",\\"file\\":{\\"size\\":80280,\\"pe\\":{},\\"type\\":\\"elf\\",\\"hash\\":{\\"sha256\\":\\"a04ac6d98ad989312783d4fe3456c53730b212c79a426fb215708b6c6daa3de3\\",\\"tlsh\\":\\"6D7312E017B517CC1371A8353BED205E9128223972AE35302E97528DF957703BAB2DBE\\",\\"ssdeep\\":\\"1536:87vbq1lGAXSEYQjbChaAU2yU23M51DjZgSQAvcYkFtZTjzBht5:8D+CAXFYQChaAUk5ljnQssL\\",\\"md5\\":\\"9b6c3518a91d23ed77504b5416bfb5b3\\"}},\\"type\\":\\"file\\"},\\"matched\\":{\\"atomic\\":\\"a04ac6d98ad989312783d4fe3456c53730b212c79a426fb215708b6c6daa3de3\\",\\"field\\":\\"myhash.mysha256\\",\\"id\\":\\"84cf452c1e0375c3d4412cb550bd1783358468a3b3b777da4829d72c7d6fb74f\\",\\"index\\":\\"filebeat-7.12.0-2021.03.10-000001\\",\\"type\\":\\"indicator_match_rule\\"}}"',
},
{ line: 2, text: ' }' },
];
@ -97,34 +97,23 @@ describe('CTI Enrichment', () => {
it('Displays threat indicator details on the threat intel tab', () => {
const expectedThreatIndicatorData = [
{ field: 'event.category', value: 'threat' },
{ field: 'event.created', value: '2021-03-10T14:51:07.663Z' },
{ field: 'event.dataset', value: 'threatintel.abusemalware' },
{ field: 'event.ingested', value: '2021-03-10T14:51:09.809069Z' },
{ field: 'event.kind', value: 'enrichment' },
{ field: 'event.module', value: 'threatintel' },
{ field: 'indicator.file.hash.md5', value: '9b6c3518a91d23ed77504b5416bfb5b3' },
{
field: 'event.reference',
value:
'https://urlhaus-api.abuse.ch/v1/download/a04ac6d98ad989312783d4fe3456c53730b212c79a426fb215708b6c6daa3de3/(opens in a new tab or window)',
},
{ field: 'event.type', value: 'indicator' },
{ field: 'file.hash.md5', value: '9b6c3518a91d23ed77504b5416bfb5b3' },
{
field: 'file.hash.sha256',
field: 'indicator.file.hash.sha256',
value: 'a04ac6d98ad989312783d4fe3456c53730b212c79a426fb215708b6c6daa3de3',
},
{
field: 'file.hash.ssdeep',
field: 'indicator.file.hash.ssdeep',
value: '1536:87vbq1lGAXSEYQjbChaAU2yU23M51DjZgSQAvcYkFtZTjzBht5:8D+CAXFYQChaAUk5ljnQssL',
},
{
field: 'file.hash.tlsh',
field: 'indicator.file.hash.tlsh',
value: '6D7312E017B517CC1371A8353BED205E9128223972AE35302E97528DF957703BAB2DBE',
},
{ field: 'file.size', value: '80280' },
{ field: 'file.type', value: 'elf' },
{ field: 'first_seen', value: '2021-03-10T08:02:14.000Z' },
{ field: 'indicator.file.size', value: '80280' },
{ field: 'indicator.file.type', value: 'elf' },
{ field: 'indicator.first_seen', value: '2021-03-10T08:02:14.000Z' },
{ field: 'indicator.type', value: 'file' },
{
field: 'matched.atomic',
value: 'a04ac6d98ad989312783d4fe3456c53730b212c79a426fb215708b6c6daa3de3',
@ -136,7 +125,6 @@ describe('CTI Enrichment', () => {
},
{ field: 'matched.index', value: 'filebeat-7.12.0-2021.03.10-000001' },
{ field: 'matched.type', value: 'indicator_match_rule' },
{ field: 'type', value: 'file' },
];
expandFirstAlert();

28
x-pack/plugins/security_solution/cypress/integration/detection_rules/indicator_match_rule.spec.ts
View file

@ -490,8 +490,6 @@ describe('indicator match', () => {
it('Investigate alert in timeline', () => {
const accessibilityText = `Press enter for options, or press space to begin dragging.`;
const threatIndicatorPath =
'../../../x-pack/test/security_solution_cypress/es_archives/threat_indicator/data.json';
loadPrepackagedTimelineTemplates();
@ -506,27 +504,21 @@ describe('indicator match', () => {
cy.get(PROVIDER_BADGE).should('have.length', 3);
cy.get(PROVIDER_BADGE).should(
'have.text',
`threat.indicator.matched.atomic: "${
`threat.enrichments.matched.atomic: "${
getNewThreatIndicatorRule().atomic
}"threat.indicator.matched.type: "indicator_match_rule"threat.indicator.matched.field: "${
}"threat.enrichments.matched.type: "indicator_match_rule"threat.enrichments.matched.field: "${
getNewThreatIndicatorRule().indicatorMappingField
}"`
);
cy.readFile(threatIndicatorPath).then((threatIndicator) => {
cy.get(INDICATOR_MATCH_ROW_RENDER).should(
'have.text',
`threat.indicator.matched.field${
getNewThreatIndicatorRule().indicatorMappingField
}${accessibilityText}matched${getNewThreatIndicatorRule().indicatorMappingField}${
getNewThreatIndicatorRule().atomic
}${accessibilityText}threat.indicator.matched.typeindicator_match_rule${accessibilityText}fromthreat.indicator.event.dataset${
threatIndicator.value.source.event.dataset
}${accessibilityText}:threat.indicator.event.reference${
threatIndicator.value.source.event.reference
}(opens in a new tab or window)${accessibilityText}`
);
});
cy.get(INDICATOR_MATCH_ROW_RENDER).should(
'have.text',
`threat.enrichments.matched.field${
getNewThreatIndicatorRule().indicatorMappingField
}${accessibilityText}matched${getNewThreatIndicatorRule().indicatorMappingField}${
getNewThreatIndicatorRule().atomic
}${accessibilityText}threat.enrichments.matched.typeindicator_match_rule${accessibilityText}`
);
});
});

12
x-pack/plugins/security_solution/public/common/components/event_details/__mocks__/index.ts
View file

@ -659,14 +659,14 @@ export const mockAlertDetailsData = [
},
{
category: 'threat',
field: 'threat.indicator',
values: [`{"first_seen":"2021-03-25T18:17:00.000Z"}`],
originalValue: [`{"first_seen":"2021-03-25T18:17:00.000Z"}`],
field: 'threat.enrichments',
values: [`{"indicator":{"first_seen":"2021-03-25T18:17:00.000Z"}}`],
originalValue: [`{"indicator":{"first_seen":"2021-03-25T18:17:00.000Z"}}`],
},
{
category: 'threat',
field: 'threat.indicator.matched',
values: `["file", "url"]`,
originalValue: ['file', 'url'],
field: 'threat.enrichments.matched.field',
values: ['host.name'],
originalValue: ['host.name'],
},
];

329
x-pack/plugins/security_solution/public/common/components/event_details/cti_details/helpers.test.tsx
View file

@ -35,7 +35,7 @@ describe('parseExistingEnrichments', () => {
const data = [
{
category: 'threat',
field: 'threat.indicator',
field: 'threat.enrichments',
isObjectArray: true,
originalValue: ['whoops'],
values: ['whoops'],
@ -48,13 +48,13 @@ describe('parseExistingEnrichments', () => {
const data = [
{
category: 'threat',
field: 'threat.indicator',
field: 'threat.enrichments',
isObjectArray: true,
originalValue: [
`{"first_seen":"2021-03-21T19:40:19.000Z","provider":"geenensp","ip":"192.168.1.19","type":"url","event":{"reference":"https://urlhaus.abuse.ch/url/1055419/","ingested":"2021-03-08T19:40:44.213673Z","created":"2021-03-08T19:40:43.160Z","kind":"other","module":"threatintel","category":"threat","type":"indicator","dataset":"threatintel.abuseurl"},"matched":{"atomic":"192.168.1.19","field":"host.ip","id":"0SIZMnoB_Blp1Ib9ZYHU","index":"filebeat-8.0.0-2021.05.28-000001","type":"url"}}`,
`{"indicator":{"first_seen":"2021-03-21T19:40:19.000Z","provider":"provider","reference":"http://reference.url","ip":"192.168.1.19","type":"ip"},"matched":{"atomic":"192.168.1.19","field":"host.ip","id":"0SIZMnoB_Blp1Ib9ZYHU","index":"filebeat-8.0.0-2021.05.28-000001","type":"indicator_match_rule"}}`,
],
values: [
`{"first_seen":"2021-03-21T19:40:19.000Z","provider":"geenensp","ip":"192.168.1.19","type":"url","event":{"reference":"https://urlhaus.abuse.ch/url/1055419/","ingested":"2021-03-08T19:40:44.213673Z","created":"2021-03-08T19:40:43.160Z","kind":"other","module":"threatintel","category":"threat","type":"indicator","dataset":"threatintel.abuseurl"},"matched":{"atomic":"192.168.1.19","field":"host.ip","id":"0SIZMnoB_Blp1Ib9ZYHU","index":"filebeat-8.0.0-2021.05.28-000001","type":"url"}}`,
`{"indicator":{"first_seen":"2021-03-21T19:40:19.000Z","provider":"provider","reference":"http://reference.url","ip":"192.168.1.19","type":"ip"},"matched":{"atomic":"192.168.1.19","field":"host.ip","id":"0SIZMnoB_Blp1Ib9ZYHU","index":"filebeat-8.0.0-2021.05.28-000001","type":"indicator_match_rule"}}`,
],
},
];
@ -62,88 +62,39 @@ describe('parseExistingEnrichments', () => {
expect(parseExistingEnrichments(data)).toEqual([
[
{
category: 'first_seen',
field: 'first_seen',
category: 'indicator',
field: 'indicator.first_seen',
isObjectArray: false,
originalValue: ['2021-03-21T19:40:19.000Z'],
values: ['2021-03-21T19:40:19.000Z'],
},
{
category: 'provider',
field: 'provider',
category: 'indicator',
field: 'indicator.provider',
isObjectArray: false,
originalValue: ['geenensp'],
values: ['geenensp'],
originalValue: ['provider'],
values: ['provider'],
},
{
category: 'ip',
field: 'ip',
category: 'indicator',
field: 'indicator.reference',
isObjectArray: false,
originalValue: ['http://reference.url'],
values: ['http://reference.url'],
},
{
category: 'indicator',
field: 'indicator.ip',
isObjectArray: false,
originalValue: ['192.168.1.19'],
values: ['192.168.1.19'],
},
{
category: 'type',
field: 'type',
category: 'indicator',
field: 'indicator.type',
isObjectArray: false,
originalValue: ['url'],
values: ['url'],
},
{
category: 'event',
field: 'event.reference',
isObjectArray: false,
originalValue: ['https://urlhaus.abuse.ch/url/1055419/'],
values: ['https://urlhaus.abuse.ch/url/1055419/'],
},
{
category: 'event',
field: 'event.ingested',
isObjectArray: false,
originalValue: ['2021-03-08T19:40:44.213673Z'],
values: ['2021-03-08T19:40:44.213673Z'],
},
{
category: 'event',
field: 'event.created',
isObjectArray: false,
originalValue: ['2021-03-08T19:40:43.160Z'],
values: ['2021-03-08T19:40:43.160Z'],
},
{
category: 'event',
field: 'event.kind',
isObjectArray: false,
originalValue: ['other'],
values: ['other'],
},
{
category: 'event',
field: 'event.module',
isObjectArray: false,
originalValue: ['threatintel'],
values: ['threatintel'],
},
{
category: 'event',
field: 'event.category',
isObjectArray: false,
originalValue: ['threat'],
values: ['threat'],
},
{
category: 'event',
field: 'event.type',
isObjectArray: false,
originalValue: ['indicator'],
values: ['indicator'],
},
{
category: 'event',
field: 'event.dataset',
isObjectArray: false,
originalValue: ['threatintel.abuseurl'],
values: ['threatintel.abuseurl'],
originalValue: ['ip'],
values: ['ip'],
},
{
category: 'matched',
@ -177,8 +128,8 @@ describe('parseExistingEnrichments', () => {
category: 'matched',
field: 'matched.type',
isObjectArray: false,
originalValue: ['url'],
values: ['url'],
originalValue: [ENRICHMENT_TYPES.IndicatorMatchRule],
values: [ENRICHMENT_TYPES.IndicatorMatchRule],
},
],
]);
@ -188,15 +139,15 @@ describe('parseExistingEnrichments', () => {
const data = [
{
category: 'threat',
field: 'threat.indicator',
field: 'threat.enrichments',
isObjectArray: true,
originalValue: [
`{"first_seen":"2021-03-21T19:40:19.000Z","provider":"other","ip":"192.168.1.19","type":"url","event":{"reference":"https://urlhaus.abuse.ch/url/1055419/","ingested":"2021-03-08T19:40:44.213673Z","created":"2021-03-08T19:40:43.160Z","kind":"other","module":"threatintel","category":"threat","type":"indicator","dataset":"threatintel.abuseurl"},"matched":{"atomic":"192.168.1.19","field":"host.ip","id":"iiL9NHoB_Blp1Ib9yoJo","index":"filebeat-8.0.0-2021.05.28-000001","type":"url"}}`,
`{"first_seen":"2021-03-21T19:40:19.000Z","provider":"geenensp","ip":"192.168.1.19","type":"url","event":{"reference":"https://urlhaus.abuse.ch/url/1055419/","ingested":"2021-03-08T19:40:44.213673Z","created":"2021-03-08T19:40:43.160Z","kind":"other","module":"threatintel","category":"threat","type":"indicator","dataset":"threatintel.abuseurl"},"matched":{"atomic":"192.168.1.19","field":"host.ip","id":"0SIZMnoB_Blp1Ib9ZYHU","index":"filebeat-8.0.0-2021.05.28-000001","type":"url"}}`,
`{"indicator":{"first_seen":"2021-03-21T19:40:19.000Z","provider":"provider","reference":"http://reference.url","ip":"192.168.1.19","type":"ip"},"matched":{"atomic":"192.168.1.19","field":"host.ip","id":"0SIZMnoB_Blp1Ib9ZYHU","index":"filebeat-8.0.0-2021.05.28-000001","type":"indicator_match_rule"}}`,
`{"indicator":{"first_seen":"2021-03-21T19:40:19.000Z","provider":"other","reference":"http://reference.url","ip":"192.168.1.19","type":"ip"},"matched":{"atomic":"192.168.1.19","field":"host.ip","id":"iiL9NHoB_Blp1Ib9yoJo","index":"filebeat-8.0.0-2021.05.28-000001","type":"indicator_match_rule"}}`,
],
values: [
`{"first_seen":"2021-03-21T19:40:19.000Z","provider":"other","ip":"192.168.1.19","type":"url","event":{"reference":"https://urlhaus.abuse.ch/url/1055419/","ingested":"2021-03-08T19:40:44.213673Z","created":"2021-03-08T19:40:43.160Z","kind":"other","module":"threatintel","category":"threat","type":"indicator","dataset":"threatintel.abuseurl"},"matched":{"atomic":"192.168.1.19","field":"host.ip","id":"iiL9NHoB_Blp1Ib9yoJo","index":"filebeat-8.0.0-2021.05.28-000001","type":"url"}}`,
`{"first_seen":"2021-03-21T19:40:19.000Z","provider":"geenensp","ip":"192.168.1.19","type":"url","event":{"reference":"https://urlhaus.abuse.ch/url/1055419/","ingested":"2021-03-08T19:40:44.213673Z","created":"2021-03-08T19:40:43.160Z","kind":"other","module":"threatintel","category":"threat","type":"indicator","dataset":"threatintel.abuseurl"},"matched":{"atomic":"192.168.1.19","field":"host.ip","id":"0SIZMnoB_Blp1Ib9ZYHU","index":"filebeat-8.0.0-2021.05.28-000001","type":"url"}}`,
`{"indicator":{"first_seen":"2021-03-21T19:40:19.000Z","provider":"provider","reference":"http://reference.url","ip":"192.168.1.19","type":"ip"},"matched":{"atomic":"192.168.1.19","field":"host.ip","id":"0SIZMnoB_Blp1Ib9ZYHU","index":"filebeat-8.0.0-2021.05.28-000001","type":"indicator_match_rule"}}`,
`{"indicator":{"first_seen":"2021-03-21T19:40:19.000Z","provider":"other","reference":"http://reference.url","ip":"192.168.1.19","type":"ip"},"matched":{"atomic":"192.168.1.19","field":"host.ip","id":"iiL9NHoB_Blp1Ib9yoJo","index":"filebeat-8.0.0-2021.05.28-000001","type":"indicator_match_rule"}}`,
],
},
];
@ -204,74 +155,111 @@ describe('parseExistingEnrichments', () => {
expect(parseExistingEnrichments(data)).toEqual([
expect.arrayContaining([
{
category: 'first_seen',
field: 'first_seen',
category: 'indicator',
field: 'indicator.first_seen',
isObjectArray: false,
originalValue: ['2021-03-21T19:40:19.000Z'],
values: ['2021-03-21T19:40:19.000Z'],
},
{
category: 'provider',
field: 'provider',
category: 'indicator',
field: 'indicator.provider',
isObjectArray: false,
originalValue: ['other'],
values: ['other'],
originalValue: ['provider'],
values: ['provider'],
},
{
category: 'ip',
field: 'ip',
category: 'indicator',
field: 'indicator.reference',
isObjectArray: false,
originalValue: ['http://reference.url'],
values: ['http://reference.url'],
},
{
category: 'indicator',
field: 'indicator.ip',
isObjectArray: false,
originalValue: ['192.168.1.19'],
values: ['192.168.1.19'],
},
{
category: 'type',
field: 'type',
category: 'indicator',
field: 'indicator.type',
isObjectArray: false,
originalValue: ['url'],
values: ['url'],
originalValue: ['ip'],
values: ['ip'],
},
{
category: 'event',
field: 'event.reference',
category: 'matched',
field: 'matched.atomic',
isObjectArray: false,
originalValue: ['https://urlhaus.abuse.ch/url/1055419/'],
values: ['https://urlhaus.abuse.ch/url/1055419/'],
originalValue: ['192.168.1.19'],
values: ['192.168.1.19'],
},
{
category: 'event',
field: 'event.ingested',
category: 'matched',
field: 'matched.field',
isObjectArray: false,
originalValue: ['2021-03-08T19:40:44.213673Z'],
values: ['2021-03-08T19:40:44.213673Z'],
originalValue: ['host.ip'],
values: ['host.ip'],
},
{
category: 'event',
field: 'event.module',
category: 'matched',
field: 'matched.id',
isObjectArray: false,
originalValue: ['threatintel'],
values: ['threatintel'],
originalValue: ['0SIZMnoB_Blp1Ib9ZYHU'],
values: ['0SIZMnoB_Blp1Ib9ZYHU'],
},
{
category: 'event',
field: 'event.category',
category: 'matched',
field: 'matched.index',
isObjectArray: false,
originalValue: ['threat'],
values: ['threat'],
originalValue: ['filebeat-8.0.0-2021.05.28-000001'],
values: ['filebeat-8.0.0-2021.05.28-000001'],
},
{
category: 'event',
field: 'event.type',
category: 'matched',
field: 'matched.type',
isObjectArray: false,
originalValue: ['indicator'],
values: ['indicator'],
originalValue: [ENRICHMENT_TYPES.IndicatorMatchRule],
values: [ENRICHMENT_TYPES.IndicatorMatchRule],
},
]),
expect.arrayContaining([
{
category: 'indicator',
field: 'indicator.first_seen',
isObjectArray: false,
originalValue: ['2021-03-21T19:40:19.000Z'],
values: ['2021-03-21T19:40:19.000Z'],
},
{
category: 'event',
field: 'event.dataset',
category: 'indicator',
field: 'indicator.provider',
isObjectArray: false,
originalValue: ['threatintel.abuseurl'],
values: ['threatintel.abuseurl'],
originalValue: ['other'],
values: ['other'],
},
{
category: 'indicator',
field: 'indicator.reference',
isObjectArray: false,
originalValue: ['http://reference.url'],
values: ['http://reference.url'],
},
{
category: 'indicator',
field: 'indicator.ip',
isObjectArray: false,
originalValue: ['192.168.1.19'],
values: ['192.168.1.19'],
},
{
category: 'indicator',
field: 'indicator.type',
isObjectArray: false,
originalValue: ['ip'],
values: ['ip'],
},
{
category: 'matched',
@ -305,115 +293,8 @@ describe('parseExistingEnrichments', () => {
category: 'matched',
field: 'matched.type',
isObjectArray: false,
originalValue: ['url'],
values: ['url'],
},
]),
expect.arrayContaining([
{
category: 'first_seen',
field: 'first_seen',
isObjectArray: false,
originalValue: ['2021-03-21T19:40:19.000Z'],
values: ['2021-03-21T19:40:19.000Z'],
},
{
category: 'provider',
field: 'provider',
isObjectArray: false,
originalValue: ['geenensp'],
values: ['geenensp'],
},
{
category: 'ip',
field: 'ip',
isObjectArray: false,
originalValue: ['192.168.1.19'],
values: ['192.168.1.19'],
},
{
category: 'type',
field: 'type',
isObjectArray: false,
originalValue: ['url'],
values: ['url'],
},
{
category: 'event',
field: 'event.reference',
isObjectArray: false,
originalValue: ['https://urlhaus.abuse.ch/url/1055419/'],
values: ['https://urlhaus.abuse.ch/url/1055419/'],
},
{
category: 'event',
field: 'event.ingested',
isObjectArray: false,
originalValue: ['2021-03-08T19:40:44.213673Z'],
values: ['2021-03-08T19:40:44.213673Z'],
},
{
category: 'event',
field: 'event.module',
isObjectArray: false,
originalValue: ['threatintel'],
values: ['threatintel'],
},
{
category: 'event',
field: 'event.category',
isObjectArray: false,
originalValue: ['threat'],
values: ['threat'],
},
{
category: 'event',
field: 'event.type',
isObjectArray: false,
originalValue: ['indicator'],
values: ['indicator'],
},
{
category: 'event',
field: 'event.dataset',
isObjectArray: false,
originalValue: ['threatintel.abuseurl'],
values: ['threatintel.abuseurl'],
},
{
category: 'matched',
field: 'matched.atomic',
isObjectArray: false,
originalValue: ['192.168.1.19'],
values: ['192.168.1.19'],
},
{
category: 'matched',
field: 'matched.field',
isObjectArray: false,
originalValue: ['host.ip'],
values: ['host.ip'],
},
{
category: 'matched',
field: 'matched.id',
isObjectArray: false,
originalValue: ['0SIZMnoB_Blp1Ib9ZYHU'],
values: ['0SIZMnoB_Blp1Ib9ZYHU'],
},
{
category: 'matched',
field: 'matched.index',
isObjectArray: false,
originalValue: ['filebeat-8.0.0-2021.05.28-000001'],
values: ['filebeat-8.0.0-2021.05.28-000001'],
},
{
category: 'matched',
field: 'matched.type',
isObjectArray: false,
originalValue: ['url'],
values: ['url'],
originalValue: [ENRICHMENT_TYPES.IndicatorMatchRule],
values: [ENRICHMENT_TYPES.IndicatorMatchRule],
},
]),
]);

18
x-pack/plugins/security_solution/public/common/components/event_details/cti_details/helpers.tsx
View file

@ -6,10 +6,7 @@
*/
import { groupBy } from 'lodash';
import {
DEFAULT_INDICATOR_SOURCE_PATH,
INDICATOR_DESTINATION_PATH,
} from '../../../../../common/constants';
import { ENRICHMENT_DESTINATION_PATH } from '../../../../../common/constants';
import {
ENRICHMENT_TYPES,
MATCHED_ATOMIC,
@ -35,7 +32,7 @@ export const parseExistingEnrichments = (
data: TimelineEventsDetailsItem[]
): TimelineEventsDetailsItem[][] => {
const threatIndicatorField = data.find(
({ field, originalValue }) => field === INDICATOR_DESTINATION_PATH && originalValue
({ field, originalValue }) => field === ENRICHMENT_DESTINATION_PATH && originalValue
);
if (!threatIndicatorField) {
return [];
@ -68,14 +65,15 @@ export const getEnrichmentValue = (enrichment: CtiEnrichment, field: string) =>
getFirstElement(enrichment[field]) as string | undefined;
/**
* These fields (e.g. 'x') may be in one of two keys depending on whether it's
* a new enrichment ('threatintel.indicator.x') or an old indicator alert
* (simply 'x'). Once enrichment has been normalized and we support the new ECS
* fields, this value should always be 'indicator.x';
* These fields (e.g. 'indicator.ip') may be in one of three places depending on whether it's:
* * a queried, legacy filebeat indicator ('threatintel.indicator.ip')
* * a queried, ECS 1.11 filebeat indicator ('threat.indicator.ip')
* * an existing indicator from an enriched alert ('indicator.ip')
*/
export const getShimmedIndicatorValue = (enrichment: CtiEnrichment, field: string) =>
getEnrichmentValue(enrichment, field) ||
getEnrichmentValue(enrichment, `${DEFAULT_INDICATOR_SOURCE_PATH}.${field}`);
getEnrichmentValue(enrichment, `threatintel.${field}`) ||
getEnrichmentValue(enrichment, `threat.${field}`);
export const getEnrichmentIdentifiers = (enrichment: CtiEnrichment): CtiEnrichmentIdentifiers => ({
id: getEnrichmentValue(enrichment, MATCHED_ID),

16
x-pack/plugins/security_solution/public/common/components/event_details/cti_details/threat_details_view.test.tsx
View file

@ -10,7 +10,6 @@ import { mount } from 'enzyme';
import { TestProviders } from '../../../mock';
import { buildEventEnrichmentMock } from '../../../../../common/search_strategy/security_solution/cti/index.mock';
import { FIRSTSEEN } from '../../../../../common/cti/constants';
import { ThreatDetailsView } from './threat_details_view';
describe('ThreatDetailsView', () => {
@ -31,11 +30,10 @@ describe('ThreatDetailsView', () => {
);
});
it('renders anchor links for event.url and event.reference', () => {
it('renders an anchor link for indicator.reference', () => {
const enrichments = [
buildEventEnrichmentMock({
'event.url': ['http://foo.bar'],
'event.reference': ['http://foo.baz'],
'threatintel.indicator.reference': ['http://foo.baz'],
}),
];
const wrapper = mount(
@ -43,7 +41,7 @@ describe('ThreatDetailsView', () => {
<ThreatDetailsView enrichments={enrichments} />
</TestProviders>
);
expect(wrapper.find('a').length).toEqual(2);
expect(wrapper.find('a').length).toEqual(1);
});
it('sorts same type of enrichments by first_seen descending', () => {
@ -52,7 +50,7 @@ describe('ThreatDetailsView', () => {
// this simulates a legacy enrichment from the old indicator match rule,
// where first_seen is available at the top level
const existingEnrichment = buildEventEnrichmentMock({
first_seen: [mostRecentDate],
'indicator.first_seen': [mostRecentDate],
});
delete existingEnrichment['threatintel.indicator.first_seen'];
const newEnrichment = buildEventEnrichmentMock({
@ -70,10 +68,10 @@ describe('ThreatDetailsView', () => {
const firstSeenRows = wrapper
.find('.euiTableRow')
.hostNodes()
.filterWhere((node) => node.text().includes(FIRSTSEEN));
.filterWhere((node) => node.text().includes('first_seen'));
expect(firstSeenRows.map((node) => node.text())).toEqual([
`first_seen${mostRecentDate}`,
`first_seen${olderDate}`,
`indicator.first_seen${mostRecentDate}`,
`indicator.first_seen${olderDate}`,
]);
});

13
x-pack/plugins/security_solution/public/common/components/event_details/cti_details/threat_details_view.tsx
View file

@ -22,12 +22,7 @@ import { FormattedMessage } from '@kbn/i18n/react';
import { StyledEuiInMemoryTable } from '../summary_view';
import { getSummaryColumns, SummaryRow, ThreatDetailsRow } from '../helpers';
import {
FIRSTSEEN,
EVENT_URL,
EVENT_REFERENCE,
ENRICHMENT_TYPES,
} from '../../../../../common/cti/constants';
import { FIRST_SEEN, ENRICHMENT_TYPES, REFERENCE } from '../../../../../common/cti/constants';
import { DEFAULT_INDICATOR_SOURCE_PATH } from '../../../../../common/constants';
import { getFirstElement } from '../../../../../common/utils/data_retrieval';
import { CtiEnrichment } from '../../../../../common/search_strategy/security_solution/cti';
@ -42,7 +37,7 @@ import { InspectButton } from '../../inspect';
import { EnrichmentButtonContent } from './enrichment_button_content';
const getFirstSeen = (enrichment: CtiEnrichment): number => {
const firstSeenValue = getShimmedIndicatorValue(enrichment, FIRSTSEEN);
const firstSeenValue = getShimmedIndicatorValue(enrichment, FIRST_SEEN);
const firstSeenDate = Date.parse(firstSeenValue ?? 'no date');
return Number.isInteger(firstSeenDate) ? firstSeenDate : new Date(-1).valueOf();
};
@ -61,7 +56,7 @@ const ThreatDetailsDescription: React.FC<ThreatDetailsRow['description']> = ({
fieldName,
value,
}) => {
const tooltipChild = [EVENT_URL, EVENT_REFERENCE].includes(fieldName) ? (
const tooltipChild = fieldName.match(REFERENCE) ? (
<EuiLink href={value} target="_blank">
{value}
</EuiLink>
@ -91,7 +86,7 @@ const buildThreatDetailsItems = (enrichment: CtiEnrichment) =>
.sort()
.map((field) => {
const displayField = field.startsWith(DEFAULT_INDICATOR_SOURCE_PATH)
? field.replace(`${DEFAULT_INDICATOR_SOURCE_PATH}.`, '')
? field.replace(`${DEFAULT_INDICATOR_SOURCE_PATH}`, 'indicator')
: field;
return {

2
x-pack/plugins/security_solution/public/common/mock/mock_detection_alerts.ts
View file

@ -90,7 +90,7 @@ export const getThreatMatchDetectionAlert = (overrides: Partial<Ecs> = {}): Ecs
},
},
threat: {
indicator: [
enrichments: [
{
matched: {
atomic: ['matched.atomic'],

11
x-pack/plugins/security_solution/public/common/mock/mock_timeline_data.ts
View file

@ -1095,18 +1095,17 @@ export const mockTimelineData: TimelineItem[] = [
_id: 'BuBP4W0BOpWiDweSoYSg',
timestamp: '2019-10-18T23:59:15.091Z',
threat: {
indicator: [
enrichments: [
{
indicator: {
provider: ['indicator_provider'],
reference: ['https://example.com'],
},
matched: {
atomic: ['192.168.1.1'],
field: ['source.ip'],
type: ['ip'],
},
event: {
dataset: ['threatintel.example_dataset'],
reference: ['https://example.com'],
},
provider: ['indicator_provider'],
},
],
},

1
x-pack/plugins/security_solution/public/timelines/components/timeline/body/renderers/cti/__snapshots__/threat_match_row.test.tsx.snap generated
View file

@ -23,7 +23,6 @@ exports[`ThreatMatchRowView matches the registered snapshot 1`] = `
<IndicatorDetails
contextId="contextId"
eventId="eventId"
indicatorDataset="dataset"
indicatorProvider="provider"
indicatorReference="http://example.com"
indicatorType="domain"

4
x-pack/plugins/security_solution/public/timelines/components/timeline/body/renderers/cti/helpers.ts
View file

@ -8,13 +8,13 @@
import { get, isEmpty } from 'lodash';
import styled from 'styled-components';
import { INDICATOR_DESTINATION_PATH } from '../../../../../../../common/constants';
import { ENRICHMENT_DESTINATION_PATH } from '../../../../../../../common/constants';
import { INDICATOR_MATCH_SUBFIELDS } from '../../../../../../../common/cti/constants';
import { Ecs } from '../../../../../../../common/ecs';
import { ThreatIndicatorEcs } from '../../../../../../../common/ecs/threat';
const getIndicatorEcs = (data: Ecs): ThreatIndicatorEcs[] => {
const threatData = get(data, INDICATOR_DESTINATION_PATH);
const threatData = get(data, ENRICHMENT_DESTINATION_PATH);
if (threatData == null) {
return [];
} else if (!Array.isArray(threatData)) {

25
x-pack/plugins/security_solution/public/timelines/components/timeline/body/renderers/cti/indicator_details.tsx
View file

@ -10,7 +10,6 @@ import { EuiFlexGroup, EuiFlexItem } from '@elastic/eui';
import { FormattedMessage } from '@kbn/i18n/react';
import {
INDICATOR_DATASET,
INDICATOR_MATCHED_TYPE,
INDICATOR_PROVIDER,
INDICATOR_REFERENCE,
@ -22,7 +21,6 @@ import { HorizontalSpacer } from './helpers';
interface IndicatorDetailsProps {
contextId: string;
eventId: string;
indicatorDataset: string | undefined;
indicatorProvider: string | undefined;
indicatorReference: string | undefined;
indicatorType: string | undefined;
@ -32,7 +30,6 @@ interface IndicatorDetailsProps {
export const IndicatorDetails: React.FC<IndicatorDetailsProps> = ({
contextId,
eventId,
indicatorDataset,
indicatorProvider,
indicatorReference,
indicatorType,
@ -58,28 +55,6 @@ export const IndicatorDetails: React.FC<IndicatorDetailsProps> = ({
/>
</EuiFlexItem>
)}
{indicatorDataset && (
<>
<EuiFlexItem grow={false}>
<HorizontalSpacer>
<FormattedMessage
defaultMessage="from"
id="xpack.securitySolution.alerts.rowRenderers.cti.threatMatch.datasetPreposition"
/>
</HorizontalSpacer>
</EuiFlexItem>
<EuiFlexItem grow={false}>
<DraggableBadge
contextId={contextId}
data-test-subj="threat-match-indicator-details-indicator-dataset"
eventId={eventId}
field={INDICATOR_DATASET}
isDraggable={isDraggable}
value={indicatorDataset}
/>
</EuiFlexItem>
</>
)}
{indicatorProvider && (
<>
<EuiFlexItem grow={false} component="span">

26
x-pack/plugins/security_solution/public/timelines/components/timeline/body/renderers/cti/threat_match_row.test.tsx
View file

@ -22,7 +22,6 @@ describe('ThreatMatchRowView', () => {
<ThreatMatchRowView
contextId="contextId"
eventId="eventId"
indicatorDataset="dataset"
indicatorProvider="provider"
indicatorReference="http://example.com"
indicatorType="domain"
@ -39,7 +38,6 @@ describe('ThreatMatchRowView', () => {
<ThreatMatchRowView
contextId="contextId"
eventId="eventId"
indicatorDataset="dataset"
indicatorProvider="provider"
indicatorReference="http://example.com"
indicatorType="domain"
@ -64,7 +62,6 @@ describe('ThreatMatchRowView', () => {
baseProps = {
contextId: 'contextId',
eventId: 'eventId',
indicatorDataset: 'dataset',
indicatorProvider: 'provider',
indicatorReference: 'http://example.com',
indicatorType: 'domain',
@ -117,29 +114,6 @@ describe('ThreatMatchRowView', () => {
expect(indicatorType.exists()).toBeFalsy();
});
it('renders the indicator dataset, if present', () => {
const wrapper = render(baseProps);
const indicatorDataset = wrapper.find(
'[data-test-subj="threat-match-indicator-details-indicator-dataset"]'
);
expect(indicatorDataset.props()).toEqual(
expect.objectContaining({
value: 'dataset',
})
);
});
it('does not render the indicator dataset, if absent', () => {
const wrapper = render({
...baseProps,
indicatorDataset: undefined,
});
const indicatorDataset = wrapper.find(
'[data-test-subj="threat-match-indicator-details-indicator-dataset"]'
);
expect(indicatorDataset.exists()).toBeFalsy();
});
it('renders the indicator provider, if present', () => {
const wrapper = render(baseProps);
const indicatorProvider = wrapper.find(

9
x-pack/plugins/security_solution/public/timelines/components/timeline/body/renderers/cti/threat_match_row.tsx
View file

@ -11,12 +11,11 @@ import { EuiFlexGroup, EuiFlexItem } from '@elastic/eui';
import { Fields } from '../../../../../../../common/search_strategy';
import {
EVENT_DATASET,
EVENT_REFERENCE,
MATCHED_ATOMIC,
MATCHED_FIELD,
MATCHED_TYPE,
PROVIDER,
REFERENCE,
} from '../../../../../../../common/cti/constants';
import { MatchDetails } from './match_details';
import { IndicatorDetails } from './indicator_details';
@ -24,7 +23,6 @@ import { IndicatorDetails } from './indicator_details';
export interface ThreatMatchRowProps {
contextId: string;
eventId: string;
indicatorDataset: string | undefined;
indicatorProvider: string | undefined;
indicatorReference: string | undefined;
indicatorType: string | undefined;
@ -47,8 +45,7 @@ export const ThreatMatchRow = ({
const props = {
contextId,
eventId,
indicatorDataset: get(data, EVENT_DATASET)[0] as string | undefined,
indicatorReference: get(data, EVENT_REFERENCE)[0] as string | undefined,
indicatorReference: get(data, REFERENCE)[0] as string | undefined,
indicatorProvider: get(data, PROVIDER)[0] as string | undefined,
indicatorType: get(data, MATCHED_TYPE)[0] as string | undefined,
isDraggable,
@ -62,7 +59,6 @@ export const ThreatMatchRow = ({
export const ThreatMatchRowView = ({
contextId,
eventId,
indicatorDataset,
indicatorProvider,
indicatorReference,
indicatorType,
@ -90,7 +86,6 @@ export const ThreatMatchRowView = ({
<IndicatorDetails
contextId={contextId}
eventId={eventId}
indicatorDataset={indicatorDataset}
indicatorProvider={indicatorProvider}
indicatorReference={indicatorReference}
indicatorType={indicatorType}

4
x-pack/plugins/security_solution/public/timelines/components/timeline/body/renderers/cti/threat_match_row_renderer.test.tsx
View file

@ -31,7 +31,7 @@ describe('threatMatchRowRenderer', () => {
it('is false for an alert with indicator data but no match', () => {
const indicatorTypeData = getThreatMatchDetectionAlert({
threat: {
indicator: [{ type: ['url'] }],
enrichments: [{ indicator: { type: ['url'] } }],
},
});
expect(threatMatchRowRenderer.isInstance(indicatorTypeData)).toBe(false);
@ -40,7 +40,7 @@ describe('threatMatchRowRenderer', () => {
it('is false for an alert with threat match fields but no data', () => {
const emptyThreatMatchData = getThreatMatchDetectionAlert({
threat: {
indicator: [{ matched: { type: [] } }],
enrichments: [{ matched: { type: [] } }],
},
});
expect(threatMatchRowRenderer.isInstance(emptyThreatMatchData)).toBe(false);

3
x-pack/plugins/security_solution/public/timelines/components/timeline/body/renderers/cti/threat_match_rows.tsx
View file

@ -10,6 +10,7 @@ import { get } from 'lodash';
import React, { Fragment } from 'react';
import styled from 'styled-components';
import { ENRICHMENT_DESTINATION_PATH } from '../../../../../../../common/constants';
import { RowRenderer } from '../../../../../../../common';
import { Fields } from '../../../../../../../common/search_strategy';
import { ID_FIELD_NAME } from '../../../../../../common/components/event_details/event_id';
@ -21,7 +22,7 @@ const SpacedContainer = styled.div`
`;
export const ThreatMatchRows: RowRenderer['renderRow'] = ({ data, isDraggable, timelineId }) => {
const indicators = get(data, 'threat.indicator') as Fields[];
const indicators = get(data, ENRICHMENT_DESTINATION_PATH) as Fields[];
const eventId = get(data, ID_FIELD_NAME);
return (

17
x-pack/plugins/security_solution/server/lib/detection_engine/migrations/create_migration.ts
View file

@ -59,6 +59,23 @@ export const createMigration = async ({
ctx._source.signal._meta = [:];
}
ctx._source.signal._meta.version = params.version;
// migrate enrichments before 7.15 to ECS 1.11
if (ctx._source.signal?.rule?.type == "threat_match" &&
ctx._source.threat?.indicator instanceof List &&
ctx._source.threat?.enrichments == null) {
ctx._source.threat.enrichments = [];
for (indicator in ctx._source.threat.indicator) {
def enrichment = [:];
enrichment.indicator = indicator;
enrichment.indicator.reference = indicator.event?.reference;
enrichment.matched = indicator.matched;
enrichment.indicator.remove("matched");
ctx._source.threat.enrichments.add(enrichment);
}
ctx._source.threat.remove("indicator");
}
`,
params: {
version,

1098
x-pack/plugins/security_solution/server/lib/detection_engine/routes/index/__snapshots__/get_signals_template.test.ts.snap generated
View file

File diff suppressed because it is too large Load diff

10
x-pack/plugins/security_solution/server/lib/detection_engine/routes/index/get_signals_template.ts
View file

@ -29,7 +29,7 @@ import aadFieldConversion from './signal_aad_mapping.json';
incremented by 10 in order to add "room" for the aforementioned patch
release
*/
export const SIGNALS_TEMPLATE_VERSION = 55;
export const SIGNALS_TEMPLATE_VERSION = 56;
/**
@constant
@type {number}
@ -78,12 +78,8 @@ export const getSignalsTemplate = (index: string, spaceId: string, aadIndexAlias
...ecsMapping.mappings.properties.threat,
properties: {
...ecsMapping.mappings.properties.threat.properties,
indicator: {
...otherMapping.mappings.properties.threat.properties.indicator,
properties: {
...otherMapping.mappings.properties.threat.properties.indicator.properties,
event: ecsMapping.mappings.properties.event,
},
enrichments: {
...otherMapping.mappings.properties.threat.properties.enrichments,
},
},
},

1057
x-pack/plugins/security_solution/server/lib/detection_engine/routes/index/other_mappings.json
View file

File diff suppressed because it is too large Load diff

2
x-pack/plugins/security_solution/server/lib/detection_engine/rules/prepackaged_timelines/index.ndjson
View file

@ -11,4 +11,4 @@
{"savedObjectId":null,"version":null,"columns":[{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"@timestamp","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"signal.rule.description","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"event.action","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"process.name","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"The working directory of the process.","columnHeaderType":"not-filtered","id":"process.working_directory","category":"process","type":"string","searchable":null,"example":"/home/alice"},{"indexes":null,"aggregatable":true,"name":null,"description":"Array of process arguments, starting with the absolute path to\nthe executable.\n\nMay be filtered to protect sensitive information.","columnHeaderType":"not-filtered","id":"process.args","category":"process","type":"string","searchable":null,"example":"[\"/usr/bin/ssh\",\"-l\",\"user\",\"10.0.0.16\"]"},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"process.pid","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"Absolute path to the process executable.","columnHeaderType":"not-filtered","id":"process.parent.executable","category":"process","type":"string","searchable":null,"example":"/usr/bin/ssh"},{"indexes":null,"aggregatable":true,"name":null,"description":"Array of process arguments.\n\nMay be filtered to protect sensitive information.","columnHeaderType":"not-filtered","id":"process.parent.args","category":"process","type":"string","searchable":null,"example":"[\"ssh\",\"-l\",\"user\",\"10.0.0.16\"]"},{"indexes":null,"aggregatable":true,"name":null,"description":"Process id.","columnHeaderType":"not-filtered","id":"process.parent.pid","category":"process","type":"number","searchable":null,"example":"4242"},{"indexes":null,"aggregatable":true,"name":null,"description":"Short name or login of the user.","columnHeaderType":"not-filtered","id":"user.name","category":"user","type":"string","searchable":null,"example":"albert"},{"indexes":null,"aggregatable":true,"name":null,"description":"Name of the host.\n\nIt can contain what `hostname` returns on Unix systems, the fully qualified\ndomain name, or a name specified by the user. The sender decides which value\nto use.","columnHeaderType":"not-filtered","id":"host.name","category":"host","type":"string","searchable":null}],"dataProviders":[{"excluded":false,"and":[],"kqlQuery":"","name":"","queryMatch":{"displayValue":"endpoint","field":"agent.type","displayField":"agent.type","value":"endpoint","operator":":"},"id":"timeline-1-4685da24-35c1-43f3-892d-1f926dbf5568","type":"default","enabled":true}],"description":"","eventType":"all","filters":[],"kqlMode":"filter","kqlQuery":{"filterQuery":{"kuery":{"kind":"kuery","expression":""},"serializedQuery":""}},"title":"Generic Endpoint Timeline","timelineType":"template","templateTimelineVersion":2,"templateTimelineId":"db366523-f1c6-4c1f-8731-6ce5ed9e5717","dateRange":{"start":1588161020848,"end":1588162280848},"savedQueryId":null,"sort":{"columnId":"@timestamp","sortDirection":"desc"},"created":1594735857110,"createdBy":"Elastic","updated":1611609999115,"updatedBy":"Elastic","eventNotes":[],"globalNotes":[],"pinnedEventIds":[],"status":"immutable"}
{"savedObjectId":null,"version":null,"columns":[{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"@timestamp","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"signal.rule.description","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"The action captured by the event.\n\nThis describes the information in the event. It is more specific than `event.category`.\nExamples are `group-add`, `process-started`, `file-created`. The value is\nnormally defined by the implementer.","columnHeaderType":"not-filtered","id":"event.action","category":"event","type":"string","searchable":null,"example":"user-password-change"},{"indexes":null,"aggregatable":true,"name":null,"description":"Array of process arguments, starting with the absolute path to\nthe executable.\n\nMay be filtered to protect sensitive information.","columnHeaderType":"not-filtered","id":"process.args","category":"process","type":"string","searchable":null,"example":"[\"/usr/bin/ssh\",\"-l\",\"user\",\"10.0.0.16\"]"},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"process.pid","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"IP address of the source (IPv4 or IPv6).","columnHeaderType":"not-filtered","id":"source.ip","category":"source","type":"ip","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"Port of the source.","columnHeaderType":"not-filtered","id":"source.port","category":"source","type":"number","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"IP address of the destination (IPv4 or IPv6).","columnHeaderType":"not-filtered","id":"destination.ip","category":"destination","type":"ip","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"destination.port","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"Short name or login of the user.","columnHeaderType":"not-filtered","id":"user.name","category":"user","type":"string","searchable":null,"example":"albert"},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"host.name","searchable":null}],"dataProviders":[{"and":[{"enabled":true,"excluded":false,"id":"timeline-1-e37e37c5-a6e7-4338-af30-47bfbc3c0e1e","kqlQuery":"","name":"{destination.ip}","queryMatch":{"displayField":"destination.ip","displayValue":"{destination.ip}","field":"destination.ip","operator":":","value":"{destination.ip}"},"type":"template"}],"enabled":true,"excluded":false,"id":"timeline-1-ec778f01-1802-40f0-9dfb-ed8de1f656cb","kqlQuery":"","name":"{source.ip}","queryMatch":{"displayField":"source.ip","displayValue":"{source.ip}","field":"source.ip","operator":":","value":"{source.ip}"},"type":"template"}],"description":"","eventType":"all","filters":[],"kqlMode":"filter","kqlQuery":{"filterQuery":{"kuery":{"kind":"kuery","expression":""},"serializedQuery":""}},"title":"Generic Network Timeline","timelineType":"template","templateTimelineVersion":2,"templateTimelineId":"91832785-286d-4ebe-b884-1a208d111a70","dateRange":{"start":1588255858373,"end":1588256218373},"savedQueryId":null,"sort":{"columnId":"@timestamp","sortDirection":"desc"},"created":1594735573866,"createdBy":"Elastic","updated":1611609960850,"updatedBy":"Elastic","eventNotes":[],"globalNotes":[],"pinnedEventIds":[],"status":"immutable"}
{"savedObjectId":null,"version":null,"columns":[{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"@timestamp","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"signal.rule.description","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"event.action","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"process.name","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"The working directory of the process.","columnHeaderType":"not-filtered","id":"process.working_directory","category":"process","type":"string","searchable":null,"example":"/home/alice"},{"indexes":null,"aggregatable":true,"name":null,"description":"Array of process arguments, starting with the absolute path to\nthe executable.\n\nMay be filtered to protect sensitive information.","columnHeaderType":"not-filtered","id":"process.args","category":"process","type":"string","searchable":null,"example":"[\"/usr/bin/ssh\",\"-l\",\"user\",\"10.0.0.16\"]"},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"process.pid","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"Absolute path to the process executable.","columnHeaderType":"not-filtered","id":"process.parent.executable","category":"process","type":"string","searchable":null,"example":"/usr/bin/ssh"},{"indexes":null,"aggregatable":true,"name":null,"description":"Array of process arguments.\n\nMay be filtered to protect sensitive information.","columnHeaderType":"not-filtered","id":"process.parent.args","category":"process","type":"string","searchable":null,"example":"[\"ssh\",\"-l\",\"user\",\"10.0.0.16\"]"},{"indexes":null,"aggregatable":true,"name":null,"description":"Process id.","columnHeaderType":"not-filtered","id":"process.parent.pid","category":"process","type":"number","searchable":null,"example":"4242"},{"indexes":null,"aggregatable":true,"name":null,"description":"Short name or login of the user.","columnHeaderType":"not-filtered","id":"user.name","category":"user","type":"string","searchable":null,"example":"albert"},{"indexes":null,"aggregatable":true,"name":null,"description":"Name of the host.\n\nIt can contain what `hostname` returns on Unix systems, the fully qualified\ndomain name, or a name specified by the user. The sender decides which value\nto use.","columnHeaderType":"not-filtered","id":"host.name","category":"host","type":"string","searchable":null}],"dataProviders":[{"excluded":false,"and":[],"kqlQuery":"","name":"{process.name}","queryMatch":{"displayValue":null,"field":"process.name","displayField":null,"value":"{process.name}","operator":":"},"id":"timeline-1-8622010a-61fb-490d-b162-beac9c36a853","type":"template","enabled":true}],"description":"","eventType":"all","filters":[],"kqlMode":"filter","kqlQuery":{"filterQuery":{"kuery":{"kind":"kuery","expression":""},"serializedQuery":""}},"title":"Generic Process Timeline","timelineType":"template","templateTimelineVersion":2,"templateTimelineId":"76e52245-7519-4251-91ab-262fb1a1728c","dateRange":{"start":1588161020848,"end":1588162280848},"savedQueryId":null,"sort":{"columnId":"@timestamp","sortDirection":"desc"},"created":1594735629389,"createdBy":"Elastic","updated":1611609848602,"updatedBy":"Elastic","eventNotes":[],"globalNotes":[],"pinnedEventIds":[],"status":"immutable"}
{"savedObjectId":null,"version":null,"columns":[{"columnHeaderType":"not-filtered","id":"@timestamp"},{"columnHeaderType":"not-filtered","id":"signal.rule.description"},{"aggregatable":true,"description":"The action captured by the event.\n\nThis describes the information in the event. It is more specific than `event.category`.\nExamples are `group-add`, `process-started`, `file-created`. The value is\nnormally defined by the implementer.","columnHeaderType":"not-filtered","id":"event.action","category":"event","type":"string","example":"user-password-change"},{"aggregatable":true,"description":"Array of process arguments, starting with the absolute path to\nthe executable.\n\nMay be filtered to protect sensitive information.","columnHeaderType":"not-filtered","id":"process.args","category":"process","type":"string","example":"[\"/usr/bin/ssh\",\"-l\",\"user\",\"10.0.0.16\"]"},{"columnHeaderType":"not-filtered","id":"process.pid"},{"aggregatable":true,"description":"IP address of the source (IPv4 or IPv6).","columnHeaderType":"not-filtered","id":"source.ip","category":"source","type":"ip"},{"aggregatable":true,"description":"Port of the source.","columnHeaderType":"not-filtered","id":"source.port","category":"source","type":"number"},{"aggregatable":true,"description":"IP address of the destination (IPv4 or IPv6).","columnHeaderType":"not-filtered","id":"destination.ip","category":"destination","type":"ip"},{"columnHeaderType":"not-filtered","id":"destination.port"},{"aggregatable":true,"description":"Short name or login of the user.","columnHeaderType":"not-filtered","id":"user.name","category":"user","type":"string","example":"albert"},{"columnHeaderType":"not-filtered","id":"host.name"}],"dataProviders":[{"excluded":false,"and":[{"excluded":false,"kqlQuery":"","name":"{threat.indicator.matched.type}","queryMatch":{"displayValue":null,"field":"threat.indicator.matched.type","displayField":null,"value":"{threat.indicator.matched.type}","operator":":"},"id":"timeline-1-ae18ef4b-f690-4122-a24d-e13b6818fba8","type":"template","enabled":true},{"excluded":false,"kqlQuery":"","name":"{threat.indicator.matched.field}","queryMatch":{"displayValue":null,"field":"threat.indicator.matched.field","displayField":null,"value":"{threat.indicator.matched.field}","operator":":"},"id":"timeline-1-7b4cf27e-6788-4d8e-9188-7687f0eba0f2","type":"template","enabled":true}],"kqlQuery":"","name":"{threat.indicator.matched.atomic}","queryMatch":{"displayValue":null,"field":"threat.indicator.matched.atomic","displayField":null,"value":"{threat.indicator.matched.atomic}","operator":":"},"id":"timeline-1-7db7d278-a80a-4853-971a-904319c50777","type":"template","enabled":true}],"description":"This Timeline template is for alerts generated by Indicator Match detection rules.","eqlOptions":{"eventCategoryField":"event.category","tiebreakerField":"","timestampField":"@timestamp","query":"","size":100},"eventType":"alert","filters":[],"kqlMode":"filter","kqlQuery":{"filterQuery":{"kuery":{"kind":"kuery","expression":""},"serializedQuery":""}},"indexNames":[".siem-signals-default"],"title":"Generic Threat Match Timeline","timelineType":"template","templateTimelineVersion":1,"templateTimelineId":"495ad7a7-316e-4544-8a0f-9c098daee76e","dateRange":{"start":1588161020848,"end":1588162280848},"savedQueryId":null,"sort":[{"sortDirection":"desc","columnId":"@timestamp"}],"created":1616696609311,"createdBy":"elastic","updated":1616788372794,"updatedBy":"elastic","eventNotes":[],"globalNotes":[],"pinnedEventIds":[],"status":"immutable"}
{"savedObjectId":null,"version":null,"columns":[{"columnHeaderType":"not-filtered","id":"@timestamp"},{"columnHeaderType":"not-filtered","id":"signal.rule.description"},{"aggregatable":true,"description":"The action captured by the event.\n\nThis describes the information in the event. It is more specific than `event.category`.\nExamples are `group-add`, `process-started`, `file-created`. The value is\nnormally defined by the implementer.","columnHeaderType":"not-filtered","id":"event.action","category":"event","type":"string","example":"user-password-change"},{"aggregatable":true,"description":"Array of process arguments, starting with the absolute path to\nthe executable.\n\nMay be filtered to protect sensitive information.","columnHeaderType":"not-filtered","id":"process.args","category":"process","type":"string","example":"[\"/usr/bin/ssh\",\"-l\",\"user\",\"10.0.0.16\"]"},{"columnHeaderType":"not-filtered","id":"process.pid"},{"aggregatable":true,"description":"IP address of the source (IPv4 or IPv6).","columnHeaderType":"not-filtered","id":"source.ip","category":"source","type":"ip"},{"aggregatable":true,"description":"Port of the source.","columnHeaderType":"not-filtered","id":"source.port","category":"source","type":"number"},{"aggregatable":true,"description":"IP address of the destination (IPv4 or IPv6).","columnHeaderType":"not-filtered","id":"destination.ip","category":"destination","type":"ip"},{"columnHeaderType":"not-filtered","id":"destination.port"},{"aggregatable":true,"description":"Short name or login of the user.","columnHeaderType":"not-filtered","id":"user.name","category":"user","type":"string","example":"albert"},{"columnHeaderType":"not-filtered","id":"host.name"}],"dataProviders":[{"excluded":false,"and":[{"excluded":false,"kqlQuery":"","name":"{threat.enrichments.matched.type}","queryMatch":{"displayValue":null,"field":"threat.enrichments.matched.type","displayField":null,"value":"{threat.enrichments.matched.type}","operator":":"},"id":"timeline-1-ae18ef4b-f690-4122-a24d-e13b6818fba8","type":"template","enabled":true},{"excluded":false,"kqlQuery":"","name":"{threat.enrichments.matched.field}","queryMatch":{"displayValue":null,"field":"threat.enrichments.matched.field","displayField":null,"value":"{threat.enrichments.matched.field}","operator":":"},"id":"timeline-1-7b4cf27e-6788-4d8e-9188-7687f0eba0f2","type":"template","enabled":true}],"kqlQuery":"","name":"{threat.enrichments.matched.atomic}","queryMatch":{"displayValue":null,"field":"threat.enrichments.matched.atomic","displayField":null,"value":"{threat.enrichments.matched.atomic}","operator":":"},"id":"timeline-1-7db7d278-a80a-4853-971a-904319c50777","type":"template","enabled":true}],"description":"This Timeline template is for alerts generated by Indicator Match detection rules.","eqlOptions":{"eventCategoryField":"event.category","tiebreakerField":"","timestampField":"@timestamp","query":"","size":100},"eventType":"alert","filters":[],"kqlMode":"filter","kqlQuery":{"filterQuery":{"kuery":{"kind":"kuery","expression":""},"serializedQuery":""}},"indexNames":[".siem-signals-default"],"title":"Generic Threat Match Timeline","timelineType":"template","templateTimelineVersion":2,"templateTimelineId":"495ad7a7-316e-4544-8a0f-9c098daee76e","dateRange":{"start":1588161020848,"end":1588162280848},"savedQueryId":null,"sort":[{"sortDirection":"desc","columnId":"@timestamp"}],"created":1616696609311,"createdBy":"elastic","updated":1616788372794,"updatedBy":"elastic","eventNotes":[],"globalNotes":[],"pinnedEventIds":[],"status":"immutable"}

2
x-pack/plugins/security_solution/server/lib/detection_engine/rules/prepackaged_timelines/threat.json
View file

@ -1 +1 @@
{"savedObjectId":null,"version":null,"columns":[{"columnHeaderType":"not-filtered","id":"@timestamp"},{"columnHeaderType":"not-filtered","id":"signal.rule.description"},{"aggregatable":true,"description":"The action captured by the event.\n\nThis describes the information in the event. It is more specific than `event.category`.\nExamples are `group-add`, `process-started`, `file-created`. The value is\nnormally defined by the implementer.","columnHeaderType":"not-filtered","id":"event.action","category":"event","type":"string","example":"user-password-change"},{"aggregatable":true,"description":"Array of process arguments, starting with the absolute path to\nthe executable.\n\nMay be filtered to protect sensitive information.","columnHeaderType":"not-filtered","id":"process.args","category":"process","type":"string","example":"[\"/usr/bin/ssh\",\"-l\",\"user\",\"10.0.0.16\"]"},{"columnHeaderType":"not-filtered","id":"process.pid"},{"aggregatable":true,"description":"IP address of the source (IPv4 or IPv6).","columnHeaderType":"not-filtered","id":"source.ip","category":"source","type":"ip"},{"aggregatable":true,"description":"Port of the source.","columnHeaderType":"not-filtered","id":"source.port","category":"source","type":"number"},{"aggregatable":true,"description":"IP address of the destination (IPv4 or IPv6).","columnHeaderType":"not-filtered","id":"destination.ip","category":"destination","type":"ip"},{"columnHeaderType":"not-filtered","id":"destination.port"},{"aggregatable":true,"description":"Short name or login of the user.","columnHeaderType":"not-filtered","id":"user.name","category":"user","type":"string","example":"albert"},{"columnHeaderType":"not-filtered","id":"host.name"}],"dataProviders":[{"excluded":false,"and":[{"excluded":false,"kqlQuery":"","name":"{threat.indicator.matched.type}","queryMatch":{"displayValue":null,"field":"threat.indicator.matched.type","displayField":null,"value":"{threat.indicator.matched.type}","operator":":"},"id":"timeline-1-ae18ef4b-f690-4122-a24d-e13b6818fba8","type":"template","enabled":true},{"excluded":false,"kqlQuery":"","name":"{threat.indicator.matched.field}","queryMatch":{"displayValue":null,"field":"threat.indicator.matched.field","displayField":null,"value":"{threat.indicator.matched.field}","operator":":"},"id":"timeline-1-7b4cf27e-6788-4d8e-9188-7687f0eba0f2","type":"template","enabled":true}],"kqlQuery":"","name":"{threat.indicator.matched.atomic}","queryMatch":{"displayValue":null,"field":"threat.indicator.matched.atomic","displayField":null,"value":"{threat.indicator.matched.atomic}","operator":":"},"id":"timeline-1-7db7d278-a80a-4853-971a-904319c50777","type":"template","enabled":true}],"description":"This Timeline template is for alerts generated by Indicator Match detection rules.","eqlOptions":{"eventCategoryField":"event.category","tiebreakerField":"","timestampField":"@timestamp","query":"","size":100},"eventType":"alert","filters":[],"kqlMode":"filter","kqlQuery":{"filterQuery":{"kuery":{"kind":"kuery","expression":""},"serializedQuery":""}},"indexNames":[".siem-signals-default"],"title":"Generic Threat Match Timeline","timelineType":"template","templateTimelineVersion":1,"templateTimelineId":"495ad7a7-316e-4544-8a0f-9c098daee76e","dateRange":{"start":1588161020848,"end":1588162280848},"savedQueryId":null,"sort":[{"sortDirection":"desc","columnId":"@timestamp"}],"created":1616696609311,"createdBy":"elastic","updated":1616788372794,"updatedBy":"elastic","eventNotes":[],"globalNotes":[],"pinnedEventIds":[],"status":"immutable"}
{"savedObjectId":null,"version":null,"columns":[{"columnHeaderType":"not-filtered","id":"@timestamp"},{"columnHeaderType":"not-filtered","id":"signal.rule.description"},{"aggregatable":true,"description":"The action captured by the event.\n\nThis describes the information in the event. It is more specific than `event.category`.\nExamples are `group-add`, `process-started`, `file-created`. The value is\nnormally defined by the implementer.","columnHeaderType":"not-filtered","id":"event.action","category":"event","type":"string","example":"user-password-change"},{"aggregatable":true,"description":"Array of process arguments, starting with the absolute path to\nthe executable.\n\nMay be filtered to protect sensitive information.","columnHeaderType":"not-filtered","id":"process.args","category":"process","type":"string","example":"[\"/usr/bin/ssh\",\"-l\",\"user\",\"10.0.0.16\"]"},{"columnHeaderType":"not-filtered","id":"process.pid"},{"aggregatable":true,"description":"IP address of the source (IPv4 or IPv6).","columnHeaderType":"not-filtered","id":"source.ip","category":"source","type":"ip"},{"aggregatable":true,"description":"Port of the source.","columnHeaderType":"not-filtered","id":"source.port","category":"source","type":"number"},{"aggregatable":true,"description":"IP address of the destination (IPv4 or IPv6).","columnHeaderType":"not-filtered","id":"destination.ip","category":"destination","type":"ip"},{"columnHeaderType":"not-filtered","id":"destination.port"},{"aggregatable":true,"description":"Short name or login of the user.","columnHeaderType":"not-filtered","id":"user.name","category":"user","type":"string","example":"albert"},{"columnHeaderType":"not-filtered","id":"host.name"}],"dataProviders":[{"excluded":false,"and":[{"excluded":false,"kqlQuery":"","name":"{threat.enrichments.matched.type}","queryMatch":{"displayValue":null,"field":"threat.enrichments.matched.type","displayField":null,"value":"{threat.enrichments.matched.type}","operator":":"},"id":"timeline-1-ae18ef4b-f690-4122-a24d-e13b6818fba8","type":"template","enabled":true},{"excluded":false,"kqlQuery":"","name":"{threat.enrichments.matched.field}","queryMatch":{"displayValue":null,"field":"threat.enrichments.matched.field","displayField":null,"value":"{threat.enrichments.matched.field}","operator":":"},"id":"timeline-1-7b4cf27e-6788-4d8e-9188-7687f0eba0f2","type":"template","enabled":true}],"kqlQuery":"","name":"{threat.enrichments.matched.atomic}","queryMatch":{"displayValue":null,"field":"threat.enrichments.matched.atomic","displayField":null,"value":"{threat.enrichments.matched.atomic}","operator":":"},"id":"timeline-1-7db7d278-a80a-4853-971a-904319c50777","type":"template","enabled":true}],"description":"This Timeline template is for alerts generated by Indicator Match detection rules.","eqlOptions":{"eventCategoryField":"event.category","tiebreakerField":"","timestampField":"@timestamp","query":"","size":100},"eventType":"alert","filters":[],"kqlMode":"filter","kqlQuery":{"filterQuery":{"kuery":{"kind":"kuery","expression":""},"serializedQuery":""}},"indexNames":[".siem-signals-default"],"title":"Generic Threat Match Timeline","timelineType":"template","templateTimelineVersion":2,"templateTimelineId":"495ad7a7-316e-4544-8a0f-9c098daee76e","dateRange":{"start":1588161020848,"end":1588162280848},"savedQueryId":null,"sort":[{"sortDirection":"desc","columnId":"@timestamp"}],"created":1616696609311,"createdBy":"elastic","updated":1616788372794,"updatedBy":"elastic","eventNotes":[],"globalNotes":[],"pinnedEventIds":[],"status":"immutable"}

231
x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/enrich_signal_threat_matches.test.ts
View file

@ -6,12 +6,12 @@
*/
import { get } from 'lodash';
import { INDICATOR_DESTINATION_PATH } from '../../../../../common/constants';
import { ENRICHMENT_DESTINATION_PATH } from '../../../../../common/constants';
import { ENRICHMENT_TYPES } from '../../../../../common/cti/constants';
import { getThreatListItemMock } from './build_threat_mapping_filter.mock';
import {
buildMatchedIndicator,
buildEnrichments,
enrichSignalThreatMatches,
groupAndMergeSignalMatches,
} from './enrich_signal_threat_matches';
@ -73,7 +73,7 @@ describe('groupAndMergeSignalMatches', () => {
});
});
describe('buildMatchedIndicator', () => {
describe('buildEnrichments', () => {
let threats: ThreatListItem[];
let queries: ThreatMatchNamedQuery[];
let indicatorPath: string;
@ -85,7 +85,14 @@ describe('buildMatchedIndicator', () => {
_id: '123',
_source: {
event: { dataset: 'abuse.ch', reference: 'https://test.com' },
threat: { indicator: { domain: 'domain_1', other: 'other_1', type: 'type_1' } },
threat: {
indicator: {
domain: 'domain_1',
other: 'other_1',
reference: 'https://test.com',
type: 'type_1',
},
},
},
}),
];
@ -100,91 +107,100 @@ describe('buildMatchedIndicator', () => {
});
it('returns an empty list if queries is empty', () => {
const indicators = buildMatchedIndicator({
const enrichments = buildEnrichments({
queries: [],
threats,
indicatorPath,
});
expect(indicators).toEqual([]);
expect(enrichments).toEqual([]);
});
it('returns the value of the matched indicator as matched.atomic', () => {
const [indicator] = buildMatchedIndicator({
const [enrichment] = buildEnrichments({
queries,
threats,
indicatorPath,
});
expect(get(indicator, 'matched.atomic')).toEqual('domain_1');
expect(get(enrichment, 'matched.atomic')).toEqual('domain_1');
});
it('returns event values as a part of threat', () => {
const [indicator] = buildMatchedIndicator({
it('does not enrich from other fields in the indicator document', () => {
const [enrichment] = buildEnrichments({
queries,
threats,
indicatorPath,
});
const expectedEvent = threats[0]._source!.event;
expect(get(indicator, 'event')).toEqual(expectedEvent);
expect(Object.keys(enrichment)).toEqual(['indicator', 'matched']);
});
it('returns the _id of the matched indicator as matched.id', () => {
const [indicator] = buildMatchedIndicator({
const [enrichment] = buildEnrichments({
queries,
threats,
indicatorPath,
});
expect(get(indicator, 'matched.id')).toEqual('123');
expect(get(enrichment, 'matched.id')).toEqual('123');
});
it('returns the _index of the matched indicator as matched.index', () => {
const [indicator] = buildMatchedIndicator({
const [enrichment] = buildEnrichments({
queries,
threats,
indicatorPath,
});
expect(get(indicator, 'matched.index')).toEqual('threat-index');
expect(get(enrichment, 'matched.index')).toEqual('threat-index');
});
it('returns the field of the matched indicator as matched.field', () => {
const [indicator] = buildMatchedIndicator({
const [enrichment] = buildEnrichments({
queries,
threats,
indicatorPath,
});
expect(get(indicator, 'matched.field')).toEqual('event.field');
expect(get(enrichment, 'matched.field')).toEqual('event.field');
});
it('returns the type of the enrichment as an indicator match type', () => {
const [indicator] = buildMatchedIndicator({
const [enrichment] = buildEnrichments({
queries,
threats,
indicatorPath,
});
expect(get(indicator, 'matched.type')).toEqual(ENRICHMENT_TYPES.IndicatorMatchRule);
expect(get(enrichment, 'matched.type')).toEqual(ENRICHMENT_TYPES.IndicatorMatchRule);
});
it('returns indicators for each provided query', () => {
it('returns enrichments for each provided query', () => {
threats = [
getThreatListItemMock({
_id: '123',
_source: {
event: { reference: 'https://test.com' },
threat: {
indicator: { domain: 'domain_1', other: 'other_1', type: 'type_1' },
indicator: {
domain: 'domain_1',
other: 'other_1',
reference: 'https://test.com',
type: 'type_1',
},
},
},
}),
getThreatListItemMock({
_id: '456',
_source: {
event: { reference: 'https://test2.com' },
threat: { indicator: { domain: 'domain_1', other: 'other_1', type: 'type_1' } },
threat: {
indicator: {
domain: 'domain_1',
other: 'other_1',
reference: 'https://test2.com',
type: 'type_1',
},
},
},
}),
];
@ -193,25 +209,30 @@ describe('buildMatchedIndicator', () => {
getNamedQueryMock({ id: '456', value: 'threat.indicator.other' }),
getNamedQueryMock({ id: '456', value: 'threat.indicator.domain' }),
];
const indicators = buildMatchedIndicator({
const enrichments = buildEnrichments({
queries,
threats,
indicatorPath,
});
expect(indicators).toHaveLength(queries.length);
expect(enrichments).toHaveLength(queries.length);
});
it('returns the indicator data specified at threat.indicator by default', () => {
const indicators = buildMatchedIndicator({
const enrichments = buildEnrichments({
queries,
threats,
indicatorPath,
});
expect(indicators).toEqual([
expect(enrichments).toEqual([
{
domain: 'domain_1',
indicator: {
domain: 'domain_1',
other: 'other_1',
type: 'type_1',
reference: 'https://test.com',
},
matched: {
atomic: 'domain_1',
id: '123',
@ -219,12 +240,6 @@ describe('buildMatchedIndicator', () => {
field: 'event.field',
type: ENRICHMENT_TYPES.IndicatorMatchRule,
},
other: 'other_1',
type: 'type_1',
event: {
reference: 'https://test.com',
dataset: 'abuse.ch',
},
},
]);
});
@ -234,14 +249,12 @@ describe('buildMatchedIndicator', () => {
getThreatListItemMock({
_id: '123',
_source: {
event: {
reference: 'https://test3.com',
},
'threat.indicator.domain': 'domain_1',
custom: {
indicator: {
path: {
indicator_field: 'indicator_field_1',
reference: 'https://test3.com',
type: 'indicator_type',
},
},
@ -250,15 +263,19 @@ describe('buildMatchedIndicator', () => {
}),
];
const indicators = buildMatchedIndicator({
const enrichments = buildEnrichments({
queries,
threats,
indicatorPath: 'custom.indicator.path',
});
expect(indicators).toEqual([
expect(enrichments).toEqual([
{
indicator_field: 'indicator_field_1',
indicator: {
indicator_field: 'indicator_field_1',
reference: 'https://test3.com',
type: 'indicator_type',
},
matched: {
atomic: 'domain_1',
id: '123',
@ -266,10 +283,6 @@ describe('buildMatchedIndicator', () => {
field: 'event.field',
type: ENRICHMENT_TYPES.IndicatorMatchRule,
},
type: 'indicator_type',
event: {
reference: 'https://test3.com',
},
},
]);
});
@ -282,14 +295,15 @@ describe('buildMatchedIndicator', () => {
}),
];
const indicators = buildMatchedIndicator({
const enrichments = buildEnrichments({
queries,
threats,
indicatorPath,
});
expect(indicators).toEqual([
expect(enrichments).toEqual([
{
indicator: {},
matched: {
atomic: undefined,
id: '123',
@ -309,14 +323,15 @@ describe('buildMatchedIndicator', () => {
}),
];
const indicators = buildMatchedIndicator({
const enrichments = buildEnrichments({
queries,
threats,
indicatorPath,
});
expect(indicators).toEqual([
expect(enrichments).toEqual([
{
indicator: {},
matched: {
atomic: undefined,
id: '123',
@ -333,28 +348,29 @@ describe('buildMatchedIndicator', () => {
getThreatListItemMock({
_id: '123',
_source: {
event: {
reference: 'https://test4.com',
},
threat: {
indicator: [
{ domain: 'foo', type: 'first' },
{ domain: 'bar', type: 'second' },
{ domain: 'foo', reference: 'https://test4.com', type: 'first' },
{ domain: 'bar', reference: 'https://test5.com', type: 'second' },
],
},
},
}),
];
const indicators = buildMatchedIndicator({
const enrichments = buildEnrichments({
queries,
threats,
indicatorPath,
});
expect(indicators).toEqual([
expect(enrichments).toEqual([
{
domain: 'foo',
indicator: {
domain: 'foo',
reference: 'https://test4.com',
type: 'first',
},
matched: {
atomic: undefined,
id: '123',
@ -362,10 +378,6 @@ describe('buildMatchedIndicator', () => {
field: 'event.field',
type: ENRICHMENT_TYPES.IndicatorMatchRule,
},
type: 'first',
event: {
reference: 'https://test4.com',
},
},
]);
});
@ -383,7 +395,7 @@ describe('buildMatchedIndicator', () => {
];
expect(() =>
buildMatchedIndicator({
buildEnrichments({
queries,
threats,
indicatorPath,
@ -404,7 +416,7 @@ describe('buildMatchedIndicator', () => {
];
expect(() =>
buildMatchedIndicator({
buildEnrichments({
queries,
threats,
indicatorPath,
@ -452,12 +464,12 @@ describe('enrichSignalThreatMatches', () => {
expect(enrichedSignals.hits.hits).toEqual([]);
});
it('preserves existing threat.indicator objects on signals', async () => {
it('preserves existing threat.enrichments objects on signals', async () => {
const signalHit = getSignalHitMock({
_source: {
'@timestamp': 'mocked',
event: { category: 'malware' },
threat: { indicator: [{ existing: 'indicator' }] },
threat: { enrichments: [{ existing: 'indicator' }] },
},
matched_queries: [matchedQuery],
});
@ -468,12 +480,16 @@ describe('enrichSignalThreatMatches', () => {
indicatorPath
);
const [enrichedHit] = enrichedSignals.hits.hits;
const indicators = get(enrichedHit._source, INDICATOR_DESTINATION_PATH);
const enrichments = get(enrichedHit._source, ENRICHMENT_DESTINATION_PATH);
expect(indicators).toEqual([
expect(enrichments).toEqual([
{ existing: 'indicator' },
{
domain: 'domain_1',
indicator: {
domain: 'domain_1',
other: 'other_1',
type: 'type_1',
},
matched: {
atomic: 'domain_1',
id: '123',
@ -481,11 +497,6 @@ describe('enrichSignalThreatMatches', () => {
field: 'event.field',
type: ENRICHMENT_TYPES.IndicatorMatchRule,
},
other: 'other_1',
type: 'type_1',
event: {
category: 'malware',
},
},
]);
});
@ -502,10 +513,11 @@ describe('enrichSignalThreatMatches', () => {
indicatorPath
);
const [enrichedHit] = enrichedSignals.hits.hits;
const indicators = get(enrichedHit._source, INDICATOR_DESTINATION_PATH);
const enrichments = get(enrichedHit._source, ENRICHMENT_DESTINATION_PATH);
expect(indicators).toEqual([
expect(enrichments).toEqual([
{
indicator: {},
matched: {
atomic: undefined,
id: '123',
@ -517,12 +529,17 @@ describe('enrichSignalThreatMatches', () => {
]);
});
it('preserves an existing threat.indicator object on signals', async () => {
it('preserves an existing threat.enrichments object on signals', async () => {
const signalHit = getSignalHitMock({
_source: {
'@timestamp': 'mocked',
event: { category: 'virus' },
threat: { indicator: { existing: 'indicator' } },
threat: {
enrichments: [
{ indicator: { existing: 'indicator' } },
{ indicator: { existing: 'indicator2' } },
],
},
},
matched_queries: [matchedQuery],
});
@ -533,12 +550,17 @@ describe('enrichSignalThreatMatches', () => {
indicatorPath
);
const [enrichedHit] = enrichedSignals.hits.hits;
const indicators = get(enrichedHit._source, INDICATOR_DESTINATION_PATH);
const enrichments = get(enrichedHit._source, ENRICHMENT_DESTINATION_PATH);
expect(indicators).toEqual([
{ existing: 'indicator' },
expect(enrichments).toEqual([
{ indicator: { existing: 'indicator' } },
{ indicator: { existing: 'indicator2' } },
{
domain: 'domain_1',
indicator: {
domain: 'domain_1',
other: 'other_1',
type: 'type_1',
},
matched: {
atomic: 'domain_1',
id: '123',
@ -546,11 +568,6 @@ describe('enrichSignalThreatMatches', () => {
field: 'event.field',
type: ENRICHMENT_TYPES.IndicatorMatchRule,
},
other: 'other_1',
type: 'type_1',
event: {
category: 'malware',
},
},
]);
});
@ -599,11 +616,15 @@ describe('enrichSignalThreatMatches', () => {
'custom_threat.custom_indicator'
);
const [enrichedHit] = enrichedSignals.hits.hits;
const indicators = get(enrichedHit._source, INDICATOR_DESTINATION_PATH);
const enrichments = get(enrichedHit._source, ENRICHMENT_DESTINATION_PATH);
expect(indicators).toEqual([
expect(enrichments).toEqual([
{
domain: 'custom_domain',
indicator: {
domain: 'custom_domain',
other: 'custom_other',
type: 'custom_type',
},
matched: {
atomic: 'custom_domain',
id: '123',
@ -611,13 +632,11 @@ describe('enrichSignalThreatMatches', () => {
field: 'event.field',
type: ENRICHMENT_TYPES.IndicatorMatchRule,
},
other: 'custom_other',
type: 'custom_type',
},
]);
});
it('merges duplicate matched signals into a single signal with multiple indicators', async () => {
it('merges duplicate matched signals into a single signal with multiple enrichments', async () => {
getMatchedThreats = async () => [
getThreatListItemMock({
_id: '123',
@ -661,11 +680,15 @@ describe('enrichSignalThreatMatches', () => {
expect(enrichedSignals.hits.hits).toHaveLength(1);
const [enrichedHit] = enrichedSignals.hits.hits;
const indicators = get(enrichedHit._source, INDICATOR_DESTINATION_PATH);
const enrichments = get(enrichedHit._source, ENRICHMENT_DESTINATION_PATH);
expect(indicators).toEqual([
expect(enrichments).toEqual([
{
domain: 'domain_1',
indicator: {
domain: 'domain_1',
other: 'other_1',
type: 'type_1',
},
matched: {
atomic: 'domain_1',
id: '123',
@ -673,14 +696,13 @@ describe('enrichSignalThreatMatches', () => {
field: 'event.field',
type: ENRICHMENT_TYPES.IndicatorMatchRule,
},
event: {
category: 'threat',
},
other: 'other_1',
type: 'type_1',
},
{
domain: 'domain_2',
indicator: {
domain: 'domain_2',
other: 'other_2',
type: 'type_2',
},
matched: {
atomic: 'domain_2',
id: '456',
@ -688,11 +710,6 @@ describe('enrichSignalThreatMatches', () => {
field: 'event.other',
type: ENRICHMENT_TYPES.IndicatorMatchRule,
},
event: {
category: 'bad',
},
other: 'other_2',
type: 'type_2',
},
]);
});

26
x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/enrich_signal_threat_matches.ts
View file

@ -11,7 +11,7 @@ import { ENRICHMENT_TYPES } from '../../../../../common/cti/constants';
import type { SignalSearchResponse, SignalSourceHit } from '../types';
import type {
GetMatchedThreats,
ThreatIndicator,
ThreatEnrichment,
ThreatListItem,
ThreatMatchNamedQuery,
} from './types';
@ -40,7 +40,7 @@ export const groupAndMergeSignalMatches = (signalHits: SignalSourceHit[]): Signa
return dedupedHits;
};
export const buildMatchedIndicator = ({
export const buildEnrichments = ({
queries,
threats,
indicatorPath,
@ -48,20 +48,18 @@ export const buildMatchedIndicator = ({
queries: ThreatMatchNamedQuery[];
threats: ThreatListItem[];
indicatorPath: string;
}): ThreatIndicator[] =>
}): ThreatEnrichment[] =>
queries.map((query) => {
const matchedThreat = threats.find((threat) => threat._id === query.id);
const indicatorValue = get(matchedThreat?._source, indicatorPath) as unknown;
const indicator = [indicatorValue].flat()[0] ?? {};
const indicator = ([indicatorValue].flat()[0] ?? {}) as Record<string, unknown>;
if (!isObject(indicator)) {
throw new Error(`Expected indicator field to be an object, but found: ${indicator}`);
}
const atomic = get(matchedThreat?._source, query.value) as unknown;
const event = get(matchedThreat?._source, 'event') as unknown;
return {
...indicator,
event,
indicator,
matched: {
atomic,
field: query.field,
@ -86,8 +84,8 @@ export const enrichSignalThreatMatches = async (
const signalMatches = uniqueHits.map((signalHit) => extractNamedQueries(signalHit));
const matchedThreatIds = [...new Set(signalMatches.flat().map(({ id }) => id))];
const matchedThreats = await getMatchedThreats(matchedThreatIds);
const matchedIndicators = signalMatches.map((queries) =>
buildMatchedIndicator({
const enrichments = signalMatches.map((queries) =>
buildEnrichments({
indicatorPath,
queries,
threats: matchedThreats,
@ -99,12 +97,12 @@ export const enrichSignalThreatMatches = async (
if (!isObject(threat)) {
throw new Error(`Expected threat field to be an object, but found: ${threat}`);
}
// We are not using INDICATOR_DESTINATION_PATH here because the code above
// and below make assumptions about its current value, 'threat.indicator',
// We are not using ENRICHMENT_DESTINATION_PATH here because the code above
// and below make assumptions about its current value, 'threat.enrichments',
// and making this code dynamic on an arbitrary path would introduce several
// new issues.
const existingIndicatorValue = get(signalHit._source, 'threat.indicator') ?? [];
const existingIndicators = [existingIndicatorValue].flat(); // ensure indicators is an array
const existingEnrichmentValue = get(signalHit._source, 'threat.enrichments') ?? [];
const existingEnrichments = [existingEnrichmentValue].flat(); // ensure enrichments is an array
return {
...signalHit,
@ -112,7 +110,7 @@ export const enrichSignalThreatMatches = async (
...signalHit._source!,
threat: {
...threat,
indicator: [...existingIndicators, ...matchedIndicators[i]],
enrichments: [...existingEnrichments, ...enrichments[i]],
},
},
};

5
x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/types.ts
View file

@ -173,8 +173,9 @@ export interface ThreatListDoc {
*/
export type ThreatListItem = estypes.SearchHit<ThreatListDoc>;
export interface ThreatIndicator {
[key: string]: unknown;
export interface ThreatEnrichment {
indicator: Record<string, unknown>;
matched: Record<string, unknown>;
}
export interface SortWithTieBreaker {

23
x-pack/plugins/timelines/common/ecs/threat/index.ts
View file

@ -6,20 +6,37 @@
*/
import { EventEcs } from '../event';
import { UrlEcs } from '../url';
interface ThreatMatchEcs {
atomic?: string[];
field?: string[];
id?: string[];
index?: string[];
type?: string[];
}
export interface ThreatIndicatorEcs {
export interface LegacyThreatIndicatorEcs {
domain?: string[];
matched?: ThreatMatchEcs;
event?: EventEcs & { reference?: string[] };
provider?: string[];
type?: string[];
}
export interface ThreatEcs {
indicator: ThreatIndicatorEcs[];
export interface ThreatIndicatorEcs {
url?: UrlEcs;
provider?: string[];
reference?: string[];
type?: string[];
}
export interface ThreatEnrichmentEcs {
indicator?: ThreatIndicatorEcs;
matched?: ThreatMatchEcs;
}
export interface ThreatEcs {
indicator?: LegacyThreatIndicatorEcs[];
enrichments?: ThreatEnrichmentEcs[];
}

11
x-pack/plugins/timelines/public/mock/mock_timeline_data.ts
View file

@ -1096,18 +1096,17 @@ export const mockTimelineData: TimelineItem[] = [
_id: 'BuBP4W0BOpWiDweSoYSg',
timestamp: '2019-10-18T23:59:15.091Z',
threat: {
indicator: [
enrichments: [
{
indicator: {
provider: ['indicator_provider'],
reference: ['https://example.com'],
},
matched: {
atomic: ['192.168.1.1'],
field: ['source.ip'],
type: ['ip'],
},
event: {
dataset: ['threatintel.example_dataset'],
reference: ['https://example.com'],
},
provider: ['indicator_provider'],
},
],
},

27
x-pack/plugins/timelines/server/search_strategy/timeline/factory/events/all/constants.ts
View file

@ -8,34 +8,33 @@
// import { CTI_ROW_RENDERER_FIELDS } from '../../../../../../common/cti/constants';
// TODO: share with security_solution/common/cti/constants.ts
export const INDICATOR_DESTINATION_PATH = 'threat.indicator';
export const ENRICHMENT_DESTINATION_PATH = 'threat.enrichments';
export const MATCHED_ATOMIC = 'matched.atomic';
export const MATCHED_FIELD = 'matched.field';
export const MATCHED_TYPE = 'matched.type';
export const INDICATOR_MATCH_SUBFIELDS = [MATCHED_ATOMIC, MATCHED_FIELD, MATCHED_TYPE];
export const INDICATOR_MATCHED_ATOMIC = `${INDICATOR_DESTINATION_PATH}.${MATCHED_ATOMIC}`;
export const INDICATOR_MATCHED_FIELD = `${INDICATOR_DESTINATION_PATH}.${MATCHED_FIELD}`;
export const INDICATOR_MATCHED_TYPE = `${INDICATOR_DESTINATION_PATH}.${MATCHED_TYPE}`;
export const INDICATOR_MATCHED_ATOMIC = `${ENRICHMENT_DESTINATION_PATH}.${MATCHED_ATOMIC}`;
export const INDICATOR_MATCHED_FIELD = `${ENRICHMENT_DESTINATION_PATH}.${MATCHED_FIELD}`;
export const INDICATOR_MATCHED_TYPE = `${ENRICHMENT_DESTINATION_PATH}.${MATCHED_TYPE}`;
export const EVENT_DATASET = 'event.dataset';
export const EVENT_REFERENCE = 'event.reference';
export const PROVIDER = 'provider';
export const FIRSTSEEN = 'first_seen';
export const INDICATOR_DATASET = `${INDICATOR_DESTINATION_PATH}.${EVENT_DATASET}`;
export const INDICATOR_EVENT_URL = `${INDICATOR_DESTINATION_PATH}.event.url`;
export const INDICATOR_FIRSTSEEN = `${INDICATOR_DESTINATION_PATH}.${FIRSTSEEN}`;
export const INDICATOR_LASTSEEN = `${INDICATOR_DESTINATION_PATH}.last_seen`;
export const INDICATOR_PROVIDER = `${INDICATOR_DESTINATION_PATH}.${PROVIDER}`;
export const INDICATOR_REFERENCE = `${INDICATOR_DESTINATION_PATH}.${EVENT_REFERENCE}`;
export const FIRST_SEEN = 'indicator.first_seen';
export const LAST_SEEN = 'indicator.last_seen';
export const PROVIDER = 'indicator.provider';
export const REFERENCE = 'indicator.reference';
export const INDICATOR_FIRSTSEEN = `${ENRICHMENT_DESTINATION_PATH}.${FIRST_SEEN}`;
export const INDICATOR_LASTSEEN = `${ENRICHMENT_DESTINATION_PATH}.${LAST_SEEN}`;
export const INDICATOR_PROVIDER = `${ENRICHMENT_DESTINATION_PATH}.${PROVIDER}`;
export const INDICATOR_REFERENCE = `${ENRICHMENT_DESTINATION_PATH}.${REFERENCE}`;
export const CTI_ROW_RENDERER_FIELDS = [
INDICATOR_MATCHED_ATOMIC,
INDICATOR_MATCHED_FIELD,
INDICATOR_MATCHED_TYPE,
INDICATOR_DATASET,
INDICATOR_REFERENCE,
INDICATOR_PROVIDER,
];

24
x-pack/plugins/timelines/server/search_strategy/timeline/factory/events/all/helpers.test.ts
View file

@ -19,7 +19,7 @@ describe('#formatTimelineData', () => {
'destination.ip',
'source.ip',
'source.geo.location',
'threat.indicator.matched.field',
'threat.enrichments.matched.field',
],
TIMELINE_EVENTS_FIELDS,
eventHit
@ -42,7 +42,7 @@ describe('#formatTimelineData', () => {
value: ['beats-ci-immutable-ubuntu-1804-1605624279743236239'],
},
{
field: 'threat.indicator.matched.field',
field: 'threat.enrichments.matched.field',
value: ['matched_field', 'other_matched_field', 'matched_field_2'],
},
{
@ -95,10 +95,10 @@ describe('#formatTimelineData', () => {
name: ['jenkins'],
},
threat: {
indicator: [
enrichments: [
{
event: {
dataset: [],
indicator: {
provider: ['yourself'],
reference: [],
},
matched: {
@ -106,11 +106,10 @@ describe('#formatTimelineData', () => {
field: ['matched_field', 'other_matched_field'],
type: [],
},
provider: ['yourself'],
},
{
event: {
dataset: [],
indicator: {
provider: ['other_you'],
reference: [],
},
matched: {
@ -118,7 +117,6 @@ describe('#formatTimelineData', () => {
field: ['matched_field_2'],
type: [],
},
provider: ['other_you'],
},
],
},
@ -466,9 +464,9 @@ describe('#formatTimelineData', () => {
});
it('builds intermediate objects at multiple levels', () => {
expect(buildObjectForFieldPath('threat.indicator.matched.atomic', eventHit)).toEqual({
expect(buildObjectForFieldPath('threat.enrichments.matched.atomic', eventHit)).toEqual({
threat: {
indicator: [
enrichments: [
{
matched: {
atomic: ['matched_atomic'],
@ -485,9 +483,9 @@ describe('#formatTimelineData', () => {
});
it('preserves multiple values for a single leaf', () => {
expect(buildObjectForFieldPath('threat.indicator.matched.field', eventHit)).toEqual({
expect(buildObjectForFieldPath('threat.enrichments.matched.field', eventHit)).toEqual({
threat: {
indicator: [
enrichments: [
{
matched: {
field: ['matched_field', 'other_matched_field'],

1
x-pack/plugins/translations/translations/ja-JP.json
View file

@ -20169,7 +20169,6 @@
"xpack.securitySolution.alerts.riskScoreMapping.riskScoreMappingTitle": "リスクスコア無効化",
"xpack.securitySolution.alerts.riskScoreMapping.riskScoreTitle": "リスクスコア",
"xpack.securitySolution.alerts.riskScoreMapping.sourceFieldTitle": "ソースフィールド",
"xpack.securitySolution.alerts.rowRenderers.cti.threatMatch.datasetPreposition": "開始",
"xpack.securitySolution.alerts.rowRenderers.cti.threatMatch.matchedVerb": "一致",
"xpack.securitySolution.alerts.rowRenderers.cti.threatMatch.providerPreposition": "提供者",
"xpack.securitySolution.alerts.severityMapping.defaultDescriptionLabel": "このルールで生成されたすべてのアラートの重要度sレベルを選択します。",

1
x-pack/plugins/translations/translations/zh-CN.json
View file

@ -20437,7 +20437,6 @@
"xpack.securitySolution.alerts.riskScoreMapping.riskScoreMappingTitle": "风险分数覆盖",
"xpack.securitySolution.alerts.riskScoreMapping.riskScoreTitle": "风险分数",
"xpack.securitySolution.alerts.riskScoreMapping.sourceFieldTitle": "源字段",
"xpack.securitySolution.alerts.rowRenderers.cti.threatMatch.datasetPreposition": "来自",
"xpack.securitySolution.alerts.rowRenderers.cti.threatMatch.matchedVerb": "已匹配",
"xpack.securitySolution.alerts.rowRenderers.cti.threatMatch.providerPreposition": "提供者",
"xpack.securitySolution.alerts.severityMapping.defaultDescriptionLabel": "选择此规则生成的所有告警的严重性级别。",

157
x-pack/test/detection_engine_api_integration/security_and_spaces/tests/alerts/alerts_compatibility.ts Normal file
View file

@ -0,0 +1,157 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { estypes } from '@elastic/elasticsearch';
import expect from '@kbn/expect';
import {
DETECTION_ENGINE_QUERY_SIGNALS_URL,
DETECTION_ENGINE_SIGNALS_MIGRATION_STATUS_URL,
} from '../../../../../plugins/security_solution/common/constants';
import {
createSignalsIndex,
deleteSignalsIndex,
finalizeSignalsMigration,
startSignalsMigration,
waitFor,
} from '../../../utils';
import { FtrProviderContext } from '../../../common/ftr_provider_context';
import { ThreatEcs } from '../../../../../plugins/security_solution/common/ecs/threat';
// eslint-disable-next-line import/no-default-export
export default ({ getService }: FtrProviderContext) => {
const esArchiver = getService('esArchiver');
const supertest = getService('supertest');
describe('Alerts Compatibility', function () {
describe('CTI', () => {
const expectedDomain = 'elastic.local';
const expectedProvider = 'provider1';
const expectedEnrichmentMatch = {
atomic: expectedDomain,
field: 'host.name',
id: '_tdUD3sBcVT20cvWAkpd',
index: 'filebeat-7.14.0-2021.08.04-000001',
type: 'indicator_match_rule',
};
beforeEach(async () => {
await esArchiver.load(
'x-pack/test/functional/es_archives/security_solution/legacy_cti_signals'
);
await createSignalsIndex(supertest);
});
afterEach(async () => {
await esArchiver.unload(
'x-pack/test/functional/es_archives/security_solution/legacy_cti_signals'
);
await deleteSignalsIndex(supertest);
});
it('allows querying of legacy enriched signals by threat.indicator', async () => {
const {
body: {
hits: { hits },
},
}: { body: estypes.SearchResponse<{ threat: ThreatEcs }> } = await supertest
.post(DETECTION_ENGINE_QUERY_SIGNALS_URL)
.set('kbn-xsrf', 'true')
.send({
query: {
nested: {
path: 'threat.indicator',
query: {
bool: {
should: [
{
exists: {
field: 'threat.indicator.first_seen',
},
},
],
},
},
},
},
})
.expect(200);
expect(hits.length).to.eql(2);
const indicators = hits.flatMap((hit) => hit._source?.threat.indicator);
const indicatorMatches = indicators.map((indicator) => indicator?.matched);
expect(indicatorMatches).to.eql([expectedEnrichmentMatch, expectedEnrichmentMatch]);
const indicatorDomains = indicators.map((indicator) => indicator?.domain);
expect(indicatorDomains).to.eql([expectedDomain, expectedDomain]);
const indicatorProviders = indicators.map((indicator) => indicator?.provider);
expect(indicatorProviders).to.eql([expectedProvider, expectedProvider]);
});
it('migrates legacy enriched signals to be queried by threat.enrichments', async () => {
const {
body: { indices },
}: {
body: { indices: Array<{ index: string; is_outdated: boolean; version: number }> };
} = await supertest
.get(DETECTION_ENGINE_SIGNALS_MIGRATION_STATUS_URL)
.set('kbn-xsrf', 'true')
.query({ from: '2021-08-01' })
.expect(200);
expect(indices.length).to.eql(1);
expect(indices[0].is_outdated).to.eql(true);
const [migration] = await startSignalsMigration({ indices: [indices[0].index], supertest });
await waitFor(async () => {
const [{ completed }] = await finalizeSignalsMigration({
migrationIds: [migration.migration_id],
supertest,
});
return completed === true;
}, `polling finalize_migration until complete`);
const {
body: {
hits: { hits },
},
}: { body: estypes.SearchResponse<{ threat: ThreatEcs }> } = await supertest
.post(DETECTION_ENGINE_QUERY_SIGNALS_URL)
.set('kbn-xsrf', 'true')
.send({
query: {
nested: {
path: 'threat.enrichments',
query: {
bool: {
should: [
{
exists: {
field: 'threat.enrichments.indicator.first_seen',
},
},
],
},
},
},
},
})
.expect(200);
expect(hits.length).to.eql(2);
const enrichments = hits.flatMap((hit) => hit._source?.threat.enrichments);
const enrichmentMatches = enrichments.map((enrichment) => enrichment?.matched);
expect(enrichmentMatches).to.eql([expectedEnrichmentMatch, expectedEnrichmentMatch]);
const enrichmentDomains = enrichments.map(
(enrichment) => enrichment?.indicator?.url?.domain
);
expect(enrichmentDomains).to.eql([expectedDomain, expectedDomain]);
const enrichmentProviders = enrichments.map(
(enrichment) => enrichment?.indicator?.provider
);
expect(enrichmentProviders).to.eql([expectedProvider, expectedProvider]);
});
});
});
};

19
x-pack/test/detection_engine_api_integration/security_and_spaces/tests/alerts/index.ts Normal file
View file

@ -0,0 +1,19 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { FtrProviderContext } from '../../../common/ftr_provider_context';
// eslint-disable-next-line import/no-default-export
export default ({ loadTestFile }: FtrProviderContext): void => {
describe('Detection engine signals/alerts compatibility', function () {
describe('', function () {
this.tags('ciGroup11');
loadTestFile(require.resolve('./alerts_compatibility'));
});
});
};

304
x-pack/test/detection_engine_api_integration/security_and_spaces/tests/create_threat_matching.ts
View file

@ -280,7 +280,7 @@ export default ({ getService }: FtrProviderContext) => {
status: 'open',
},
threat: {
indicator: get(fullSignal, 'threat.indicator'),
enrichments: get(fullSignal, 'threat.enrichments'),
},
});
});
@ -446,21 +446,19 @@ export default ({ getService }: FtrProviderContext) => {
const threats = hits.map((hit) => hit._source?.threat);
expect(threats).to.eql([
{
indicator: [
enrichments: [
{
description: "domain should match the auditbeat hosts' data's source.ip",
domain: '159.89.119.67',
event: {
category: 'threat',
created: '2021-01-26T11:09:05.529Z',
dataset: 'threatintel.abuseurl',
ingested: '2021-01-26T11:09:06.595350Z',
kind: 'enrichment',
module: 'threatintel',
reference: 'https://urlhaus.abuse.ch/url/978783/',
type: 'indicator',
indicator: {
description: "domain should match the auditbeat hosts' data's source.ip",
domain: '159.89.119.67',
first_seen: '2021-01-26T11:09:04.000Z',
provider: 'geenensp',
url: {
full: 'http://159.89.119.67:59600/bin.sh',
scheme: 'http',
},
type: 'url',
},
first_seen: '2021-01-26T11:09:04.000Z',
matched: {
atomic: '159.89.119.67',
id: '978783',
@ -468,31 +466,23 @@ export default ({ getService }: FtrProviderContext) => {
field: 'destination.ip',
type: ENRICHMENT_TYPES.IndicatorMatchRule,
},
provider: 'geenensp',
type: 'url',
url: {
full: 'http://159.89.119.67:59600/bin.sh',
scheme: 'http',
},
},
],
},
{
indicator: [
enrichments: [
{
description: "domain should match the auditbeat hosts' data's source.ip",
domain: '159.89.119.67',
event: {
category: 'threat',
created: '2021-01-26T11:09:05.529Z',
dataset: 'threatintel.abuseurl',
ingested: '2021-01-26T11:09:06.595350Z',
kind: 'enrichment',
module: 'threatintel',
reference: 'https://urlhaus.abuse.ch/url/978783/',
type: 'indicator',
indicator: {
description: "domain should match the auditbeat hosts' data's source.ip",
domain: '159.89.119.67',
first_seen: '2021-01-26T11:09:04.000Z',
provider: 'geenensp',
url: {
full: 'http://159.89.119.67:59600/bin.sh',
scheme: 'http',
},
type: 'url',
},
first_seen: '2021-01-26T11:09:04.000Z',
matched: {
atomic: '159.89.119.67',
id: '978783',
@ -500,12 +490,6 @@ export default ({ getService }: FtrProviderContext) => {
field: 'destination.ip',
type: ENRICHMENT_TYPES.IndicatorMatchRule,
},
provider: 'geenensp',
type: 'url',
url: {
full: 'http://159.89.119.67:59600/bin.sh',
scheme: 'http',
},
},
],
},
@ -549,14 +533,19 @@ export default ({ getService }: FtrProviderContext) => {
const { hits } = signalsOpen.hits;
const [threat] = hits.map((hit) => hit._source?.threat) as Array<{
indicator: unknown[];
enrichments: unknown[];
}>;
assertContains(threat.indicator, [
assertContains(threat.enrichments, [
{
description: 'this should match auditbeat/hosts on both port and ip',
first_seen: '2021-01-26T11:06:03.000Z',
ip: '45.115.45.3',
indicator: {
description: 'this should match auditbeat/hosts on both port and ip',
first_seen: '2021-01-26T11:06:03.000Z',
ip: '45.115.45.3',
port: 57324,
provider: 'geenensp',
type: 'url',
},
matched: {
atomic: '45.115.45.3',
id: '978785',
@ -564,24 +553,16 @@ export default ({ getService }: FtrProviderContext) => {
field: 'source.ip',
type: ENRICHMENT_TYPES.IndicatorMatchRule,
},
port: 57324,
provider: 'geenensp',
type: 'url',
event: {
category: 'threat',
created: '2021-01-26T11:09:05.529Z',
dataset: 'threatintel.abuseurl',
ingested: '2021-01-26T11:09:06.616763Z',
kind: 'enrichment',
module: 'threatintel',
reference: 'https://urlhaus.abuse.ch/url/978782/',
type: 'indicator',
},
},
{
description: 'this should match auditbeat/hosts on ip',
first_seen: '2021-01-26T11:06:03.000Z',
ip: '45.115.45.3',
indicator: {
description: 'this should match auditbeat/hosts on ip',
first_seen: '2021-01-26T11:06:03.000Z',
ip: '45.115.45.3',
provider: 'other_provider',
type: 'ip',
},
matched: {
atomic: '45.115.45.3',
id: '978787',
@ -589,18 +570,6 @@ export default ({ getService }: FtrProviderContext) => {
field: 'source.ip',
type: ENRICHMENT_TYPES.IndicatorMatchRule,
},
provider: 'other_provider',
type: 'ip',
event: {
category: 'threat',
created: '2021-01-26T11:09:05.529Z',
dataset: 'threatintel.abuseurl',
ingested: '2021-01-26T11:09:06.616763Z',
kind: 'enrichment',
module: 'threatintel',
reference: 'https://urlhaus.abuse.ch/url/978782/',
type: 'indicator',
},
},
]);
});
@ -651,14 +620,19 @@ export default ({ getService }: FtrProviderContext) => {
const { hits } = signalsOpen.hits;
const [threat] = hits.map((hit) => hit._source?.threat) as Array<{
indicator: unknown[];
enrichments: unknown[];
}>;
assertContains(threat.indicator, [
assertContains(threat.enrichments, [
{
description: 'this should match auditbeat/hosts on both port and ip',
first_seen: '2021-01-26T11:06:03.000Z',
ip: '45.115.45.3',
indicator: {
description: 'this should match auditbeat/hosts on both port and ip',
first_seen: '2021-01-26T11:06:03.000Z',
ip: '45.115.45.3',
port: 57324,
provider: 'geenensp',
type: 'url',
},
matched: {
atomic: '45.115.45.3',
id: '978785',
@ -666,19 +640,6 @@ export default ({ getService }: FtrProviderContext) => {
field: 'source.ip',
type: ENRICHMENT_TYPES.IndicatorMatchRule,
},
port: 57324,
provider: 'geenensp',
type: 'url',
event: {
category: 'threat',
created: '2021-01-26T11:09:05.529Z',
dataset: 'threatintel.abuseurl',
ingested: '2021-01-26T11:09:06.616763Z',
kind: 'enrichment',
module: 'threatintel',
reference: 'https://urlhaus.abuse.ch/url/978782/',
type: 'indicator',
},
},
// We do not merge matched indicators during enrichment, so in
// certain circumstances a given indicator document could appear
@ -686,9 +647,15 @@ export default ({ getService }: FtrProviderContext) => {
// threat.indicator.matched data). That's the case with the
// first and third indicators matched, here.
{
description: 'this should match auditbeat/hosts on both port and ip',
first_seen: '2021-01-26T11:06:03.000Z',
ip: '45.115.45.3',
indicator: {
description: 'this should match auditbeat/hosts on both port and ip',
first_seen: '2021-01-26T11:06:03.000Z',
ip: '45.115.45.3',
port: 57324,
provider: 'geenensp',
type: 'url',
},
matched: {
atomic: 57324,
id: '978785',
@ -696,24 +663,15 @@ export default ({ getService }: FtrProviderContext) => {
field: 'source.port',
type: ENRICHMENT_TYPES.IndicatorMatchRule,
},
port: 57324,
provider: 'geenensp',
type: 'url',
event: {
category: 'threat',
created: '2021-01-26T11:09:05.529Z',
dataset: 'threatintel.abuseurl',
ingested: '2021-01-26T11:09:06.616763Z',
kind: 'enrichment',
module: 'threatintel',
reference: 'https://urlhaus.abuse.ch/url/978782/',
type: 'indicator',
},
},
{
description: 'this should match auditbeat/hosts on ip',
first_seen: '2021-01-26T11:06:03.000Z',
ip: '45.115.45.3',
indicator: {
description: 'this should match auditbeat/hosts on ip',
first_seen: '2021-01-26T11:06:03.000Z',
ip: '45.115.45.3',
provider: 'other_provider',
type: 'ip',
},
matched: {
atomic: '45.115.45.3',
id: '978787',
@ -721,18 +679,6 @@ export default ({ getService }: FtrProviderContext) => {
field: 'source.ip',
type: ENRICHMENT_TYPES.IndicatorMatchRule,
},
provider: 'other_provider',
type: 'ip',
event: {
category: 'threat',
created: '2021-01-26T11:09:05.529Z',
dataset: 'threatintel.abuseurl',
ingested: '2021-01-26T11:09:06.616763Z',
kind: 'enrichment',
module: 'threatintel',
reference: 'https://urlhaus.abuse.ch/url/978782/',
type: 'indicator',
},
},
]);
});
@ -787,13 +733,23 @@ export default ({ getService }: FtrProviderContext) => {
expect(signalsOpen.hits.hits.length).equal(2);
const { hits } = signalsOpen.hits;
const threats = hits.map((hit) => hit._source?.threat) as Array<{ indicator: unknown[] }>;
const threats = hits.map((hit) => hit._source?.threat) as Array<{
enrichments: unknown[];
}>;
assertContains(threats[0].indicator, [
assertContains(threats[0].enrichments, [
{
description: "domain should match the auditbeat hosts' data's source.ip",
domain: '159.89.119.67',
first_seen: '2021-01-26T11:09:04.000Z',
indicator: {
description: "domain should match the auditbeat hosts' data's source.ip",
domain: '159.89.119.67',
first_seen: '2021-01-26T11:09:04.000Z',
provider: 'geenensp',
type: 'url',
url: {
full: 'http://159.89.119.67:59600/bin.sh',
scheme: 'http',
},
},
matched: {
atomic: '159.89.119.67',
id: '978783',
@ -801,27 +757,16 @@ export default ({ getService }: FtrProviderContext) => {
field: 'destination.ip',
type: ENRICHMENT_TYPES.IndicatorMatchRule,
},
provider: 'geenensp',
type: 'url',
url: {
full: 'http://159.89.119.67:59600/bin.sh',
scheme: 'http',
},
event: {
category: 'threat',
created: '2021-01-26T11:09:05.529Z',
dataset: 'threatintel.abuseurl',
ingested: '2021-01-26T11:09:06.595350Z',
kind: 'enrichment',
module: 'threatintel',
reference: 'https://urlhaus.abuse.ch/url/978783/',
type: 'indicator',
},
},
{
description: 'this should match auditbeat/hosts on both port and ip',
first_seen: '2021-01-26T11:06:03.000Z',
ip: '45.115.45.3',
indicator: {
description: 'this should match auditbeat/hosts on both port and ip',
first_seen: '2021-01-26T11:06:03.000Z',
ip: '45.115.45.3',
port: 57324,
provider: 'geenensp',
type: 'url',
},
matched: {
atomic: '45.115.45.3',
id: '978785',
@ -829,24 +774,16 @@ export default ({ getService }: FtrProviderContext) => {
field: 'source.ip',
type: ENRICHMENT_TYPES.IndicatorMatchRule,
},
port: 57324,
provider: 'geenensp',
type: 'url',
event: {
category: 'threat',
created: '2021-01-26T11:09:05.529Z',
dataset: 'threatintel.abuseurl',
ingested: '2021-01-26T11:09:06.616763Z',
kind: 'enrichment',
module: 'threatintel',
reference: 'https://urlhaus.abuse.ch/url/978782/',
type: 'indicator',
},
},
{
description: 'this should match auditbeat/hosts on both port and ip',
first_seen: '2021-01-26T11:06:03.000Z',
ip: '45.115.45.3',
indicator: {
description: 'this should match auditbeat/hosts on both port and ip',
first_seen: '2021-01-26T11:06:03.000Z',
ip: '45.115.45.3',
port: 57324,
provider: 'geenensp',
type: 'url',
},
matched: {
atomic: 57324,
id: '978785',
@ -854,27 +791,22 @@ export default ({ getService }: FtrProviderContext) => {
field: 'source.port',
type: ENRICHMENT_TYPES.IndicatorMatchRule,
},
port: 57324,
provider: 'geenensp',
type: 'url',
event: {
category: 'threat',
created: '2021-01-26T11:09:05.529Z',
dataset: 'threatintel.abuseurl',
ingested: '2021-01-26T11:09:06.616763Z',
kind: 'enrichment',
module: 'threatintel',
reference: 'https://urlhaus.abuse.ch/url/978782/',
type: 'indicator',
},
},
]);
assertContains(threats[1].indicator, [
assertContains(threats[1].enrichments, [
{
description: "domain should match the auditbeat hosts' data's source.ip",
domain: '159.89.119.67',
first_seen: '2021-01-26T11:09:04.000Z',
indicator: {
description: "domain should match the auditbeat hosts' data's source.ip",
domain: '159.89.119.67',
first_seen: '2021-01-26T11:09:04.000Z',
provider: 'geenensp',
type: 'url',
url: {
full: 'http://159.89.119.67:59600/bin.sh',
scheme: 'http',
},
},
matched: {
atomic: '159.89.119.67',
id: '978783',
@ -882,22 +814,6 @@ export default ({ getService }: FtrProviderContext) => {
field: 'destination.ip',
type: ENRICHMENT_TYPES.IndicatorMatchRule,
},
provider: 'geenensp',
type: 'url',
url: {
full: 'http://159.89.119.67:59600/bin.sh',
scheme: 'http',
},
event: {
category: 'threat',
created: '2021-01-26T11:09:05.529Z',
dataset: 'threatintel.abuseurl',
ingested: '2021-01-26T11:09:06.595350Z',
kind: 'enrichment',
module: 'threatintel',
reference: 'https://urlhaus.abuse.ch/url/978783/',
type: 'indicator',
},
},
]);
});

4
x-pack/test/detection_engine_api_integration/security_and_spaces/tests/index.ts
View file

@ -60,5 +60,9 @@ export default ({ loadTestFile }: FtrProviderContext): void => {
describe('', function () {
loadTestFile(require.resolve('./keyword_family/index'));
});
describe('', function () {
loadTestFile(require.resolve('./alerts/index'));
});
});
};

50
x-pack/test/detection_engine_api_integration/utils.ts
View file

@ -45,6 +45,8 @@ import {
DETECTION_ENGINE_PREPACKAGED_URL,
DETECTION_ENGINE_QUERY_SIGNALS_URL,
DETECTION_ENGINE_RULES_URL,
DETECTION_ENGINE_SIGNALS_FINALIZE_MIGRATION_URL,
DETECTION_ENGINE_SIGNALS_MIGRATION_URL,
INTERNAL_IMMUTABLE_KEY,
INTERNAL_RULE_ID_KEY,
} from '../../plugins/security_solution/common/constants';
@ -1355,6 +1357,54 @@ export const deleteMigrations = async ({
);
};
interface CreateMigrationResponse {
index: string;
migration_index: string;
migration_id: string;
}
export const startSignalsMigration = async ({
indices,
supertest,
}: {
supertest: SuperTest<supertestAsPromised.Test>;
indices: string[];
}): Promise<CreateMigrationResponse[]> => {
const {
body: { indices: created },
}: { body: { indices: CreateMigrationResponse[] } } = await supertest
.post(DETECTION_ENGINE_SIGNALS_MIGRATION_URL)
.set('kbn-xsrf', 'true')
.send({ index: indices })
.expect(200);
return created;
};
interface FinalizeMigrationResponse {
id: string;
completed?: boolean;
error?: unknown;
}
export const finalizeSignalsMigration = async ({
migrationIds,
supertest,
}: {
supertest: SuperTest<supertestAsPromised.Test>;
migrationIds: string[];
}): Promise<FinalizeMigrationResponse[]> => {
const {
body: { migrations },
}: { body: { migrations: FinalizeMigrationResponse[] } } = await supertest
.post(DETECTION_ENGINE_SIGNALS_FINALIZE_MIGRATION_URL)
.set('kbn-xsrf', 'true')
.send({ migration_ids: migrationIds })
.expect(200);
return migrations;
};
export const getOpenSignals = async (
supertest: SuperTest<supertestAsPromised.Test>,
es: KibanaClient,

487
x-pack/test/functional/es_archives/security_solution/legacy_cti_signals/data.json Normal file
View file

@ -0,0 +1,487 @@
{
"type": "doc",
"value": {
"id": "47fc9c7b3220142effa382d78dfa828a5ca79fd5773140af1a1d96045628f171",
"index": ".siem-signals-default-000001",
"source": {
"@timestamp": "2021-08-04T04:14:58.974Z",
"agent": {
"ephemeral_id": "07c24b1e-3663-4372-b982-f2d831e033eb",
"hostname": "elastic.local",
"id": "ce7741d9-3f0a-466d-8ae6-d7d8f883fcec",
"name": "elastic.local",
"type": "auditbeat",
"version": "7.14.0"
},
"ecs": {
"version": "1.10.0"
},
"event": {
"action": "process_stopped",
"category": [
"process"
],
"dataset": "process",
"kind": "signal",
"module": "system",
"type": [
"end"
]
},
"host": {
"architecture": "x86_64",
"hostname": "elastic.local",
"id": "1633D595-A115-5BF5-870B-A471B49446C3",
"ip": [ "192.168.1.1" ],
"mac": [ "aa:bb:cc:dd:ee:ff" ],
"name": "elastic.local",
"os": {
"build": "20G80",
"family": "darwin",
"kernel": "20.6.0",
"name": "Mac OS X",
"platform": "darwin",
"type": "macos",
"version": "10.16"
}
},
"message": "Process mdworker_shared (PID: 32162) by user elastic STOPPED",
"process": {
"args": [
"/System/Library/Frameworks/CoreServices.framework/Frameworks/Metadata.framework/Versions/A/Support/mdworker_shared",
"-s",
"mdworker",
"-c",
"MDSImporterWorker",
"-m",
"com.apple.mdworker.shared"
],
"entity_id": "Bu0panfqz24AEqa+",
"executable": "/System/Library/Frameworks/CoreServices.framework/Frameworks/Metadata.framework/Versions/A/Support/mdworker_shared",
"hash": {
"sha1": "5f3233fd75c14b315731684d59b632df36a731a6"
},
"name": "mdworker_shared",
"pid": 32162,
"ppid": 1,
"start": "2021-08-04T04:07:26.900Z",
"working_directory": "/"
},
"service": {
"type": "system"
},
"signal": {
"_meta": {
"version": 45
},
"ancestors": [
{
"depth": 0,
"id": "yddfD3sBcVT20cvWFEs2",
"index": "auditbeat-7.14.0-2021.08.04-000001",
"type": "event"
}
],
"depth": 1,
"original_event": {
"action": "process_stopped",
"category": [
"process"
],
"dataset": "process",
"kind": "event",
"module": "system",
"type": [
"end"
]
},
"original_time": "2021-08-04T04:14:53.490Z",
"parent": {
"depth": 0,
"id": "yddfD3sBcVT20cvWFEs2",
"index": "auditbeat-7.14.0-2021.08.04-000001",
"type": "event"
},
"parents": [
{
"depth": 0,
"id": "yddfD3sBcVT20cvWFEs2",
"index": "auditbeat-7.14.0-2021.08.04-000001",
"type": "event"
}
],
"rule": {
"actions": [
],
"author": [
],
"created_at": "2021-08-04T04:14:51.576Z",
"created_by": "elastic",
"description": "These should match a dummy indicator in filebeat.",
"enabled": true,
"exceptions_list": [
],
"false_positives": [
],
"filters": [
],
"from": "now-1200s",
"id": "832f86f0-f4da-11eb-989d-b758d09dbc85",
"immutable": false,
"index": [
"auditbeat-*"
],
"interval": "5m",
"language": "kuery",
"license": "",
"max_signals": 100,
"meta": {
"from": "15m",
"kibana_siem_app_url": "http://localhost:5601/app/security"
},
"name": "Indicator Match on host.name",
"output_index": ".siem-signals-default",
"query": "host.name: \"elastic.local\" ",
"references": [
],
"risk_score": 21,
"risk_score_mapping": [
],
"rule_id": "2cc5f982-0de6-4325-9225-cdf88d3731ce",
"severity": "low",
"severity_mapping": [
],
"tags": [
],
"threat": [
],
"threat_filters": [
],
"threat_index": [
"filebeat-*"
],
"threat_indicator_path": "",
"threat_language": "kuery",
"threat_mapping": [
{
"entries": [
{
"field": "host.name",
"type": "mapping",
"value": "threatintel.indicator.domain"
}
]
}
],
"threat_query": "threatintel.indicator.type : \"url\"",
"throttle": null,
"to": "now",
"type": "threat_match",
"updated_at": "2021-08-04T04:14:52.129Z",
"updated_by": "elastic",
"version": 1
},
"status": "open"
},
"threat": {
"indicator": [
{
"domain": "elastic.local",
"event": {
"category": "threat",
"created": "2021-08-04T03:53:30.761Z",
"dataset": "threatintel.abuseurl",
"ingested": "2021-08-04T03:53:37.514040Z",
"kind": "enrichment",
"module": "threatintel",
"reference": "https://urlhaus.abuse.ch/url/12345/",
"type": "indicator"
},
"first_seen": "2021-08-03T20:35:17.000Z",
"matched": {
"atomic": "elastic.local",
"field": "host.name",
"id": "_tdUD3sBcVT20cvWAkpd",
"index": "filebeat-7.14.0-2021.08.04-000001",
"type": "indicator_match_rule"
},
"provider": "provider1",
"type": "url",
"url": {
"domain": "elastic.local",
"extension": "php",
"full": "http://elastic.local/thing",
"original": "http://elastic.local/thing",
"path": "/thing",
"scheme": "http"
}
}
]
},
"user": {
"effective": {
"group": {
"id": "20"
},
"id": "501"
},
"group": {
"id": "20",
"name": "staff"
},
"id": "501",
"name": "elastic",
"saved": {
"group": {
"id": "20"
},
"id": "501"
}
}
}
}
}
{
"type": "doc",
"value": {
"id": "0527411874b23bcea85daf5bf7dcacd144536ba6d92d3230a4a0acfb7de7f512",
"index": ".siem-signals-default-000001",
"source": {
"@timestamp": "2021-08-04T04:14:58.973Z",
"agent": {
"ephemeral_id": "07c24b1e-3663-4372-b982-f2d831e033eb",
"hostname": "elastic.local",
"id": "ce7741d9-3f0a-466d-8ae6-d7d8f883fcec",
"name": "elastic.local",
"type": "auditbeat",
"version": "7.14.0"
},
"ecs": {
"version": "1.10.0"
},
"event": {
"action": "process_started",
"category": [
"process"
],
"dataset": "process",
"kind": "signal",
"module": "system",
"type": [
"start"
]
},
"host": {
"architecture": "x86_64",
"hostname": "elastic.local",
"id": "1633D595-A115-5BF5-870B-A471B49446C3",
"ip": [ "192.168.1.1" ],
"mac": [ "aa:bb:cc:dd:ee:ff" ],
"name": "elastic.local",
"os": {
"build": "20G80",
"family": "darwin",
"kernel": "20.6.0",
"name": "Mac OS X",
"platform": "darwin",
"type": "macos",
"version": "10.16"
}
},
"message": "Process mdworker_shared (PID: 32306) by user elastic STARTED",
"process": {
"args": [
"/System/Library/Frameworks/CoreServices.framework/Frameworks/Metadata.framework/Versions/A/Support/mdworker_shared",
"-s",
"mdworker",
"-c",
"MDSImporterWorker",
"-m",
"com.apple.mdworker.shared"
],
"entity_id": "wfc7zUuEinqxUbZ6",
"executable": "/System/Library/Frameworks/CoreServices.framework/Frameworks/Metadata.framework/Versions/A/Support/mdworker_shared",
"hash": {
"sha1": "5f3233fd75c14b315731684d59b632df36a731a6"
},
"name": "mdworker_shared",
"pid": 32306,
"ppid": 1,
"start": "2021-08-04T04:14:48.830Z",
"working_directory": "/"
},
"service": {
"type": "system"
},
"signal": {
"_meta": {
"version": 45
},
"ancestors": [
{
"depth": 0,
"id": "yNdfD3sBcVT20cvWFEs2",
"index": "auditbeat-7.14.0-2021.08.04-000001",
"type": "event"
}
],
"depth": 1,
"original_event": {
"action": "process_started",
"category": [
"process"
],
"dataset": "process",
"kind": "event",
"module": "system",
"type": [
"start"
]
},
"original_time": "2021-08-04T04:14:53.490Z",
"parent": {
"depth": 0,
"id": "yNdfD3sBcVT20cvWFEs2",
"index": "auditbeat-7.14.0-2021.08.04-000001",
"type": "event"
},
"parents": [
{
"depth": 0,
"id": "yNdfD3sBcVT20cvWFEs2",
"index": "auditbeat-7.14.0-2021.08.04-000001",
"type": "event"
}
],
"rule": {
"actions": [
],
"author": [
],
"created_at": "2021-08-04T04:14:51.576Z",
"created_by": "elastic",
"description": "These should match a dummy indicator in filebeat.",
"enabled": true,
"exceptions_list": [
],
"false_positives": [
],
"filters": [
],
"from": "now-1200s",
"id": "832f86f0-f4da-11eb-989d-b758d09dbc85",
"immutable": false,
"index": [
"auditbeat-*"
],
"interval": "5m",
"language": "kuery",
"license": "",
"max_signals": 100,
"meta": {
"from": "15m",
"kibana_siem_app_url": "http://localhost:5601/app/security"
},
"name": "Indicator Match on host.name",
"output_index": ".siem-signals-default",
"query": "host.name: \"elastic.local\" ",
"references": [
],
"risk_score": 21,
"risk_score_mapping": [
],
"rule_id": "2cc5f982-0de6-4325-9225-cdf88d3731ce",
"severity": "low",
"severity_mapping": [
],
"tags": [
],
"threat": [
],
"threat_filters": [
],
"threat_index": [
"filebeat-*"
],
"threat_indicator_path": "",
"threat_language": "kuery",
"threat_mapping": [
{
"entries": [
{
"field": "host.name",
"type": "mapping",
"value": "threatintel.indicator.domain"
}
]
}
],
"threat_query": "threatintel.indicator.type : \"url\"",
"throttle": null,
"to": "now",
"type": "threat_match",
"updated_at": "2021-08-04T04:14:52.129Z",
"updated_by": "elastic",
"version": 1
},
"status": "open"
},
"threat": {
"indicator": [
{
"domain": "elastic.local",
"event": {
"category": "threat",
"created": "2021-08-04T03:53:30.761Z",
"dataset": "threatintel.abuseurl",
"ingested": "2021-08-04T03:53:37.514040Z",
"kind": "enrichment",
"module": "threatintel",
"reference": "https://urlhaus.abuse.ch/url/12345/",
"type": "indicator"
},
"first_seen": "2021-08-03T20:35:17.000Z",
"matched": {
"atomic": "elastic.local",
"field": "host.name",
"id": "_tdUD3sBcVT20cvWAkpd",
"index": "filebeat-7.14.0-2021.08.04-000001",
"type": "indicator_match_rule"
},
"provider": "provider1",
"type": "url",
"url": {
"domain": "elastic.local",
"extension": "php",
"full": "http://elastic.local/thing",
"original": "http://elastic.local/thing",
"path": "/thing",
"scheme": "http"
}
}
]
},
"user": {
"effective": {
"group": {
"id": "20"
},
"id": "501"
},
"group": {
"id": "20",
"name": "staff"
},
"id": "501",
"name": "elastic",
"saved": {
"group": {
"id": "20"
},
"id": "501"
}
}
}
}
}

344
x-pack/test/functional/es_archives/security_solution/legacy_cti_signals/mappings.json Normal file
View file

@ -0,0 +1,344 @@
{
"type": "index",
"value": {
"aliases": {
".alerts-security.alerts-default": {
"is_write_index": false
},
".siem-signals-default": {
"is_write_index": true
}
},
"index": ".siem-signals-default-000001",
"mappings": {
"_meta": {
"aliases_version": 1,
"version": 45
},
"dynamic": "false",
"properties": {
"@timestamp": {
"type": "date"
},
"threat": {
"properties": {
"framework": {
"ignore_above": 1024,
"type": "keyword"
},
"indicator": {
"properties": {
"as": {
"properties": {
"number": {
"type": "long"
},
"organization": {
"properties": {
"name": {
"fields": {
"text": {
"norms": false,
"type": "text"
}
},
"ignore_above": 1024,
"type": "keyword"
}
}
}
}
},
"confidence": {
"ignore_above": 1024,
"type": "keyword"
},
"dataset": {
"ignore_above": 1024,
"type": "keyword"
},
"description": {
"type": "wildcard"
},
"domain": {
"ignore_above": 1024,
"type": "keyword"
},
"email": {
"properties": {
"address": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"event": {
"properties": {
"action": {
"ignore_above": 1024,
"type": "keyword"
},
"category": {
"ignore_above": 1024,
"type": "keyword"
},
"code": {
"ignore_above": 1024,
"type": "keyword"
},
"created": {
"type": "date"
},
"dataset": {
"ignore_above": 1024,
"type": "keyword"
},
"duration": {
"type": "long"
},
"end": {
"type": "date"
},
"hash": {
"ignore_above": 1024,
"type": "keyword"
},
"id": {
"ignore_above": 1024,
"type": "keyword"
},
"ingested": {
"type": "date"
},
"kind": {
"ignore_above": 1024,
"type": "keyword"
},
"module": {
"ignore_above": 1024,
"type": "keyword"
},
"original": {
"doc_values": false,
"ignore_above": 1024,
"index": false,
"type": "keyword"
},
"outcome": {
"ignore_above": 1024,
"type": "keyword"
},
"provider": {
"ignore_above": 1024,
"type": "keyword"
},
"reason": {
"ignore_above": 1024,
"type": "keyword"
},
"reference": {
"ignore_above": 1024,
"type": "keyword"
},
"risk_score": {
"type": "float"
},
"risk_score_norm": {
"type": "float"
},
"sequence": {
"type": "long"
},
"severity": {
"type": "long"
},
"start": {
"type": "date"
},
"timezone": {
"ignore_above": 1024,
"type": "keyword"
},
"type": {
"ignore_above": 1024,
"type": "keyword"
},
"url": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"first_seen": {
"type": "date"
},
"geo": {
"properties": {
"city_name": {
"ignore_above": 1024,
"type": "keyword"
},
"continent_name": {
"ignore_above": 1024,
"type": "keyword"
},
"country_iso_code": {
"ignore_above": 1024,
"type": "keyword"
},
"country_name": {
"ignore_above": 1024,
"type": "keyword"
},
"location": {
"type": "geo_point"
},
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"region_iso_code": {
"ignore_above": 1024,
"type": "keyword"
},
"region_name": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"ip": {
"type": "ip"
},
"last_seen": {
"type": "date"
},
"marking": {
"properties": {
"tlp": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"matched": {
"properties": {
"atomic": {
"ignore_above": 1024,
"type": "keyword"
},
"field": {
"ignore_above": 1024,
"type": "keyword"
},
"type": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"module": {
"ignore_above": 1024,
"type": "keyword"
},
"port": {
"type": "long"
},
"provider": {
"ignore_above": 1024,
"type": "keyword"
},
"scanner_stats": {
"type": "long"
},
"sightings": {
"type": "long"
},
"type": {
"ignore_above": 1024,
"type": "keyword"
}
},
"type": "nested"
},
"tactic": {
"properties": {
"id": {
"ignore_above": 1024,
"type": "keyword"
},
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"reference": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"technique": {
"properties": {
"id": {
"ignore_above": 1024,
"type": "keyword"
},
"name": {
"fields": {
"text": {
"norms": false,
"type": "text"
}
},
"ignore_above": 1024,
"type": "keyword"
},
"reference": {
"ignore_above": 1024,
"type": "keyword"
},
"subtechnique": {
"properties": {
"id": {
"ignore_above": 1024,
"type": "keyword"
},
"name": {
"fields": {
"text": {
"norms": false,
"type": "text"
}
},
"ignore_above": 1024,
"type": "keyword"
},
"reference": {
"ignore_above": 1024,
"type": "keyword"
}
}
}
}
}
}
}
}
},
"settings": {
"index": {
"lifecycle": {
"name": ".siem-signals-default",
"rollover_alias": ".siem-signals-default"
},
"mapping": {
"total_fields": {
"limit": "10000"
}
},
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}