mirror of
https://github.com/elastic/kibana.git
synced 2025-04-24 17:59:23 -04:00
[Security Solution][CTI] Update legacy CTI signals to latest ECS threat fields (#107988)
* WIP: Adding integration test
* Replace threat.indicator mappings with threat.enrichments mappings
The nested threat.indicator mappings were experimental, and replaced by
threat.enrichmentsin ECS 1.10. While these fields are also experimental,
they fix the conflict between CTI data's normal threat.indicator
mappings.
* Add threat.enrichments mappings to our signals template mappings
event.* is no longer nested within here; it was determined that event
fields were not relevant to enrichment. All relevant ECS fieldsets
(file, pe, etc) are now nested under threat.enrichments.
* Update snapshot with newest threat.enrichments mappings
This test is a snapshot of the actual mappings applied by our templates. Looks good to me!
* Update ECS types to match latest
We now have two threat fields we care about for CTI, for legacy and
official ECS.
* Add a basic test for behavior of legacy enriched signals.
They're still queryable by threat.indicator, meaning that any existing
dashboards will still work.
* WIP: First pass at a data migration for CTI signals
* Defines reindex script to move things around
* Adds integration tests to make sure the migration and new mappings
work
* Need to test a few more things and verify corner cases
* Need to extract some helpers from tests
* Bump our template version to ensure devs roll over
Marshall bumped to 55, giving us 10 versions for 7.14.x updates.
However, devs would not otherwise roll over and get my mapping updates
without destroying their signals index and rebuilding (which is also not
the same thing, exactly), so this trades having one higher signals
version for a more streamlined dev workflow.
* More robust guard against data migration
We only attempt to migrate legacy enrichments if the document:
* is a signal from an indicator match rule
* has a `threat.indicator` field
* does not have a `threat.enrichments` field
* Minor reorder of operations to make logic clearer
* Add more assertions around our signals data migration
Tests a few more pieces of the resulting document, giving more
confidence that it's the correct transformation (and mappings).
This also modifies/anonymizes the data that was originally generated on
a work machine.
* Remove outdated note
This was for when these tests were driven via the UI; the API is more
responsive and now synchronization is currently needed here, beyond the
200 responses.
* Fix typo in comment
These fields are in ECS 1.11.
* Update snapshot test
We bumped the version previously, causing this test to become outdated.
* Update ECS typings in timelines plugin
These were copied from the security_solution plugin. I updated those,
but neglected to update these.
Until there's a better mechanism for deduplication here, I'm going to
kick the can and update both for now.
* Update enrichments logic to read/write from threat.enrichments
* indicator match rule logic
* we now simply copy from the specified indicator path, and place that
in `threat.enrichments.indicator`
* event enrichment API logic
* We were previously returning fields from `indicator.*`, we now
include the `indicator.*` suffix in order to be more consistent with
the sibling `matched.*` fields
* row renderer logic
* removal of dataset
* updates relevant to API changes above
* Fix logical error in generating links from indicator fields
We want to link the reference field, not a `first_seen` field.
* Always include the indicator prefix in first-party indicator fields
Prior to this change we would display e.g. `threatintel.indicator.foo`
for investigation enrichment fields. Now that the structure has changed
slightly and we return both `indicator.*` and `matched.*` fields for
existing enrichents, we want to display investigation enrichment
similarly.
* Update indicator match rule integration tests
Now that we've updated our enrichment logic, we need to update our
enrichment tests.
* Remove unused translation
* Update example row renderer data for enriched alerts
* Update parallel CTI constants to get our CTI row renderer working
We were not requesting the necessary fields for our row renderer, since
these constants (specifically CTI_ROW_RENDERER_FIELDS) now exist in both
security_solution and the timelines plugin. I had updated the former,
but only the latter is actually used.
* Update CTI enrichment UI tests
* Update prepackaged threat timeline template with new threat fields
Also bumps the timelineTemplateVersion.
* Update Indicator Match rule tests
These needed three things:
* Update to timeline template (see previous commit)
* Changing expectations from `threat.indicator` to `threat.enrichments`
* Update row renderer expectation to exclude dataset
* Update mock data with newest CTI enrichment fields
* Fix assertion on our threat details
These fields are prefixed with `indicator` now because:
1. This data pertains to the indicator, not the match per se
2. The actual field is prefixed with indicator (or, it at least
specifies an indicator in the case of a custom threat index (via
threat_indicator_path))
* Update test data and tests for our field parsing helpers
* Update more event-parsing tests
Ths one involved updating a mock in another package.
* Modify our helper function to support old filebeat indicators
When we query indicators for enrichment matches, the current expectation
is that we'll be querying 7.14 filebeat modules, which have an indicator
path of 'threatintel.indicator'. The only place that matters on the UI
is on the threat intel panel, where these indicators come back with such
a prefix.
This change has one behavior: it brings back the `provider` field on the
Alert summary tab for queried enrichments from filebeat modules.
* Update variable and method names to be more consistent with internal terminology
Indicators come from a CTI index. Enrichments are the application of
indicator data to other documents, and contain both indicator fields and
matched context.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
This commit is contained in:
Ryland Herrick
GitHub
parent
5f947c2531
commit
a9844db461
42 changed files with 3478 additions and 1048 deletions
|
|
@ -6,20 +6,37 @@
|
|||
*/
|
||||
|
||||
import { EventEcs } from '../event';
|
||||
import { UrlEcs } from '../url';
|
||||
|
||||
interface ThreatMatchEcs {
|
||||
atomic?: string[];
|
||||
field?: string[];
|
||||
id?: string[];
|
||||
index?: string[];
|
||||
type?: string[];
|
||||
}
|
||||
|
||||
export interface ThreatIndicatorEcs {
|
||||
export interface LegacyThreatIndicatorEcs {
|
||||
domain?: string[];
|
||||
matched?: ThreatMatchEcs;
|
||||
event?: EventEcs & { reference?: string[] };
|
||||
provider?: string[];
|
||||
type?: string[];
|
||||
}
|
||||
|
||||
export interface ThreatEcs {
|
||||
indicator: ThreatIndicatorEcs[];
|
||||
export interface ThreatIndicatorEcs {
|
||||
url?: UrlEcs;
|
||||
provider?: string[];
|
||||
reference?: string[];
|
||||
type?: string[];
|
||||
}
|
||||
|
||||
export interface ThreatEnrichmentEcs {
|
||||
indicator?: ThreatIndicatorEcs;
|
||||
matched?: ThreatMatchEcs;
|
||||
}
|
||||
|
||||
export interface ThreatEcs {
|
||||
indicator?: LegacyThreatIndicatorEcs[];
|
||||
enrichments?: ThreatEnrichmentEcs[];
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1096,18 +1096,17 @@ export const mockTimelineData: TimelineItem[] = [
|
|||
_id: 'BuBP4W0BOpWiDweSoYSg',
|
||||
timestamp: '2019-10-18T23:59:15.091Z',
|
||||
threat: {
|
||||
indicator: [
|
||||
enrichments: [
|
||||
{
|
||||
indicator: {
|
||||
provider: ['indicator_provider'],
|
||||
reference: ['https://example.com'],
|
||||
},
|
||||
matched: {
|
||||
atomic: ['192.168.1.1'],
|
||||
field: ['source.ip'],
|
||||
type: ['ip'],
|
||||
},
|
||||
event: {
|
||||
dataset: ['threatintel.example_dataset'],
|
||||
reference: ['https://example.com'],
|
||||
},
|
||||
provider: ['indicator_provider'],
|
||||
},
|
||||
],
|
||||
},
|
||||
|
|
|
|||
|
|
@ -8,34 +8,33 @@
|
|||
// import { CTI_ROW_RENDERER_FIELDS } from '../../../../../../common/cti/constants';
|
||||
|
||||
// TODO: share with security_solution/common/cti/constants.ts
|
||||
export const INDICATOR_DESTINATION_PATH = 'threat.indicator';
|
||||
export const ENRICHMENT_DESTINATION_PATH = 'threat.enrichments';
|
||||
|
||||
export const MATCHED_ATOMIC = 'matched.atomic';
|
||||
export const MATCHED_FIELD = 'matched.field';
|
||||
export const MATCHED_TYPE = 'matched.type';
|
||||
export const INDICATOR_MATCH_SUBFIELDS = [MATCHED_ATOMIC, MATCHED_FIELD, MATCHED_TYPE];
|
||||
|
||||
export const INDICATOR_MATCHED_ATOMIC = `${INDICATOR_DESTINATION_PATH}.${MATCHED_ATOMIC}`;
|
||||
export const INDICATOR_MATCHED_FIELD = `${INDICATOR_DESTINATION_PATH}.${MATCHED_FIELD}`;
|
||||
export const INDICATOR_MATCHED_TYPE = `${INDICATOR_DESTINATION_PATH}.${MATCHED_TYPE}`;
|
||||
export const INDICATOR_MATCHED_ATOMIC = `${ENRICHMENT_DESTINATION_PATH}.${MATCHED_ATOMIC}`;
|
||||
export const INDICATOR_MATCHED_FIELD = `${ENRICHMENT_DESTINATION_PATH}.${MATCHED_FIELD}`;
|
||||
export const INDICATOR_MATCHED_TYPE = `${ENRICHMENT_DESTINATION_PATH}.${MATCHED_TYPE}`;
|
||||
|
||||
export const EVENT_DATASET = 'event.dataset';
|
||||
export const EVENT_REFERENCE = 'event.reference';
|
||||
export const PROVIDER = 'provider';
|
||||
export const FIRSTSEEN = 'first_seen';
|
||||
|
||||
export const INDICATOR_DATASET = `${INDICATOR_DESTINATION_PATH}.${EVENT_DATASET}`;
|
||||
export const INDICATOR_EVENT_URL = `${INDICATOR_DESTINATION_PATH}.event.url`;
|
||||
export const INDICATOR_FIRSTSEEN = `${INDICATOR_DESTINATION_PATH}.${FIRSTSEEN}`;
|
||||
export const INDICATOR_LASTSEEN = `${INDICATOR_DESTINATION_PATH}.last_seen`;
|
||||
export const INDICATOR_PROVIDER = `${INDICATOR_DESTINATION_PATH}.${PROVIDER}`;
|
||||
export const INDICATOR_REFERENCE = `${INDICATOR_DESTINATION_PATH}.${EVENT_REFERENCE}`;
|
||||
export const FIRST_SEEN = 'indicator.first_seen';
|
||||
export const LAST_SEEN = 'indicator.last_seen';
|
||||
export const PROVIDER = 'indicator.provider';
|
||||
export const REFERENCE = 'indicator.reference';
|
||||
|
||||
export const INDICATOR_FIRSTSEEN = `${ENRICHMENT_DESTINATION_PATH}.${FIRST_SEEN}`;
|
||||
export const INDICATOR_LASTSEEN = `${ENRICHMENT_DESTINATION_PATH}.${LAST_SEEN}`;
|
||||
export const INDICATOR_PROVIDER = `${ENRICHMENT_DESTINATION_PATH}.${PROVIDER}`;
|
||||
export const INDICATOR_REFERENCE = `${ENRICHMENT_DESTINATION_PATH}.${REFERENCE}`;
|
||||
|
||||
export const CTI_ROW_RENDERER_FIELDS = [
|
||||
INDICATOR_MATCHED_ATOMIC,
|
||||
INDICATOR_MATCHED_FIELD,
|
||||
INDICATOR_MATCHED_TYPE,
|
||||
INDICATOR_DATASET,
|
||||
INDICATOR_REFERENCE,
|
||||
INDICATOR_PROVIDER,
|
||||
];
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ describe('#formatTimelineData', () => {
|
|||
'destination.ip',
|
||||
'source.ip',
|
||||
'source.geo.location',
|
||||
'threat.indicator.matched.field',
|
||||
'threat.enrichments.matched.field',
|
||||
],
|
||||
TIMELINE_EVENTS_FIELDS,
|
||||
eventHit
|
||||
|
|
@ -42,7 +42,7 @@ describe('#formatTimelineData', () => {
|
|||
value: ['beats-ci-immutable-ubuntu-1804-1605624279743236239'],
|
||||
},
|
||||
{
|
||||
field: 'threat.indicator.matched.field',
|
||||
field: 'threat.enrichments.matched.field',
|
||||
value: ['matched_field', 'other_matched_field', 'matched_field_2'],
|
||||
},
|
||||
{
|
||||
|
|
@ -95,10 +95,10 @@ describe('#formatTimelineData', () => {
|
|||
name: ['jenkins'],
|
||||
},
|
||||
threat: {
|
||||
indicator: [
|
||||
enrichments: [
|
||||
{
|
||||
event: {
|
||||
dataset: [],
|
||||
indicator: {
|
||||
provider: ['yourself'],
|
||||
reference: [],
|
||||
},
|
||||
matched: {
|
||||
|
|
@ -106,11 +106,10 @@ describe('#formatTimelineData', () => {
|
|||
field: ['matched_field', 'other_matched_field'],
|
||||
type: [],
|
||||
},
|
||||
provider: ['yourself'],
|
||||
},
|
||||
{
|
||||
event: {
|
||||
dataset: [],
|
||||
indicator: {
|
||||
provider: ['other_you'],
|
||||
reference: [],
|
||||
},
|
||||
matched: {
|
||||
|
|
@ -118,7 +117,6 @@ describe('#formatTimelineData', () => {
|
|||
field: ['matched_field_2'],
|
||||
type: [],
|
||||
},
|
||||
provider: ['other_you'],
|
||||
},
|
||||
],
|
||||
},
|
||||
|
|
@ -466,9 +464,9 @@ describe('#formatTimelineData', () => {
|
|||
});
|
||||
|
||||
it('builds intermediate objects at multiple levels', () => {
|
||||
expect(buildObjectForFieldPath('threat.indicator.matched.atomic', eventHit)).toEqual({
|
||||
expect(buildObjectForFieldPath('threat.enrichments.matched.atomic', eventHit)).toEqual({
|
||||
threat: {
|
||||
indicator: [
|
||||
enrichments: [
|
||||
{
|
||||
matched: {
|
||||
atomic: ['matched_atomic'],
|
||||
|
|
@ -485,9 +483,9 @@ describe('#formatTimelineData', () => {
|
|||
});
|
||||
|
||||
it('preserves multiple values for a single leaf', () => {
|
||||
expect(buildObjectForFieldPath('threat.indicator.matched.field', eventHit)).toEqual({
|
||||
expect(buildObjectForFieldPath('threat.enrichments.matched.field', eventHit)).toEqual({
|
||||
threat: {
|
||||
indicator: [
|
||||
enrichments: [
|
||||
{
|
||||
matched: {
|
||||
field: ['matched_field', 'other_matched_field'],
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue