* rename siem to security_solution
* rename siem to security solution inside of code
* rename translation keys
* fix snapshot
* replace siem for security solution in tutorial
* missing translation to be renamed
* fix types for api test integration
* updates runner file to match the new path
* change category for kibana settings
* miss renaming in advance settings
* fixes cypress tests
* fix api integration test
* fix new translation
* fix unit test
* update translation i18n
* update translation i18n II
Co-authored-by: Gloria Hornero <snootchie.boochies@gmail.com>
* introduce lists plugin for use by executor
* adds getListClient function on setup
* refactors searchAfterBulkCreate to integrate with the lists plugin so we only generate signals from events not in the list
* fixes type check issues
* fixes unit tests, adds field and other parameters for using lists in executor.
* cleaning up types and exports, updates to match new contracts with lists client from master
* prior to this commit the refactored while loop was doing more search after loops than it needed to and this fixes two bugs in the list filter function where we were returning the wrong count, and we were not accessing the right field on the event
* exception lists are optional
* use exceptions list format, this works with given sample query in scripts
* updates tests and fixes type issues
* updates README doc in detection engine with example for rule with list exception
* adds one test and removes commented out code
* fix sample rule json from 30s to 5m
* fix sample rule json from 30s to 5m
* remove unused import
* more cleanup
* e2e test for prepackaged rules was failing because lists was undefined in the siem plugin and was preventing the registration of the rule alert type. I removed this but once lists is ready for prime time we should consider adding the null check back
* can't reuse the same env var since the tests are setting the ELASTIC_XPACK_SIEM_LISTS_FEATURE env var to true without enabling the lists plugin
* fixes from pr review, still needs more TLC
* exports listspluginsetup type from top-level in lists plugin, fixes logic for empty exceptions list, updates types
* utilize type.is to remove as casting, also do null checks and throw an error when exceptionItem is malformed. This will change in the very near future once the new json format for exception lists is incorporated
* fix type issues after merging master into branch
* update mock
* remove bad null check for ml plugin before registering rule alert type in siem plugin
* prettier linting
* adds test for filter events with list
* pr comments
* adds logic for included vs excluded and updates tests
* update test cases for search after bulk create to default to included for exception lists
* filter out non-list exception items from the loop
## Summary
Adds backend e2e basic license type tests for the detection engine API. Previously we only had tests for full security space that was platinum based licensing. These tests now cover the basic license test cases. This covers test cases for machine learning to ensure you cannot create machine learning based detection rules under the basic license. Instead those tests will return the expected 403 forbidden.
Testing just the subset of the tests from this PR locally:
```sh
node scripts/functional_tests --config x-pack/test/detection_engine_api_integration/basic/config.ts
```
You do want to go to the jenkins tests on CI and ensure you can see some of the strings such as the newer forbidden messages showing up as passing.
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
Completes the migration of all Alerting Services plugins onto the Kibana Platform
It includes:
1. Actions plugin
2. Alerting plugin
3. Task Manager plugin
4. Triggers UI plugin
And touches the Uptime and Siem plugins as their use of the Task Manager relied on some of the legacy lifecycle to work (registering AlertTypes and Telemetry tasks after the Start stage has already began). The fix was simply to moves these registrations to the Setup stage.
* Move server code into NP folder
* NP config is not yet used
* Relative imports are somewhat broken
* Move common folder into NP
* Move cypress folder into NP
* Move scripts folder into NP
* Move misc. config into NP folder
A few of these were moved into the cypress folder as they're
cypress-specific.
I tried to update all the relative paths but some are likely broken. I'm
not going to know until other stuff is fixed, though.
* Move value for siem index pattern into common/constants
The other default values live in there, this is no different.
* Update paths following file move
If this was referencing the full project, it now references both paths
(legacy for UI, and NP for server).
* Fix typescript errors related to module resolution
These are mostly updating imports to the common/ folder on the UI side
(since things changed relative to those files).
* Replace Legacy Config with NP Config
* Updates plugin to use NP config
* defines new config previously coming from savedObjects config
* cleans up legacy types
Conflicts:
x-pack/plugins/siem/server/lib/detection_engine/routes/rules/export_rules_route.ts
x-pack/plugins/siem/server/lib/detection_engine/routes/rules/import_rules_route.ts
x-pack/plugins/siem/server/lib/detection_engine/rules/types.ts
x-pack/plugins/siem/server/plugin.ts
x-pack/plugins/siem/server/routes/index.ts
x-pack/plugins/siem/server/types.ts
* Remove local SIEM tsconfig
This was originally added to address an issue with tsserver, but that
issue is no longer relevant. The presence of this file confuses
typescript into thinking that siem is a separate TS project.
* Update kibana.json to declare our dependencies
These are not necessarily correct in terms of what's required/optional,
but this is what's declared in our types.
* Remove legacy plugin instantiation
* Removes legacy instantiation of server plugin, which is now handled by
NP
* Loosens legacy config spec so we no longer have to duplicate config
types
* Update tests with NP config
These were written against the old Hapi config function; now, we just
have a POJO.
* Update es_archiver helpers' paths
I'm not quite sure if these are working yet, but they're no longer
throwing errors.
* Ignore restricted path on script
This was cribbed from infra, who has made a similar change.
* Ignore restricted path on temporary savedObject mappings import
This will be changed subsequently when we switch to the NP form of
savedObject type registration.
* Add symlink to lockfile
* Fix paths on circular deps script
* Add separate config for Rule and Timeline saved objects
We had previously used the savedObjects' config, but those are not
currently exposed to us on New Platform. For now, we're going to split
this into two sets of values for the SOs we deal with
importing/exporting within the SIEM app, with the same defaults as
savedObjects.
* Fixing relative paths within cypress
These are strings that wouldn't be caught by typescript.
## Summary
* https://github.com/elastic/kibana/issues/60022
* Adds the feature flag for simple list values
* Adds the boolean filters of "and", "and not" to further filter based on simple values
* Adds unit tests and e2e tests for the values.
* Most tests can include the simple list values but some have to be skipped until we move those to more functions or just enable simple list values as a permanent feature.
* DOES NOT FILTER ON THE VALUES JUST YET (That will be a follow on PR)
## Testing:
To turn on/off the feature flag do this with an env variable (set this in your .bashrc/.zshrc):
```ts
export ELASTIC_XPACK_SIEM_LISTS_FEATURE=true
```
Expect to see this error in the console when the environment variable is set:
```ts
server log [11:41:16.245] [error][plugins][siem] You have activated the lists feature flag which is NOT currently supported for SIEM! You should turn this feature flag off immediately by un-setting the environment variable: ELASTIC_XPACK_SIEM_LISTS_FEATURE and restarting Kibana
```
Expect create and update to work when the environment variable is set and look like this:
```ts
./update_rule.sh ./rules/updates/update_list.json
{
"created_at": "2020-03-15T17:42:37.074Z",
"updated_at": "2020-03-15T17:54:22.427Z",
"created_by": "yo",
"description": "Query with a list",
"enabled": true,
"false_positives": [],
"from": "now-6m",
"id": "c602e3f6-713b-4f43-9bdd-b60fbfead1c5",
"immutable": false,
"interval": "5m",
"rule_id": "query-with-list",
"language": "kuery",
"output_index": ".siem-signals-hassanabad-frank-default",
"max_signals": 100,
"risk_score": 1,
"name": "Query with a list",
"query": "user.name: root or user.name: admin",
"references": [],
"severity": "high",
"updated_by": "yo",
"tags": [],
"to": "now",
"type": "query",
"threat": [],
"version": 6,
"lists": [
{
"field": "source.ip",
"boolean_operator": "and",
"values": [
{
"name": "127.0.0.1",
"type": "value"
}
]
},
{
"field": "host.name",
"boolean_operator": "and not",
"values": [
{
"name": "rock01",
"type": "value"
}
]
}
],
"status": "succeeded",
"status_date": "2020-03-15T17:42:40.718Z",
"last_success_at": "2020-03-15T17:42:40.718Z",
"last_success_message": "succeeded"
}
```
```ts
./post_rule.sh ./rules/queries/query_with_list.json
{
"created_at": "2020-03-15T17:42:37.074Z",
"updated_at": "2020-03-15T17:42:37.116Z",
"created_by": "yo",
"description": "Query with a list",
"enabled": true,
"false_positives": [],
"from": "now-6m",
"id": "c602e3f6-713b-4f43-9bdd-b60fbfead1c5",
"immutable": false,
"interval": "5m",
"rule_id": "query-with-list",
"language": "kuery",
"output_index": ".siem-signals-hassanabad-frank-default",
"max_signals": 100,
"risk_score": 1,
"name": "Query with a list",
"query": "user.name: root or user.name: admin",
"references": [],
"severity": "high",
"updated_by": "yo",
"tags": [],
"to": "now",
"type": "query",
"threat": [],
"version": 1,
"lists": [
{
"field": "source.ip",
"boolean_operator": "and",
"values": [
{
"name": "127.0.0.1",
"type": "value"
}
]
},
{
"field": "host.name",
"boolean_operator": "and not",
"values": [
{
"name": "rock01",
"type": "value"
},
{
"name": "mothra",
"type": "value"
}
]
}
]
}
```
```ts
./patch_rule.sh ./rules/patches/update_list.json
{
"created_at": "2020-03-15T18:02:52.434Z",
"updated_at": "2020-03-15T18:02:57.675Z",
"created_by": "yo",
"description": "Query with a list",
"enabled": true,
"false_positives": [],
"from": "now-6m",
"id": "40b7c2fb-83b4-4820-bf7c-056f3a631126",
"immutable": false,
"interval": "5m",
"rule_id": "query-with-list",
"language": "kuery",
"output_index": ".siem-signals-hassanabad-frank-default",
"max_signals": 100,
"risk_score": 1,
"name": "Query with a list",
"query": "user.name: root or user.name: admin",
"references": [],
"severity": "high",
"updated_by": "yo",
"tags": [],
"to": "now",
"type": "query",
"threat": [],
"version": 1,
"lists": [
{
"field": "source.ip",
"boolean_operator": "and",
"values": [
{
"name": "127.0.0.1",
"type": "value"
}
]
},
{
"field": "host.name",
"boolean_operator": "and not",
"values": [
{
"name": "rock01",
"type": "value"
},
{
"name": "mothra",
"type": "value"
}
]
}
],
"status": "succeeded",
"status_date": "2020-03-15T18:02:56.426Z",
"last_success_at": "2020-03-15T18:02:56.426Z",
"last_success_message": "succeeded"
}
```
```ts
./get_rule_by_rule_id.sh query-with-list
{
"created_at": "2020-03-15T18:10:07.657Z",
"updated_at": "2020-03-15T18:10:08.479Z",
"created_by": "yo",
"description": "Query with a list",
"enabled": true,
"false_positives": [],
"from": "now-6m",
"id": "9854162b-003c-47be-af59-8c3c9545aafa",
"immutable": false,
"interval": "5m",
"rule_id": "query-with-list",
"language": "kuery",
"output_index": ".siem-signals-hassanabad-frank-default",
"max_signals": 100,
"risk_score": 1,
"name": "Query with a list",
"query": "user.name: root or user.name: admin",
"references": [],
"severity": "high",
"updated_by": "yo",
"tags": [],
"to": "now",
"type": "query",
"threat": [],
"version": 1,
"lists": [
{
"field": "source.ip",
"boolean_operator": "and",
"values": [
{
"name": "127.0.0.1",
"type": "value"
}
]
},
{
"field": "host.name",
"boolean_operator": "and not",
"values": [
{
"name": "rock01",
"type": "value"
},
{
"name": "mothra",
"type": "value"
}
]
}
],
"status": "going to run",
"status_date": "2020-03-15T18:10:10.738Z"
}
```
Expect these errors when the environment variable is not set:
```ts
./post_rule.sh ./rules/queries/query_with_list.json
{
"statusCode": 400,
"error": "Bad Request",
"message": "[request body]: child \"lists\" fails because [\"lists\" is not allowed]"
}
```
```ts
./update_rule.sh ./rules/queries/query_with_list.json
{
"statusCode": 400,
"error": "Bad Request",
"message": "[request body]: child \"lists\" fails because [\"lists\" is not allowed]"
}
```
```ts
./patch_rule.sh ./rules/patches/update_list.json
{
"statusCode": 400,
"error": "Bad Request",
"message": "[request body]: child \"lists\" fails because [\"lists\" is not allowed]"
}
```
Expect that this is _backwards_ compatible with the feature flag but not necessarily _forwards_ compatible. This means:
* You can have older data that never had lists and it will show up as an empty list when you query it. (backwards compatible)
* You _might_ have lists and remove the env. variable and get back items as if the list was not there for (forwards compatible)
* You can export without lists, flip on the env flag and import with newer lists feature (backwards compatible)
* You can export lists and it will _not_ work with an older system (not forwards compatible)
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
