* Separated out service layer for trusted apps.
* Improved the type structure a bit to avoid using explicit string literals and to add possibility to return OS specific parts of trusted app object in type safe manner.
* Added support for mapping of trusted app to exception item and back.
* Changed schema to support signer in the API.
* Renamed utils to mapping.
* Exported some types in lists plugin and used them in trusted apps.
* Added tests for mapping.
* Added tests for service.
* Switched deletion to use exceptions for not found case.
* Added resetting of the mocks in service layer tests.
* Added handlers tests.
* Refactored mapping tests to be more granular based on the case.
* Restored lowercasing of hash.
* Added schema tests for signer field.
* Removed the grouped tests (they were split into tests for separate concerns).
* Corrected the tests.
* Lowercased the hashes in the service test.
* Moved the lowercasing to the right location.
* Fixed the tests.
* Added test for lowercasing hash value.
* Introduced OperatingSystem enum instead of current types.
* Removed os list constant in favour of separate lists in places that use it (each place has own needs to the ordering).
* Fixed the missed OperatingSystem enum usage.
* bump version to 4.1.1-rc
* fix code to run kbn bootstrap
* fix errors
* DO NOT MERGE. mute errors and ping teams to fix them
* Address EuiSelectableProps configuration in discover sidebar
* use explicit type for EuiSelectable
* update to ts v4.1.2
* fix ts error in EuiSelectable
* update docs
* update prettier with ts version support
* Revert "update prettier with ts version support"
This reverts commit 3de48db3ec.
* address another new problem
Co-authored-by: Chandler Prall <chandler.prall@gmail.com>
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* ExceptionIdentifiers -> ExceptionListIdentifiers
* Pass refreshRule through to alert context menu
* Fix type errors and rename refreshRule to onRuleChange for consistency
* Finish adding .lower to exceptionable fields
* Add back migrations
* .lower -> .caseless
* Add separate field for os type
* updates
* Type updates
* Switch over to osTypes
* get rid of _tags
* Add tests for schema validation
* Remove remaining references to _tags
* Another round of test fixes
* DefaultArray tests
* More test fixes
* Fix remaining test failures
* types / tests
* more test updates
* lowercase os values
* Address feedback + fix test failure
* tests
* Fix integration test
* process.executable.path -> process.executable.caseless
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This is a bug that was introduced by moi in 76537. This previous PR added a sort_field and sort_order to the call for fetching exception lists' items so that the exception item order in the viewer wouldn't jump around any time there was an update. I noticed however that when a rule had both endpoint and detection lists associated with it, when trying to fetch items from both types of lists, the following error shows:
```
"Unable to sort multiple types by field created_at, not a root property"
```
## Summary
Mistmatch caught between the io-ts type and the corresponding typescript enum. Currently, io-ts does not have support for enums - as a workaround I had made a matching typescript enum type. Tests were added to try to ensure the two stayed in sync, but didn't do a straight up comparison of the two.
Updated the tests to now error if the keys do not match.
## Summary
**Components affected:** ExceptionsViewer
**Current behavior:**
- when a user edits an exception item, the order of the exception items in the viewer changes. This creates confusion and looks like updates weren't applied (even though they were, just item order changed)
**New behavior:**
- when a user edits an exception item, the order of the exception items in the viewer don't change. Sort order is now based on `created_at`
* bump ts to v4
* MOAR RAM
* fix type errors for OSS
* first pass on x-pack errors
* second pass on x-pack type errors
* 3rd pass on x-pack type-errors
* mute errors if complex cases
* don't delete if spread suffices
* mute other complex cases
* make User fields optional
* fix optional types
* fix tests
* fix typings for time_range
* fix type errors in x-pack/tests
* rebuild kbn-pm
* remove leftovers from master update
* fix alert tests
* [Telemetry Checker] TS4 Fixes
* bump to 4.0.1-rc
* fix new errors in master
* bump typescript-eslint to version supporting TS v4 syntax
* fix merge commit errors
* update to the stable TS version 4.0.2
* bump ts-eslint to version supporting ts v4
* fix typo
* fix type errors after merge
* update ts in another new package.json
* TEMP: remove me
* Revert "TEMP: remove me"
This reverts commit dc0fc3bae6.
* [Telemetry] Update snapshot for new TS4 SyntaxKind
* bump prettier to support TS v4 syntax
* fix prettier rules
* last style change
* fix new type errors
Co-authored-by: Alejandro Fernández Haro <alejandro.haro@elastic.co>
* Trusted Apps initial setup for route registration
* Added types for TrustedApp entries
* trusted apps list API returns results
* use methods and const from latest PR merge to lists
* a quick generator for trusted apps entries
* support cli options for trusted app data loader
* Add mocked `createTrustedAppsList()` method to `ExceptionListClientMock`
* tests fro trusted apps route handlers
* tests for trusted apps schema
* Correct name of mock method
* Fix service to ensure return value of `getExceptionList` service throws if service not available
* Fix types
* Refactor TrustedApp type + code review feedback
* Simplify our kibana mocks
* Simpler mock factory that returns an object instead of a thunk
* We can use mockReturnValue instead of mockImplementation to
accomplish the same
* Allows us to replace createStartServices mock
* Uses unknown instead of any for mocks
* Clean up our manual use of kibana mocks in tests
* Since our useKibana mock returns a consistent mock, we can modify its
return value instead of re-mocking the entire thing
* Removes unnecessary uses of clearing/resetting mocks
* If your mocks are configured at the beginning of each test this is
usually unnecessary.
* I left one case of clearAllMocks in all_cases/index.test since it
defined several mock functions that were persistent across tests, and
it was easier than moving their definitions to a beforeEach
* Removes some unnecessary overrides that seemed due to storage
previously not being mocked
* Rename some old occurrences of SIEM
* Cross-reference similar hooks via JSDoc
There's a good chance that the consumer might want the OTHER hook, so
let's make that discoverable.
* Adds jest tests for our useListsConfig hook
* adds mocks for the hooks upon which it depends
* Add a mock for our useListsConfig hook
Leverages this mock factory in our manual mock for this hook.
* Remove unneeded eslint exception
* Move kibana_react mocks into their own .mock file
We're trying to consolidate mocks to this pattern so that they're easier
to find and reuse.
* Remove intermediate mock factory
This was only being consumed by our general createStartServicesMock.
* Replace security_solution's alias for a core mock
This is just noise/tech debt, we should use the core mock directly when
we can.
* Remove unnecessary wrapper around core mocks
Instead let's just reference the core mocks themselves.
* Remove outdated references from upstream
* More accurate mock
Throw an error of the same type if an unexpected key is used.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Fixes a bug where the list privileges was returning the `.list` privileges twice instead of returning it once and returning the `.items` privileges second with the call. No UI has to change as the way it was written was dynamic to grab the first key found.
This also adds the functional tests to `x-pack/scripts/functional_tests.js` which was not there originally so the end to tend tests should actually run on the CI machine where it was not running on CI before.
Adds the functional tests to the code owners file as well.
Ensure that you go to the test results page from the Jenkins build:
<img width="901" alt="Screen Shot 2020-08-18 at 1 13 18 AM" src="https://user-images.githubusercontent.com/1151048/90482180-13f7c800-e0f0-11ea-92f2-b30a8fffe84e.png">
And ensure you see the tests under:
```
X-Pack Lists Integration Tests
```
Then click through it and ensure they are shown as running and passing
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* Route options timeout -> timeout.payload
* timeout.idleSocket can now be specified per route
* Removing nested ternary
* Fixing integration tests
* Trying to actually fix the integration tests. Existing tests are hitting
idle socket timeout, not the payload timeout
* Fixing payload post timeout integration test
* Fixing PUT and DELETE payload sending too long tests
* Fixing type-script errors
* GET routes can't specify the payload timeout, they can't accept payloads
* Removing some redundancy in the tests
* Adding 'Transfer-Encoding: chunked' to the POST test
* Fixing POST/GET/PUT quick tests
* Adding idleSocket timeout test
* Removing unnecessary `isSafeMethod` call
* Updating documentation
* Removing PUT/DELETE integration tests
* Working around the HapiJS bug
* Deleting unused type import
* The socket can be undefined...
This occurs when using @hapi/shot directly or indirectly via
Server.inject. In these scenarios, there isn't a socket. This can also
occur when a "fake request" is used by the hacky background jobs:
Reporting and Alerting...
* Update src/core/server/http/http_server.ts
Co-authored-by: Josh Dover <me@joshdover.com>
* Adding payload timeout functional tests
* Adding idle socket timeout functional tests
* Adding better comments, using ?? instead of ||
* Fixing the plugin fixture TS
* Fixing some typescript errors
* Fixing plugin fixture tsconfig.json
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Josh Dover <me@joshdover.com>
* Add separate io-ts types for endpoint exception entries
* Fix text typos
* Fix test
* address review comments
* Add extra entry validation when adding items to endpoint_list
* fix test
* fix tests again
* really actually fix the tests
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This is the basics of end to end tests, so there could be a lot more, but this ties to cover the basics of the tests.
Test with:
```ts
node scripts/functional_tests --config x-pack/test/lists_api_integration/security_and_spaces/config.ts
```
Adds these tests for the route counter parts:
* create_exception_list_items.ts
* create_exception_lists.ts
* delete_exception_list_items.ts
* delete_exception_lists.ts
* find_exception_list_items.ts
* find_exception_lists.ts
* read_exception_list_items.ts
* read_exception_lists.ts
* update_exception_list_items.ts
* update_exception_lists.ts
Fixes a few minor strings, other tests, but no large bugs found with these tests
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
## Summary
Adds initial set of end to end tests for lists
You can run all of these with the command from kibana root:
```ts
node scripts/functional_tests --config x-pack/test/lists_api_integration/security_and_spaces/config.ts
```
Fixes a few minor bugs found such as...
* Validation for importing lists was not checking if the indexes were created first
* Some wording for the error messages had duplicate words within them
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
Using the `formatErrors` util would result in duplicate error messages sometimes. Was noticing this in particular when using union types, where the type validation would check every item in a union and report an error for each one. This resulted in large, repeating errors.
Used `uniq` to filter out duplicates. Updated unit tests.
## Summary
Addresses feedback from https://github.com/elastic/kibana/pull/72748
- Updates `plugins/lists` tests text from `should not validate` to `should FAIL validation` after feedback that previous text is a bit confusing and can be interpreted to mean that validation is not conducted
- Remove unnecessary spreads from one of my late night PRs
- Removes `siem_common_deps` in favor of `shared_imports` in `plugins/lists`
- Updates `build_exceptions_query.test.ts` to use existing mocks
## Summary
This PR focuses on addressing issues around the pagination and functionality of rules with numerous (2+) exception lists.
- Updated the `use_exception_list.ts` hook to make use of the new multi list find API
- Updated the viewer to make use of the new multi list find API
- Previously was doing a lot of the filtering and paging manually (and badly) in the UI, now the _find takes care of all that
- Added logic for showing `No results` text if user filter/search returns no items
- Previously would show the `This rule has not exceptions` text
## Summary
This PR addresses the following:
- Adds `list_id` to `rule.exceptions_list` - this is needed in a number of features
- Updated `getExceptions` in `x-pack/plugins/security_solution/server/lib/detection_engine/signals/utils.ts` to use the latest exception item find endpoint that accepts an array of lists (previously was looping through lists and conducting a `find` for each)
- Updated prepackaged rule that makes reference to global endpoint list to include `list_id`
- Updates `formatAboutStepData` in `x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/create/helpers.ts` to include exception list `list_id`
* Validate exception list size when adding new items
* Update comment
* Extract list size validation and apply to endpoint route also
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
By default the upload time limit for payloads is 10 seconds. This is really too short and we were getting internal QA bug reports that uploads are timing out on large value list importing. This PR adds the plumbing and unit tests to make the timeout configurable for routes.
* Adds a single timeout option for routes and then normalizes that through Hapi for the socket, payload, and server timeouts.
* Adds unit tests which test the various options
* Adds integration tests which test the various options
* Adds some NOTES about where there are odd behaviors/bugs within Hapi around validations and the timeouts
* Adds a configurable 5 minute timeout to the large value lists route
**Manual testing of the feature**
You can manually test this by adding a configurable option to your chrome network throttle like so below where you throttle upload by some configurable amount. I chose to use 300 kbs/s upload
<img width="556" alt="Screen Shot 2020-07-23 at 11 26 01 AM" src="https://user-images.githubusercontent.com/1151048/88318015-5ab3f700-ccd7-11ea-9d9b-7e3649ec65de.png">
And then run an import of large value lists using a large enough file that it will exceed 5 minutes:
After 5 minutes you should see this message within your server side messages if you have configured your kibana.dev.yml to allow for these messages:
```ts
server respons [10:52:31.377] [access:lists-all] POST /api/lists/items/_import?type=keyword 408 318292ms - 9.0B
```
Note that it should show you that it is trying to return a `408` after `318292ms` the timeout period. Sometimes you will get the 408 in the browser and sometimes the browser actually will not respect the 408 and continue staying in a pending state forever. This seems to be browser side issue and not a client/user land issue. If you get the browser message it will be this error toaster
### Checklist
- [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [x] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
### For maintainers
- [x] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)
Add validation to reject when value list and other exception type are entries in the same exception item. Also adds tests for this situation on the schema validation
## Summary
This PR is somewhat of an intermediary step. Comments on exception list items are denormalized. We initially decided that we would not add `uuid` to comments, but found that it is in fact necessary. This is intermediary in the sense that what we ideally want to have is a dedicated `comments` CRUD route.
Also just note that I added a callout for when a version conflict occurs (ie: exception item was updated by someone else while a user is editing the same item).
With this PR users are able to:
- Create comments when creating exception list items
- Add new comments on exception item update
Users will currently be blocked from:
- Deleting comments
- Updating comments
- Updating exception item if version conflict is found
* adapt retryCallCluster for new ES client
* review comments
* retry on 408 ResponseError
* remove legacy retry functions
* use Migrator Es client in SO migration
* update migration tests
* improve ES typings and mocks
* migrate decorate ES errors
* add repository es client
* use new es client in so repository
* update repository tests
* fix migrator integration tests
* declare _seq_no & _primary_term on get response. _source expect to be a string
* make _sourceIncludes and refresh compatible with the client
* add test for repository_es_client
* move ApiResponse to es client mocks
* TEMP: handle wait_for as true for deleteByNamespace
* add tests for migration_es_client
* TEMP: skip test for deleteByNamespace refresh
* pass ignore as transport option in mget
* log both es client and response errors
* fix update method test failures
* update deleteByNamespace refresh settings
es doesn't support 'refresh: wait_for' for `updateByQuery` endpoint
* update repository tests. we do not allow customising wait_for
* do not delegate retry logic to es client
* fix type errors after master merged
* fix repository tests
* fix security solutions code
SO doesn't throw Error with status code anymore. Always use SO error helpers
* switch error conditions to use SO error helpers
* cleanup
* address comments about mocks
* use isResponseError helper
* address comments
* fix type errors
Co-authored-by: pgayvallet <pierre.gayvallet@elastic.co>
* Refresh rule details when exception list modal modifies the rule
This addresses a bug where, when opening the exceptions modal for the first
time and creating exceptions, the details page does not reflect these
created exceptions until a full refresh.
This is due to the hook performing the refresh being dependent on the
rule's exceptions_list attribute, which is not populated until after
opening the modal. Because the UI is not informed of the rule update, it
did not know to refresh the rule.
This adds the machinery necessary to make the above work. It:
* adds a new hook for fetching/refreshing a rule
* Adds an onRuleChange callback to both the ExceptionsViewer and the
mutating AddExceptionModal
* passes the refresh function in as the onRuleChange callback
There's currently a gross intermediate state here where the loading screen is
displayed while the rule refreshes in the background; I'll be fixing
that shortly.
* Do not show loading/blank state while refreshing rule
On Rule Details, when the Add Exceptions modal creates the rule's
exception list, we refresh quietly in the background by setting our rule
from null -> ruleA -> ruleB instead of null -> ruleA -> null -> ruleB.
This also simplifies the loading logic in a few places now that we're
using our new rule: we mainly care whether or not our rule is populated.
* Display toast error if rule fetch fails
This should now have feature parity with useRule, while additionally
providing a function to refresh the rule.
* Refactor tests to leverage existing helpers
* Add return type to our callback function
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Limits the lists to 9 megs upload size so we don't blow up smaller Kibana installs. Users can change/override this using the switch of `xpack.lists.maxImportPayloadBytes` like so:
```
xpack.lists.maxImportPayloadBytes: 40000000
```
That will increase the amount of bytes that can pushed through REST endpoints from 9 megs to something like 40 megs if the end users want to increase the size of their lists and have enough memory in Kibana.
Metrics and suggestions from testing looks like:
```ts
Kibana with 1 gig of memory can upload ~10 megs of a list before possible out of memory issue
Kibana with 2 gig of memory can upload ~20 megs of a list before possible out of memory issue
```
Things can vary depending on the speed of the uploads of the lists where faster connections to Kibana but slower connections from Kibana to Elastic Search can influence the numbers.
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* wip - comment and sample json for exceptions
* promise.all for OR-ing exception items and quick-start script
* logging, added/updated json sample scripts, fixed missing await on filter with lists
* WIP
* bug fix where two lists when 'anded' together were not filtering down result set
* undo changes from testing
* fix changes to example json and fixes missed conflict with master
* update log message and fix type errors
* change log statement and add unit test for when exception items without a value list are passed in to the filter function
* fix failing test
* update expect on one test and adds a new test to ensure anding of value lists when appearing in different exception items
* update test after rebasing with master
* properly ands exception item entries together with proper test cases
* fix test (log statement tests - need to come up with a better way to cover these)
* cleans up json examples
* rename test and use 'every' in lieu of 'some' when determining if the filter logic should execute
## Summary
This PR updates the exception list entries schemas.
- **Prior:** `entries` could be `undefined` or empty array on `ExceptionListItemSchema`
- **Now:** `entries` is a required field that cannot be empty - there's really no use for an item without `entries`
- **Prior:** `field` and `value` could be empty string in `EntryMatch`
- **Now:** `field` and `value` can no longer be empty strings
- **Prior:** `field` could be empty string and `value` could be empty array in `EntryMatchAny`
- **Now:** `field` and `value` can no longer be empty string and array respectively
- **Prior:** `field` and `list.id` could be empty string in `EntryList`
- **Now:** `field` and `list.id` can no longer be empty strings
- **Prior:** `field` could be empty string in `EntryExists`
- **Now:** `field` can no longer be empty string
- **Prior:** `field` could be empty string in `EntryNested`
- **Now:** `field` can no longer be empty string
- **Prior:** `entries` could be empty array in `EntryNested`
- **Now:** `entries` can no longer be empty array
### Summary
The intent is to get the data structures in similar to rules so that we can have eventually immutable and versioned lists in later releases without too much hassle of upgrading the list and list item data structures.
* Adds version and immutability data structures to the exception lists and the value lists.
* Adds an optional version number to the update route of each so that you can modify the number either direction or you can omit it and it works like the detection rules where it will auto-increment the number.
* Does _not_ add a version and immutability to the exception list items and value list items.
* Does _not_ update the version number when you add a new exception list item or value list item.
**Examples:**
❯ ./post_list.sh
```json
{
"_version": "WzAsMV0=",
"id": "ip_list",
"created_at": "2020-07-21T20:31:11.679Z",
"created_by": "yo",
"description": "This list describes bad internet ip",
"immutable": false,
"name": "Simple list with an ip",
"tie_breaker_id": "d6bd7552-84d1-4f95-88c4-cc504517b4e5",
"type": "ip",
"updated_at": "2020-07-21T20:31:11.679Z",
"updated_by": "yo",
"version": 1
}
```
❯ ./post_exception_list.sh
```json
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "WzMzOTgsMV0=",
"created_at": "2020-07-21T20:31:35.933Z",
"created_by": "yo",
"description": "This is a sample endpoint type exception",
"id": "2c24b100-cb91-11ea-a872-adfddf68361e",
"immutable": false,
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "c11c4d53-d0be-4904-870e-d33ec7ca387f",
"type": "detection",
"updated_at": "2020-07-21T20:31:35.952Z",
"updated_by": "yo",
"version": 1
}
```
```json
❯ ./update_list.sh
{
"_version": "WzEsMV0=",
"created_at": "2020-07-21T20:31:11.679Z",
"created_by": "yo",
"description": "Some other description here for you",
"id": "ip_list",
"immutable": false,
"name": "Changed the name here to something else",
"tie_breaker_id": "d6bd7552-84d1-4f95-88c4-cc504517b4e5",
"type": "ip",
"updated_at": "2020-07-21T20:31:47.089Z",
"updated_by": "yo",
"version": 2
}
```
```json
❯ ./update_exception_list.sh
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "WzMzOTksMV0=",
"created_at": "2020-07-21T20:31:35.933Z",
"created_by": "yo",
"description": "Different description",
"id": "2c24b100-cb91-11ea-a872-adfddf68361e",
"immutable": false,
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "c11c4d53-d0be-4904-870e-d33ec7ca387f",
"type": "endpoint",
"updated_at": "2020-07-21T20:31:56.628Z",
"updated_by": "yo",
"version": 2
}
```
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* Add loading spinners to Value Lists modal
While export or a delete is pending, we display a loading spinner
instead of the button that was clicked.
Since state is controlled in the parent, we must pass this additional
state in the same way; the table component simply reacts to this state.
* Fix bug with useAsync and multiple calls
Multiple calls to start() would not previously reset the hook's state,
where useEffect on the hook's state would fire improperly as subsequent
calls would not travel the same undefined -> result path.
* Fix style of loading spinner
This fits the size of the button it's replacing, so no shifting occurs
when replacing elements.
* Better styling of spinner
Keep it roughly the same size as the icons themselves, and fill the
space with margin.
* Fix circular dependency in value lists modal
Moves our shared types into a separate module to prevent a circular
dependency.
## Summary
* Adds list permissions as a feature control to SIEM.
* Separates the controls between two, one of which is `access:lists-all` and the other is `access:lists-read`
* Grants SIEM the ability to utilize both depending on which feature mode the space is in.
## Summary
* Adds conflict versioning by exposing the "_version" from the saved object system. It renames "version" to "_version" so that we can use regular "version" later for versioning things for pre-packaged lists abilities.
* Utilizes `t.OutputOf` in the requests and the data types to give us more correctly types
* Removes the `Identity` utility as that is adding confusion and can confuse vs code rather than improves things
* Removes extra types that were causing confusion which was an idiom from io-ts
* Changes the wording of `Partial` by removing that and instead focuses the request types on either client side or server side at this point.
NOTE: The UI can migrate to holding onto the `_version` and then push it back down when it wants to migrate to using the conflict resolution. If the UI does not push it down, then a value of undefined will be used which is indicating that no conflict errors are wanted.
Output example of posting an exception list:
❯ ./post_exception_list.sh
```ts
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "Wzk4NiwxXQ==",
"created_at": "2020-07-17T18:59:22.872Z",
"created_by": "yo",
"description": "This is a sample endpoint type exception",
"id": "a08795b0-c85f-11ea-b1a6-c155df988a92",
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c",
"type": "detection",
"updated_at": "2020-07-17T18:59:22.891Z",
"updated_by": "yo"
}
```
Output example of posting an exception list item
❯ ./post_exception_list_item.sh
```ts
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "Wzk4NywxXQ==",
"comments": [],
"created_at": "2020-07-17T18:59:30.286Z",
"created_by": "yo",
"description": "This is a sample endpoint type exception",
"entries": [
{
"field": "actingProcess.file.signer",
"operator": "excluded",
"type": "exists"
},
{
"field": "host.name",
"operator": "included",
"type": "match_any",
"value": [
"some host",
"another host"
]
}
],
"id": "a4f2b800-c85f-11ea-b1a6-c155df988a92",
"item_id": "simple_list_item",
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "1dc456bc-7aa9-44b4-bca3-131689cf729f",
"type": "simple",
"updated_at": "2020-07-17T18:59:30.304Z",
"updated_by": "yo"
}
```
Output example of when you get an exception list:
❯ ./get_exception_list.sh simple_list
```ts
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "WzEwNzcsMV0=",
"created_at": "2020-07-17T18:59:22.872Z",
"created_by": "yo",
"description": "Different description",
"id": "a08795b0-c85f-11ea-b1a6-c155df988a92",
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c",
"type": "endpoint",
"updated_at": "2020-07-17T20:01:24.958Z",
"updated_by": "yo"
}
```
Example of the error you get if you do an update of an exception list and someone else has changed it:
```ts
{
"message": "[exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1]: [version_conflict_engine_exception] [exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1], with { index_uuid=\"a2mgXBO6Tl2ULDq-MTs1Tw\" & shard=\"0\" & index=\".kibana-hassanabad_1\" }",
"status_code": 409
}
```
Lists are the same way and flavor, they encode the _version the same way that saved objects do. To see those work you run these scripts:
```ts
./post_list.sh
./post_list_item.sh
./find_list.sh
./find_list_item.sh
```
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* Overview Alerts Histogram stacking defaults to signal.rule.name
Since this is now the default for all AlertsHistograms, I've moved this
default upstream into the histogram itself.
* Replace magic strings with our constant ENDPOINT_LIST_ID
Also replaced a few unintentional uses of this string with the
non-reserved 'endpoint_list_id'.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
* Adds the createEndpointListSchema to the create_endpoint_list_route for API boundary checks
* Adds unit tests to the requests
* Fixes a few bugs found in the find endpoints for types
* Fixes or deletes the skipped tests
* Updated TODO blocks and removed ones that are obsolete
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* Adds specific endpoint_list REST API and API for abilities to autocreate the endpoint_list if it gets deleted
* Added the check against prepackaged list
* Updated to use LIST names
* Removed the namespace where it does not belong
* Updates per code review an extra space that was added
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Add Frontend components for Value Lists Management Modal
Imports and uses the hooks provided by the lists plugin. Tests coming
next.
* Update value list components to use newest Lists API
* uses useEffect on a task's state instead of promise chaining
* handles the fact that API calls can be rejected with strings
* uses exportList function instead of hook
* Close modal on outside click
* Add hook for using a cursor with paged API calls.
For e.g. findLists, we can send along a cursor to optimize our query. On
the backend, this cursor is used as part of a search_after query.
* Better implementation of useCursor
* Does not require args for setCursor as they're already passed to the
hook
* Finds nearest cursor for the same page size
Eventually this logic will also include sortField as part of the
hash/lookup, but we do not currently use that on the frontend.
* Fixes useCursor hook functionality
We were previously storing the cursor on the _current_ page, when it's
only truly valid for the _next_ page (and beyond).
This was causing a few issues, but now that it's fixed everything works
great.
* Add cursor to lists query
This allows us to search_after a previous page's search, if available.
* Do not validate response of export
This is just a blob, so we have nothing to validate.
* Fix double callback post-import
After uploading a list, the modal was being shown twice. Declaring the
constituent state dependencies separately fixed the issue.
* Update ValueListsForm to manually abort import request
These hooks no longer care about/expose an abort function. In this one
case where we need that functionality, we can do it ourselves relatively
simply.
* Default modal table to five rows
* Update translation keys following plugin rename
* Try to fit table contents on a single row
Dates were wrapping (and raw), and so were wrapped in a FormattedDate
component. However, since this component didn't wrap, we needed to
shrink/truncate the uploaded_by field as well as allow the fileName to
truncate.
* Add helper function to prevent tests from logging errors
https://github.com/enzymejs/enzyme/issues/2073 seems to be an ongoing
issue, and causes components with useEffect to update after the test is
completed.
waitForUpdates ensures that updates have completed within an act()
before continuing on.
* Add jest tests for our form, table, and modal components
* Fix translation conflict
* Add more waitForUpdates to new overview page tests
Each of these logs a console.error without them.
* Fix bad merge resolution
That resulted in duplicate exports.
* Make cursor an optional parameter to findLists
This param is an optimization and not required for basic functionality.
* Tweaking Table column sizes
Makes actions column smaller, leaving more room for everything else.
* Fix bug where onSuccess is called upon pagination change
Because fetchLists changes when pagination does, and handleUploadSuccess
changes with fetchLists, our useEffect in Form was being fired on every
pagination change due to its onSuccess changing.
The solution in this instance is to remove fetchLists from
handleUploadSuccess's dependencies, as we merely want to invoke
fetchLists from it, not change our reference.
* Fix failing test
It looks like this broke because EuiTable's pagination changed from a
button to an anchor tag.
* Hide page size options on ValueLists modal table
These have style issues, and anything above 5 rows causes the modal to
scroll, so we're going to disable it for now.
* Update error callbacks now that we have Errors
We don't display the nice errors in the case of an ApiError right now,
but this is better than it was.
* Synchronize delete with the subsequent fetch
Our start() no longer resolves in a meaningful way, so we instead need
to perform the refetch in an effect watching the result of our delete.
* Cast our unknown error to an Error
useAsync generally does not know how what its tasks are going to be
rejected with, hence the unknown.
For these API calls we know that it will be an Error, but I don't
currently have a way to type that generally. For now, we'll cast it
where we use it.
* Import lists code from our new, standardized modules
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
* Adds the ability for exception lists to be multi-list queried
* Fixes a bunch of script issues where I did not update everywhere I needed to use `ip_list` and deletes an old list that now lives within the new/lists folder
* Fixes a few io-ts issues with Encode Decode while I was in there.
* Adds two more types and their tests for supporting converting between comma separated strings and arrays for GET calls.
* Fixes one weird circular dep issue while adding more types.
You now send into the find an optional comma separated list of exception lists their namespace type and any filters like so:
```ts
GET /api/exception_lists/items/_find?list_id=simple_list,endpoint_list&namespace_type=single,agnostic&filtering=filter1,filter2"
```
And this will return the results of both together with each filter applied to each list. If you use a sort field and ordering it will order across the lists together as if they are one list. Filter is optional like before. If you provide less filters than there are lists, the lists will only apply the filters to each list until it runs out of filters and then not filter the other lists.
If at least one list is found this will _not_ return a 404 but it will _only_ query the list(s) it did find. If none of the lists are found, then this will return a 404 not found exception.
**Script testing**
See these files for more information:
* find_exception_list_items.sh
* find_exception_list_items_by_filter.sh
But basically you can create two lists and an item for each of the lists:
```ts
./post_exception_list.sh ./exception_lists/new/exception_list.json
./post_exception_list_item.sh ./exception_lists/new/exception_list_item.json
./post_exception_list.sh ./exception_lists/new/exception_list_agnostic.json
./post_exception_list_item.sh ./exception_lists/new/exception_list_item_agnostic.json
```
And then you can query these two lists together:
```ts
./find_exception_list_items.sh simple_list,endpoint_list single,agnostic
```
Or for filtering you can query both and add a filter for each one:
```ts
./find_exception_list_items_by_filter.sh simple_list,endpoint_list "exception-list.attributes.name:%20Sample%20Endpoint%20Exception%20List,exception-list-agnostic.attributes.name:%20Sample%20Endpoint%20Exception%20List" single,agnostic
```
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
