## Rule duplication with/without exceptions
Majority of work done by @yctercero in this
[branch](https://github.com/yctercero/kibana/tree/dupe)
Some integration tests are left, but PR is ready for review.
2 flow when you duplicate rule:
### Without exceptions
Don't duplicate any exceptions
### With exceptions
Shared exceptions should duplicate reference
Rule default exceptions are not duplicated by reference, but create a
copy of exceptions. So if you remove it from duplicate rules, the
original rule is not changed.
https://user-images.githubusercontent.com/7609147/200863319-4cb56749-42dd-42d8-8896-f45782c21838.mov
# TODO;
[] integrations tests
[] cypress tests
Co-authored-by: Yara Tercero <yara.tercero@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* squashed commit of updates to add/edit flyouts for exception, added cypress tests and unit tests
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Devin W. Hurley <devin.hurley@elastic.co>
## Summary
Adds components shared between new add/edit exception flyouts. Does not yet modify the flyouts themselves. Trying to break down what would be an even larger PR into chunks.
## Summary
**API changes**
- Adds API for determining the list-rule references.
- Updates the exception items find api to include the `search` param which allows for simple search queries - used with the EUI search bar
**UI updates**
- Moved the exception components into new `rule_exceptions` folder per suggested folder structure updates listed [here](https://github.com/elastic/kibana/issues/138600)
- Updates the rule details tabs to split endpoint and rule exceptions into their own tabs
- Updates the viewer utilities header now that these different exception types are split
- Updates exception item UI to match new designs
- Updates the UI for when there are no items
- Removes `use_exception_list_items` hook as it is no longer in use
- Flyouts (add/edit) remain untouched
* new route for create an exception list and return the existing one if alredy exists
* Fixes unit test and shows error when ignore_existing set to false and there is a conflict
* Remove query param and update route name to be more specific
* Fixes unit test
* Enforce list_id and type types for internal route. Added unit tests
* Uses existing constants to define list_ids
* Don't create host isolation exeptions api client if not needed when checking links availability
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
The lists plugin stores large value lists in two data indices - `.lists-*` and `.items-*`. These were still using the legacy ES template. This PR updates relevant routes to now use the new index templates.
- `createListsIndexRoute` now uses the new template routes and checks for legacy templates to delete them
- `deleteListsIndex` now uses up to date ES API
- Updates the templates to follow new structure
* Creating empty @kbn/core-saved-objects-common package
* start moving types around
* start fixing imports
* fix entrypoint exports
* fix external import
* create explicit ISavedObjectsRepository interface
* fix another external usage
* rewrite browser exports
* create explicit SavedObjectsClientContract interface
* move client/repository types to @kbn/core-saved-objects-api-server
* start fixing imports
* one more
* fix global re-exports
* fix some browser-side imports
* fix more violations
* prepare the browser-side client
* fix one more usage
* fix external usage
* fix more external usages
* one more
* Create @kbn/core-saved-objects-api-browser package
* fix more usages of error helper
* fix more internal imports
* use interface for SSO
* adapt more imports
* damn those types were a mess
* fix more usages of SSO
* Revert "fix more usages of SSO"
This reverts commit 07a12e5353.
* Revert "use interface for SSO"
This reverts commit 6240fc86c5.
* export the interface with the old name instead.
* adapt tests concrete usages of SSO
* export reference type, more fixes
* this gonna be long
* one more
* other resolve type change
* more usages
* Am I getting close?
* yet more fixes
* back to client impl
* fix bulkGetting undefined
* fix SS mock
* some cleanup
* self-review
* fix new usages
* Implement wildcard exceptions for detection rules
* Fix index pattern retrieval on edit exceptions flyout
* Fix API integration test logic
* Fix entry_renderer linting
* Remove bad fix idea
* Add 'does not match' operator to UI
* Fix test
* Add unit tests
* Add wildcard exceptions to list of DE exception operators
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* chore(NA): eslint rule for disallowing naked eslint-disable
* chore(NA): export new rule and update docs
* chore(NA): creation of rule in ts
* chore(NA): new corrected rule in ts
* refact(NA): remove old logic from older plugin
* docs(NA): update documentation
* docs(NA): update documentation
* docs(NA): update documentation
* refact(NA): include edge cases for better locating errors
* chore(NA): changed regex name
* docs(NA): correct name rule on docs
* refact(NA): use dedent in the template literals
* refact(NA): check for undefined
* fix(NA): introduces support for eslint-disable-line
* chore(NA): fix extra space
* test(NA): created more test cases
* chore(NA): rename plugin to eslint-plugin-disable
* docs(NA): update nav and operations landing page ids for eslint rule
* test(NA): use messageIds on test
* chore(NA): complete naked eslint disables with specific rules
* chore(NA): specific rules for a few naked eslint disable
* chore(NA): add focused eslint disable on big reindex_operation_with_large_error_message.ts file
* chore(NA): changes according PR feedback
* chore(NA): include specific eslint rules on latest naked eslint disable
* chore(NA): missing eslint disable specific rule
* fix(NA): remove comment for js annotator
* chore(NA): re add eslint focused disable rule to x-pack/plugins/osquery/cypress/support/coverage.ts
* chore(NA): re add eslint focused disable rule to x-pack/plugins/osquery/cypress/support/coverage.ts
* chore(NA): re add eslint focused disable rule to x-pack/plugins/osquery/cypress/support/coverage.ts
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Introduce the IKibanaRequest interface
* swap to keeping the KibanaRequest name for the interface
* adapt usages
* fix mock
* export CoreKibanaRequest for now...
* adapt imports from tests
* lint
* more missed usages in tests
* fix another instanceof...
* improve tsdoc
Addresses #86258
The variable tracking state needed to be cleared on operator change. If you didn't change operators, then invalidating and then validating an entry worked as expected, but if you switched operators, the error state was not being cleared causing the builder to read that there was an error state.
* Add exceptions to threshold timeline
* Tests and error handling
* Fix unit tests
* Add alias for exceptions filter
* Fix tests
* Type fixes
Co-authored-by: Marshall Main <marshall.main@elastic.co>
* Update warning text for event filter matches operator when file path has wildcards
fixes elastic/security-team/issues/3199
* update text
review changes
### Summary
Addresses https://github.com/elastic/kibana/issues/124742
#### Issue TLDR
Import of rules that reference exception items with comments fail. Failure message states that comments cannot include `created_at`, `created_by`, `id`.
## Summary
Exposes the functionality of
* search_after
* point in time (pit)
From saved objects to the exception lists. This _DOES NOT_ expose these to the REST API just yet. Rather this exposes it at the API level to start with and changes code that had hard limits of 10k and other limited loops. I use the batching of 1k for this at a time as I thought that would be a decent batch guess and I see other parts of the code changed to it. It's easy to change the 1k if we find we need to throttle back more as we get feedback from others.
See this PR where `PIT` and `search_after` were first introduced: https://github.com/elastic/kibana/pull/89915
See these 2 issues where we should be using more paging and PIT (Point in Time) with search_after: https://github.com/elastic/kibana/issues/93770https://github.com/elastic/kibana/issues/103944
The new methods added to the `exception_list_client.ts` client class are:
* openPointInTime
* closePointInTime
* findExceptionListItemPointInTimeFinder
* findExceptionListPointInTimeFinder
* findExceptionListsItemPointInTimeFinder
* findValueListExceptionListItemsPointInTimeFinder
The areas of functionality that have been changed:
* Exception list exports
* Deletion of lists
* Getting exception list items when generating signals
Note that currently we use our own ways of looping over the saved objects which you can see in the codebase such as this older way below which does work but had a limitation of 10k against saved objects and did not do point in time (PIT)
Older way example (deprecated):
```ts
let page = 1;
let ids: string[] = [];
let foundExceptionListItems = await findExceptionListItem({
filter: undefined,
listId,
namespaceType,
page,
perPage: PER_PAGE,
pit: undefined,
savedObjectsClient,
searchAfter: undefined,
sortField: 'tie_breaker_id',
sortOrder: 'desc',
});
while (foundExceptionListItems != null && foundExceptionListItems.data.length > 0) {
ids = [
...ids,
...foundExceptionListItems.data.map((exceptionListItem) => exceptionListItem.id),
];
page += 1;
foundExceptionListItems = await findExceptionListItem({
filter: undefined,
listId,
namespaceType,
page,
perPage: PER_PAGE,
pit: undefined,
savedObjectsClient,
searchAfter: undefined,
sortField: 'tie_breaker_id',
sortOrder: 'desc',
});
}
return ids;
```
But now that is replaced with this newer way using PIT:
```ts
// Stream the results from the Point In Time (PIT) finder into this array
let ids: string[] = [];
const executeFunctionOnStream = (response: FoundExceptionListItemSchema): void => {
const responseIds = response.data.map((exceptionListItem) => exceptionListItem.id);
ids = [...ids, ...responseIds];
};
await findExceptionListItemPointInTimeFinder({
executeFunctionOnStream,
filter: undefined,
listId,
maxSize: undefined, // NOTE: This is unbounded when it is "undefined"
namespaceType,
perPage: 1_000,
savedObjectsClient,
sortField: 'tie_breaker_id',
sortOrder: 'desc',
});
return ids;
```
We also have areas of code that has perPage listed at 10k or a constant that represents 10k which this removes in most areas (but not all areas):
```ts
const items = await client.findExceptionListsItem({
listId: listIds,
namespaceType: namespaceTypes,
page: 1,
pit: undefined,
perPage: MAX_EXCEPTION_LIST_SIZE, // <--- Really bad to send in 10k per page at a time
searchAfter: undefined,
filter: [],
sortOrder: undefined,
sortField: undefined,
});
```
That is now:
```ts
// Stream the results from the Point In Time (PIT) finder into this array
let items: ExceptionListItemSchema[] = [];
const executeFunctionOnStream = (response: FoundExceptionListItemSchema): void => {
items = [...items, ...response.data];
};
await client.findExceptionListsItemPointInTimeFinder({
executeFunctionOnStream,
listId: listIds,
namespaceType: namespaceTypes,
perPage: 1_000,
filter: [],
maxSize: undefined, // NOTE: This is unbounded when it is "undefined"
sortOrder: undefined,
sortField: undefined,
});
```
Left over areas will be handled in separate PR's because they are in other people's code ownership areas.
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* validation for Pre GET one of host isolation exceptions.
* adjust checks for host isolation validation
* Add validation for import for all artifacts
* Validate host isolation exceptions exports
* Validate host isolation exceptions multi list find
* Validate host isolation exceptions single list find
* Validate host isolation exceptions Summary
* add FTR tests to validate authz
* Update all exception extension point handlers to use the ExceptionListClient passed in on context
* Refactored ExceptionListItemGenerator a bit and added methods to get Host Isolation exceptions
* Update handlers to immediately exit if the namespace_type is not `agnostic`
* Improved `log.info` messages in artifact and policy services
* Add `lists-summary` to Security solution `all` feature privilege (was missing)
* Add an instance of ExceptionListClient with server extension points turned off to the `context` provided to callbacks
* Unit test cases to validate context
* Don't show a default value '-' for emoty descriptions on artifacts list. Also removes empty spaces
* Update copy to say 'event filters' instead of 'exceptions'
* Decrease spacing between avatar and comments textbox
* Adds extra spacing between last exception builder field and the buttons group
* Reduces effect scope togle width to by dynamic depending on translations
* Makes effected policy button group persistent across different artifact forms
* Removes unused import
* Center button group for small devices
* update summary endpoint to use filters and use that for fleet event filters cards
fixes elastic/security-team/issues/2513
* update tests
fixes elastic/security-team/issues/2513
* update host isolation card to show total as the actual number of artifacts
fixes elastic/kibana/issues/121507
* fix types
missing merge updates
* use named constant for isolation exception list
review changes
* Update fleet_integration_event_filters_card.tsx
review changes
* fix the total on summary api
review suggestions
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* add extension point for import by stream
* add extension point for import by array
* Add mocks and tests for import
* adjust tests for import to use `ExceptionListClient#importExceptionListAndItems()`
* Export the additional Extension point Types from server
* Extension point for getting one exception item
* Extension point for single list `find*`
* Extension point for multi list `find*`
* extension point for export exceptions list
* extension point for get summary
* extension point for Delete exception item
## Lists Plugin changes:
- Modified ExceptionListClient to accept an optional KibanaRequest when instantiating a new instance of the class
- Changes the extension points callback argument structure to an object having context and data. Context provides to the callbacks the HTTP request so that additional validation can be performed (ex. Authz to certain features)
- ExtensionPointStorageClient#pipeRun() will now throw if an extension point callback also throws an error (instead of logging it and continuing on with callback execution)
- ErrorWithStatusCode was export'ed out of the server (as ListsErrorWithStatusCode) and available for use by dependent plugins
## Security Solution Plugin (endpoint) changes:
- Added new getEndpointAuthz(request) and getExceptionListsClient() methods to EndpointAppContextService
- Added new server lists integration modules. Registers extension points with the Lists plugin for create and update of exception items. Currently validates only Trusted Apps
- Added exception item artifact validators:
- a BaseValidator with several generic and reusable methods that can be applied to any artifact
- a TrustedAppValidator to specifically validate Trusted Applications
- Refactor:
- moved EndpointFleetServices to its own folder and also renamed it to include the word Factory (will help in the future if we create server-side service clients for working with Endpoint Policies)
- Created common Artifact utilities and const's for working with ExceptionListItemSchema items
* Lists plugin framework for registering extension points
* Support for two extension points for Exceptions List
* `ExceptionListClient` changed to executed extension points
* Security Solution: Change security solution `getExceptionListClient()` to use the Lists plugin factory
## Summary
Without the added overwrite support for exceptions separate from rules, unexpected user behavior experienced. This PR does the following:
- Updates the import rules modal text to account for exceptions
- Updates the import rules modal logic to account for the exceptions overwrite option
- Users can now select to overwrite rules, exceptions or both
- Updates the backend logic in the rules import route to batch checking if the exception lists referenced by the rules trying to be imported exist. If the list does not exist, it removes the reference before trying to import the rule. Previously, this check was being done one by one for each rule.
- Added effort to try to speed up the import after added exceptions logic from original PR slowed down functionality
