* Makes width 100% to allow multilang
* Removes state/index types and move those types into the parent types file
* Allows fill form from existing exception by id. Adds unit tests. Fixes wrong comments display when there is more than one comment.
* Allows user update an existing event filter. Adds unit tests. Fixes some wrong behaviours when opening the flyout after create/update action
* Fixes typo
* Fixes wrong entry type
* Uses selectors when it's possible instead of accessing directly to state object
* Fixes typechecks
* Allows edit from the card edit button. Removes unused imports and fixes some types
* Reverts type name
* Changes reducer to don't add entry to the list manually after creation, list will be reloaded with api call. Also check always if data exists to display the add new entry button at the first time
## Summary
Adds deprecation to all the io-ts types and copies of them found in the code base.
Phase 1 (Completed): Copy all the utilities to the `packages/kbn-securitysolution-io-ts-utils`
Phase 2: Add all the deprecation messages about them to the code base
Phase 3+: Teams and others will eventually remove/replace them with the utils from `kbn-securitysolution-io-ts-utils`
* show operator dropdown for path field
refs elastic/security-team/issues/543
* update translation to use consistent values
refs elastic/security-team/issues/543
* update schema to validate path values
refs elastic/security-team/issues/543
* add tests for field and operator values
refs elastic/security-team/issues/543
* review changes
refs elastic/security-team/issues/543
* update schema to enforce dropdown validation for PATH field
refs elastic/security-team/issues/543
* add tests for schema updates
refs 1deab39453
refs elastic/security-team/issues/543
* optimise dropdown list for re-renders
refs elastic/security-team/issues/543
* align input fields and keep alignments when resized
refs elastic/security-team/issues/543
* correctly enter operator data on trusted app CRUD
refs elastic/security-team/issues/543
* update tests
refs 2ac56ee839
refs elastic/security-team/issues/543
* remove redundant code
review changes
* better type assertion
review changes
* move operator options out of component
- these do not depend on component props and thus no need to have it within a useMemo callback.
- review changes
* derive keys from operator entry field
review changes
* update type
* use custom styles for aligning input fields
review changes
* add a custom type for trusted_apps operator
undo changes from list plugin and server/lib/detection_engine
refs 2ac56ee839
refs elastic/security-team/issues/543
* add wildcard entry type
refs elastic/security-team/issues/543
refs https://github.com/elastic/kibana/pull/97623#pullrequestreview-642618462
* use the new entry type
refs elastic/security-team/issues/543
refs https://github.com/elastic/kibana/pull/97623#pullrequestreview-642618462
* update tests
refs elastic/security-team/issues/543
refs https://github.com/elastic/kibana/pull/97623#pullrequestreview-642618462
* update name for wildcard type so that it can be used also for cased inputs
refs elastic/security-team/issues/543
refs f9cb7eddda
* update artifacts to support wildcard entries
refs elastic/security-team/issues/543
* add tests for list schemas
refs f9cb7eddda
refs elastic/security-team/issues/543
* add placeholders for path values
review changes
elastic/kibana/pull/97623#discussion_r620617999
* ignore type check for now
* add type assertion
refs 284352ec9a
* remove unnecessary test
refs 2ac56ee839
* fix types
refs f9cb7eddda
refs b3f5dc4553
* add a note to entries
review changes
refs dbd3532149
* remove redundant type assertions
review changes
refs bcf615ac98
refs b3f5dc4553
* move placeholder text logic to utils
review changes elastic/kibana/pull/97623#discussion_r621673881
refs 6f2d0d7810
* pass the style as prop
review changes
* update api doc
CI check suggestion
* make placeholderText a function expression
review suggestion
elastic/kibana/pull/97623/commits/2dc4fd390cf5ea0e4fa67b3f5fc2561cbb29555e
* use semantic names for functions
refs 330731ebfc
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Initial version of event filtering form/dialog. Pending to add all redux services
* Uses redux store instead of props to get the form values
* Manage errors on redux
* Creates even filter list on service constructor
* Add os type selector depending on form parent by props. Also added create action
* Allows add exception to an event. This commit has to be reviewed and maybe it will change depending on next changes
* Fix imports because changes on ExceptionBuilder component and add needed type export
* Adds constants. Rename eventFilters to eventFilter. Add http wrapper as a hook to check if the list has been created or not
* Adds missing files on last commit.
* Relocate async resource state to be shared between different pages
* Use async resource state to manage async operations on components. Relocate initial entry status to an utils module instead of hook.
* Adds comments into redux store from component
* Fixes typechecks and wrong imports
* Fixes translations and adds subheader and description modal
* Relocates form description
* Removes unused import
* Sanitize entries before submit to remove entry.id
* Missed file on last commit
* Use specific fields for endpoint_event type builder
* Split error field for each kind of errors to prevent unexpected renders. Adds unit test for event filter form component
* Set event.kind == event by default
* Changes folder names. Add notifications when success. Remove default event.king
* Adds notifications when api error and fixed multiple notifications showed for same error
* Adds new test for event filter modal and changes component name to be consistent
* Adds unit tests for event filter notification
* Adds middleware unit tests. Also isolate common event for all tests
* Adds unit tests for event filter reducer
* Adds unit tests for event filter selector
* Fixes same key on different multilanguages. Fixes naming incoherence
* Adds feature flag for event filtering
* Fixes unit tests and weird behavior when changing items after name or comments on event filter form
* Removes unused import
* Fixes unit tests. Add imports from lists plugin. Add expects on tests. Change some names
* Renames everything from eventFilter to eventFilters (plural)
* Rename state variable
* Create hook for notifications instead of a component. Removes className from modal body.
* Updates available fields for enpoint events builder
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Remove `getMockTheme` and use `EuiThemeProvider` from the kibana_react plugin.
Use the CSF-style decorators with `EuiThemeProvider` in the stories.
No functional changes, but should be less code to maintain.
* generate endpoint event filters artifacts
* Add ExperimentalFeature object to the initialization params of ManifestManager
* create event filters artifacts if feature flag is on
* change artifact migration to be less chatty in the logs (also: don't reference Fleet)
* Added new const to List plugin for new Endpont Event Filter list
* Data Generator for event filters ++ script to load event filters (WIP)
* refactor `generate_data` to use `BaseDataGenerator` class
## Summary
Moves part of the exceptions UI out of the security solution plugin and into the lists plugin. In order to keep PRs (relatively) small, I am moving single components at a time. This should also then help more easily pinpoint the source of any issues that come up along the way.
The next couple PRs will focus on the exception builder. This one in particular is focused on moving over the `ExceptionBuilderComponent` which deals with rendering numerous exception items and their entries.
Quick Summary:
- `x-pack/plugins/security_solution/public/common/components/exceptions/builder/` → ` x-pack/plugins/lists/public/exceptions/components/builder/`
- Corresponding unit test file moved as well
- Updated security solution exception builder to pull `ExceptionBuilderComponent` from lists plugin
## Summary
Moves part of the exceptions UI out of the security solution plugin and into the lists plugin. In order to keep PRs (relatively) small, I am moving single components at a time. This should also then help more easily pinpoint the source of any issues that come up along the way.
The next couple PRs will focus on the exception builder. This one in particular is focused on moving over the `BuilderExceptionItem` which deals with rendering the individual exception items.
## Summary
Beginning to move the exceptions UI out of the security solution plugin and into the lists plugin. In order to keep PRs (relatively) small, I plan to move single components at a time. This should also then help more easily pinpoint the source of any issues that come up along the way.
The next couple PRs will focus on the exception builder. This one in particular is focused on moving over the `BuilderEntryItem` which deals with rendering the individual exception item entries. An entry can be of type `match`, `match_any`, `list`, `exists`, or `nested`. The component makes use of the autocomplete fields which use the index patterns to display possible fields and field values.
One of the decisions made in this PR was to have consumers of the `BuilderEntryItem` pass through the autocomplete service as opposed to the `lists` plugin adding it as a dependency. The reason being that it is likely that plugins using the lists plugin will already be consuming either the data plugin or if alerting takes exceptions in, then they'll be consuming alerting. In an effort to avoid some possible icky circular dependency issues, though it best to make the service passed in, as we had already been doing with the hooks in the `lists` plugin.
* Added sync_master file for tracking/triggering PRs for merging master into feature branch
* removed unnecessary (temporary) markdown file
* Trusted apps by policy api (#88025)
* Initial version of API for trusted apps per policy.
* Fixed compilation errors because of missing new property.
* Mapping from tags to policies and back. (No testing)
* Fixed compilation error after pulling in main.
* Fixed failing tests.
* Separated out the prefix in tag for policy reference into constant.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [SECURITY_SOLUTION][ENDPOINT] Ability to create a Trusted App as either Global or Policy Specific (#88707)
* Create form supports selecting policies or making Trusted app global
* New component `EffectedPolicySelect` - for selecting policies
* Enhanced `waitForAction()` test utility to provide a `validate()` option
* [SECURITY SOLUTION][ENDPOINT] UI for editing Trusted Application items (#89479)
* Add Edit button to TA card UI
* Support additional url params (`show`, `id`)
* Refactor TrustedAppForm to support Editing of an existing entry
* [SECURITY SOLUTION][ENDPOINT] API (`PUT`) for Trusted Apps Edit flow (#90333)
* New API route for Update (`PUT`)
* Connect UI to Update (PUT) API
* Add `version` to TrustedApp type and return it on the API responses
* Refactor - moved some public/server shared modules to top-level `common/*`
* [SECURITY SOLUTION][ENDPOINT] Trusted Apps API to retrieve a single Trusted App item (#90842)
* Get One Trusted App API - route, service, handler
* Adjust UI to call GET api to retrieve trusted app for edit
* Deleted ununsed trusted app types file
* Add UI handling of non-existing TA for edit or when id is missing in url
* [Security Solution][Endpoint] Multiple misc. updates/fixes for Edit Trusted Apps (#91656)
* correct trusted app schema to ensure `version` is not exposed on TS type for POST
* Added updated_by, updated_on properties to TrustedApp
* Refactored TA List view to fix bug where card was not updated on a successful edit
* Test cases for card interaction from the TA List view
* Change title of policy selection to `Assignment`
* Selectable Policy CSS adjustments based on UX feedback
* Fix failing server tests
* [Security Solution][Endpoint] Trusted Apps list API KQL filtering support (#92611)
* Fix bad merge from master
* Fix trusted apps generator
* Add `kuery` to the GET (list) Trusted Apps api
* Refactor schema with Put method after merging changes with master
* WIP: allow effectScope only when feature flag is enabled
* Fixes errors with non declared logger
* Uses experimental features module to allow or not effectScope on create/update trusted app schema
* Set default value for effectScope when feature flag is disabled
* Adds experimentals into redux store. Also creates hook to retrieve a feature flag value from state
* Hides effectPolicy when feature flag is not enabled
* Fixes unit test mocking hook and adds new test case
* Changes file extension for custom hook
* Adds new unit test for custom hook
* Hides horizontal bar with feature flag
* Compress text area depending on feature flag
* Fixes failing test because feature flag
* Fixes wrong import and unit test
* Thwrows error if invalid feature flag check
* Adds snapshoot checks with feature flag enabled/disabled
* Test snapshots
* Changes type name
* Add experimentalFeatures in app context
* Fixes type checks due AppContext changes
* Fixes test due changes on custom hook
Co-authored-by: Paul Tavares <paul.tavares@elastic.co>
Co-authored-by: Bohdan Tsymbala <bohdan.tsymbala@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Paul Tavares <56442535+paul-tavares@users.noreply.github.com>
* Use client from branch
* Get type checking working in core
* Fix types in other plugins
* Update client types + remove type errors from core
* migrate Task Manager Elasticsearch typing from legacy library to client library
* use SortOrder instead o string in alerts
* Update client types + fix core type issues
* fix maps ts errors
* Update Lens types
* Convert Search Profiler body from a string to an object to conform to SearchRequest type.
* Fix SOT types
* Fix/mute Security/Spaces plugins type errors.
* Fix bootstrap types
* Fix painless_lab
* corrected es typing in Event Log
* Use new types from client for inferred search responses
* Latest type defs
* Integrate latest type defs for APM/UX
* fix core errors
* fix telemetry errors
* fix canvas errors
* fix data_enhanced errors
* fix event_log errors
* mute lens errors
* fix or mute maps errors
* fix reporting errors
* fix security errors
* mute errors in task_manager
* fix errors in telemetry_collection_xpack
* fix errors in data plugins
* fix errors in alerts
* mute errors in index_management
* fix task_manager errors
* mute or fix lens errors
* fix upgrade_assistant errors
* fix or mute errors in index_lifecycle_management
* fix discover errors
* fix core tests
* ML changes
* fix core type errors
* mute error in kbn-es-archiver
* fix error in data plugin
* fix error in telemetry plugin
* fix error in discover
* fix discover errors
* fix errors in task_manager
* fix security errors
* fix wrong conflict resolution
* address errors with upstream code
* update deps to the last commit
* remove outdated comments
* fix core errors
* fix errors after update
* adding more expect errors to ML
* pull the lastest changes
* fix core errors
* fix errors in infra plugin
* fix errors in uptime plugin
* fix errors in ml
* fix errors in xpack telemetry
* fix or mute errors in transform
* fix errors in upgrade assistant
* fix or mute fleet errors
* start fixing apm errors
* fix errors in osquery
* fix telemetry tests
* core cleanup
* fix asMutableArray imports
* cleanup
* data_enhanced cleanup
* cleanup events_log
* cleaup
* fix error in kbn-es-archiver
* fix errors in kbn-es-archiver
* fix errors in kbn-es-archiver
* fix ES typings for Hit
* fix SO
* fix actions plugin
* fix fleet
* fix maps
* fix stack_alerts
* fix eslint problems
* fix event_log unit tests
* fix failures in data_enhanced tests
* fix test failure in kbn-es-archiver
* fix test failures in index_pattern_management
* fixing ML test
* remove outdated comment in kbn-es-archiver
* fix error type in ml
* fix eslint errors in osquery plugin
* fix runtime error in infra plugin
* revert changes to event_log cluser exist check
* fix eslint error in osquery
* fixing ML endpoint argument types
* fx types
* Update api-extractor docs
* attempt fix for ese test
* Fix lint error
* Fix types for ts refs
* Fix data_enhanced unit test
* fix lens types
* generate docs
* Fix a number of type issues in monitoring and ml
* fix triggers_actions_ui
* Fix ILM functional test
* Put search.d.ts typings back
* fix data plugin
* Update typings in typings/elasticsearch
* Update snapshots
* mute errors in task_manager
* mute fleet errors
* lens. remove unnecessary ts-expect-errors
* fix errors in stack_alerts
* mute errors in osquery
* fix errors in security_solution
* fix errors in lists
* fix errors in cases
* mute errors in search_examples
* use KibanaClient to enforce promise-based API
* fix errors in test/ folder
* update comment
* fix errors in x-pack/test folder
* fix errors in ml plugin
* fix optional fields in ml api_integartoon tests
* fix another casting problem in ml tests
* fix another ml test failure
* fix fleet problem after conflict resolution
* rollback changes in security_solution. trying to fix test
* Update type for discover rows
* uncomment runtime_mappings as its outdated
* address comments from Wylie
* remove eslint error due to any
* mute error due to incompatibility
* Apply suggestions from code review
Co-authored-by: John Schulz <github.com@jfsiii.org>
* fix type error in lens tests
* Update x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_service.ts
Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
* Update x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_service.test.ts
Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
* update deps
* fix errors in core types
* fix errors for the new elastic/elasticsearch version
* remove unused type
* remove unnecessary manual type cast and put optional chaining back
* ML: mute Datafeed is missing indices_options
* Apply suggestions from code review
Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com>
* use canary pacakge instead of git commit
Co-authored-by: Josh Dover <me@joshdover.com>
Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com>
Co-authored-by: Gidi Meir Morris <github@gidi.io>
Co-authored-by: Nathan Reese <reese.nathan@gmail.com>
Co-authored-by: Wylie Conlon <wylieconlon@gmail.com>
Co-authored-by: CJ Cenizal <cj@cenizal.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
Co-authored-by: restrry <restrry@gmail.com>
Co-authored-by: James Gowdy <jgowdy@elastic.co>
Co-authored-by: John Schulz <github.com@jfsiii.org>
Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
### Summary
In preparation for moving all the exceptions UI components into the lists plugin adds some linting, adds the lists plugin to the i18n config and adds storybook support. Tried to add a bit stricter linting than exists in the security solution right now, rules that we've talked about wanting to enable.
* Escape quotes in list ids and quote the id in KQL query
* Remove decodeURIComponent because too many KQL queries don't handle quotes
* Add quotes to user supplied IDs for other KQL queries
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
### Summary
This PR is a follow-up to #89066 - which fixed the same issue occurring with indicator match lists UI. The lack of stable ids for exception item entries resulted in some funky business by where invalid values could overwrite other values when deleting entries in the builder.
Doing a quick refactor to help with #90634 . While working on #90634 found that I would introduce a circular dependency if I didn't refactor the hook used by the exception builder component to make use of the existing useApi hook in the lists plugin.
#90634 adds temporary ids to item entries to mitigate some React key requirements and the logic to remove/add these id's isn't at the boundary but in the hooks (so as not to pollute the data for everyone wanting to use the api.
An upside is that it removed some of the looping and seemed to speed things up a bit. I briefly considered adding the bulk SO endpoints for exception items, but they are still experimental.
Addresses #88450
Issue
Search was not working as expected was because the exception list property name is mapped as a keyword - this means it does not get tokenized which is why one word searches were working but if the name included multiple words and was partial, it was not filtering properly.
* Initial version of adding artifacts per policy to the manifest.
* Minor renaming to convey the purpose of the variable.
* Added ability to override list item mock data.
* Changed function signature to be more reusable.
* Implementationg of support of artifacts per policy in the manifest data structure.
* Added saved objects migrations.
* Renamed the endpoint to reflect that it's artifacts endpoint.
* Fixed tests.
* Fixed the manifest data.
* Fixed linting errors (result of merge).
* Updated ES mappings for manifest in all test setups.
* Updated hash in the mappings.
* Fixed the typo that lead to failing test.
* Fixed the problem with manifest not being dispatched to policies if there are same artifact names but different content. Artifact name in the ManifestSchema is not unique id, hence added decoded_sha256 to the comparison. Added test case to cover this.
* Fixed the problem with the task flow when failure to dispatch to policies will result in commited manifest and no redispatch on next task run. Changed tests to reflect new flow (actually restored previous flow).
* Forgot to commit changes in mock.
* Made other tests more readable using same varialbe naming pattern.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* first pass
* migrate more plugins
* migrate yet more plugins
* more oss plugins
* fix test file
* change Plugin signature on the client-side too
* fix test types
* migrate OSS client-side plugins
* migrate OSS client-side test plugins
* migrate xpack client-side plugins
* revert fix attempt on fleet plugin
* fix presentation start signature
* fix yet another signature
* add warnings for server-side async plugins in dev mode
* remove unused import
* fix isPromise
* Add client-side deprecations
* update migration examples
* update generated doc
* fix xpack unit tests
* nit
* (will be reverted) explicitly await for license to be ready in the auth hook
* Revert "(will be reverted) explicitly await for license to be ready in the auth hook"
This reverts commit fdf73feb
* restore await on on promise contracts
* Revert "(will be reverted) explicitly await for license to be ready in the auth hook"
This reverts commit fdf73feb
* Revert "restore await on on promise contracts"
This reverts commit c5f2fe51
* add delay before starting tests in FTR
* update deprecation ts doc
* add explicit contract for monitoring setup
* migrate monitoring plugin to sync
* change plugin timeout to 10sec
* use delay instead of silence
## Summary
Addresses issue issue 87110
**Issue**
When prepackaged rules were created during the Endpoint Security enrollment flow, the endpoint exceptions list was failing to be created. As a result, when a user navigated to the `Endpoint Security` rule to add an exception it would display errors and not allow a user to add exceptions.
* remove canUserCRUD from signal actions and remove refresh param from open_close_signals route. 'refresh' requires maintenance / manage / all privileges for signals index
* adds 'maintenance' to privileges route
* fix unit teset typing
* update tests, updated lists e2e tests since it relies on the readPrivileges function of SIEM so any changes to the expected response from there must also be changed in the lists privileges route
* update scripts roles to include maintenance for roles that do not have privileges higher than 'maintenance'
* fix open-close signals integration test
### Summary
This PR concentrates on fixing the deletion on the exceptions list table view. This fix is intermediary and a more thorough, backend solution is needed. Currently, if you delete an exception list, it deletes the exception list SO, but does not remove references to it from rules. This PR allows for a quick fix conducting this logic client side.
## Summary
Currently working on issues related to exceptions and it was noted on a separate PR that the request payload validation being done in the client side API calls was unnecessary. It was helpful in development, but not of any added value in production. Not only that, but the extra validations also add to the performance hit.
Removed the payload validation and formatted the code to follow the same pattern as that in the value lists api file.
Tested that exceptions flows not affected by testing out exceptions CRUD flows.
## Summary
This PR addresses a fix on the exceptions list table export functionality. A dedicated route for exception list export needed to be created. List is exported into an `.ndjson` format.
Exception lists consist of two elements - the list itself, and its items. The export file should now contain both these elements, the list followed by its items.
## Summary
Resolves https://github.com/elastic/kibana/issues/77324, https://github.com/elastic/kibana/issues/77325, resolves https://github.com/elastic/kibana/issues/77325, and resolves https://github.com/elastic/kibana/issues/81302
This PR addresses referential integrity issues when deleting value lists. Previously when deleting value lists, any references in Exception Lists/Items would be left behind. This PR introduces a new confirmation modal when deleting value lists that are referenced in either space aware (`simple`) or space `agnostic` exception lists.
Also includes:
* Fixed Lists plugin `quick_start.sh` as it was using endpoint exception list + value lists (unsupported)
* Adds `quick_start_value_list_references.sh` to create exception lists/items, value lists, and references to easily test
* Add support to `findExceptionList` for searching for both `simple` and `agnostic` list types
* Two new query params have been added to the `deleteListRoute`
* `ignoreReferences` (default:false) when true, maintains pre-7.11 behavior of deleting value list without performing any additional checks.
* NOTE: As written, this becomes an API breaking change as existing existing calls to the same API will `409` conflict if references exist. cc @jmikell821 @DonNateR
* `deleteReferences` (default:false) to perform dry run and identify referenced exception lists/items
## Testing
To test, run `quick_start_value_list_references.sh` and it will create all the necessary resources/references to easily exercise the above functionality. The below diagram details the resources created and how the references are wired up.
> Creates three different exception lists and value lists, and associates as
> below to test referential integrity functionality.
>
> NOTE: Endpoint lists don't support value lists, and are not tested here
>
> EL: Exception list
> ELI Exception list Item
> VL: Value list
>
> EL1 EL2 (Agnostic) EL3
> | | |
> ELI1 ELI2 ELI3
> |\ /| |
> | \ / | |
> | \ / | |
> | \ / | |
> | \/ | |
> | /\ | |
> | / \ | |
> | / \ | |
> | / \ | |
> |/ \| |
> VL1 VL2 VL3 VL4
> ips.txt ip_range.txt text.txt hosts.txt
>
Corner cases to be aware of:
* An exception item may have multiple value list entries -- only referenced value list entries should be removed
* There is no API for removing individual entries. If all entries are references the entire item is deleted. If only some entries are references, the item is updated via a `PUT` (no `PATCH` support for exception items)
* It's not possible via the UI to create a space agnostic list that has value list exception items (only agnostic endpoint exception lists can be created and they do not support value lists). Please use above script to exercise this behavior.
Additional notes:
* Once the Exception List table is introduced (https://github.com/elastic/kibana/pull/85465), we can add an enhancement for deeplinking to exception lists from the reference error modal.
* The `deleteListRoute` response has been updated to include the responses from the reference checks to provide maximum flexibility
* There is no bulk API for deleting exception list items, and so they are iterated over via the `deleteExceptionListItem` API.
##### Reference error modal
<p align="center">
<img width="500" src="https://user-images.githubusercontent.com/2946766/102199153-813e1e80-3e80-11eb-8a9b-af116ca13df9.gif" />
</p>
##### Overflow example
<p align="center">
<img width="500" src="https://user-images.githubusercontent.com/2946766/102199032-5784f780-3e80-11eb-81c7-17283d002ce4.gif" />
</p>
### Checklist
Delete any items that are not applicable to this PR.
- [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
- [X] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
### For maintainers
- [X] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
* Adds new SO type for persisting our signals migrations
* WIP: Migration status SO client
Trying to forge a patterrn using io-ts to validate at runtime. I think
I've got it working but I want to refactor the pipeline out into a
reusable function(s).
* Implements our SavedObjects service for signals migrations
* Defines a simple client that delegates to the base SO client with
our SO type
* Defines a service that consumes the simpler client, adding validations
and data transforms on top.
* Refactoring migration code to work with saved objects
As opposed to the previous ephemeral, encoded tokens, we now retrieve migration
info from saved objects.
At the API level, this means that both the create and finalize endpoints
receive a list of concrete indices. No more passing around tokens.
As both endpoints are idempotent, users can hammer them as much as they
want with the same lists of indices. Redundant creates and finalizes
will be met with inline 400 messages, and as one continues to poll the
finalize endpoint they should see more and more indices respond with
"completed: true"
* Fixing integration tests first, and anything upstream breaking them
* Clean up API integration tests
* standardize assignment of responses (with types)
* deletes migration SOs as test cleanup
* Split API tests into separate files
This was getting big and unwieldy; this splits these into one file per
endpoint.
* Refactor: split existing migration service functionality into atomic functions
This will allow us to repurpose the service to compose more
functionality and be more specifically useful, while keeping the
component logic separate.
* WIP: moving logic into migrationService.create
* Splitting get_migration_status into component functions
getMigrationStatus was really two separate aggregations, so I split them
out and we recompose them in the necessary routes.
* Move finalization logic into function
* migrationService exposes this as .finalize()
* adds an error field to our migration SO
* We currently only have one error that we persist there, but it would
be very time-consuming to track down that information were it not
there.
* Adds function for migration "deletion" logic
* migrationService leverages this function
* adds new boolean to our savedObject
* deletes unused function (deleteMigrationSavedObject)
* Adds route for soft-deletion of migrations
* Updating tests related to migration status
* Adding/updating mocks/unit tests necessary to satisfy the things I
need to test
* I mainly wanted to test that the the status endpoint filtered out the
deleted migrations; this was accomplished with a unit test after
fleshing out some mocks/sample data.
* Move old migration service tests to the relevant function tests
This logic was previously moved out into component functions; this moves
the tests accordingly.
* Add some unit tests around our reindex call
* Fix create migration route tests
Mocks out our migration functions, rather than stubbing ES calls
directly.
* Updates finalize route unit tests
Addresses functionality that hasn't been moved to finalizeMigration()
* Unit tests our finalization logic
Fixes a bug where we weren't accounting for soft-deleted migrations.
ALso updates our test migration SO to have a status of 'pending' as
that's a more useful default.
* Fixes finalization integration tests
These were failing due:
* a change in the migration status API response
* a bug I introduced in the finalize route
* Adds tests for our migration deletion endpoint
* unit tests
* API integration tests
* Caught/fixed bug with deleting a successful migration
* Fixes types
Removes unused code.
* Prevent race condition due to template rollover during migration
If a user has an out of date index (v1) relative to the template (v2), but the
template itself is out of date (newest is v3), then it's possible that
the template is rolled over to v3 after the v1-v2 migration has been
created but before the new index has been created.
In such a case, the new index would receive the v3 mappings but would
incorrectl be marked as v2. This shouldn't necessarily be an issue, but
it's an unnecessary state that can easily be prevented with the guard
introduced here.
* Add real usernames to migration savedObjects
In addition to the SOs themselves giving us observability into what
migration actions were performed, this gives us the additional info of
_who_ performed the action.
* Index minimal migration SO fields needed for current functionality
* Add additional migration info to status endpoint
This will allow users to finalize a migration if they've lost the
response to their POST call.
* Finalize endpoint receives an array of migration IDs, not indices
This disambiguates _which_ migrations we were finalizing if you passed
an index (which was previously: the most recent migration).
* Fix type errors in tests after we threaded through username
* Update responsibilities of migration finalize/delete endpoints
Discussions with @marshallmain lead to the following refactor:
* finalize does not delete tasks
* finalize only applies cleanup policy to a failed migration
* delete takes an array of migration ids (like finalize)
* delete hard-deletes the SavedObject of a completed (failed or
successful) migration
This gives a bit more flexibility with the endpoints, as well as
disambiguates the semantics: it just deletes migrations!
* Fix tests that were broken during refactoring
* Fix type errors
I removed some logic here but forgot the imports :(
* Move outdated integration test
In the case of a successful migration, application of the cleanup policy
is done by the deletion endpoint. In the interest of data preservation,
we do not delete a sourceIndex unless it is explicitly deleted.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes different bugs/issues when using exceptions with value based lists for both the UI, the backend, and the large value based lists. See https://github.com/elastic/kibana/issues/79516, but this also fixes several other bugs found mentioned below.
For the front end UI:
* Adds the ability to specify value based lists that are IP Ranges when the source event is an IP. Before you could only match IP to IP and the IP Ranges lists could not be used.
* Breaks down a few functions into smaller functions for unit test writing abilities.
You can now add ip ranges as list values for the UI when before it would not show up:
<img width="1035" alt="Screen Shot 2020-12-07 at 2 15 39 PM" src="https://user-images.githubusercontent.com/1151048/101406552-d6819b00-3896-11eb-9fb5-4c7c2ad93b2e.png">
For value based lists:
* Fixes text data type to use "and" between matching using `operator: 'and'` and changes it from a `terms query to a `match` query
* Adds new API for searching against types called `searchListItemByValues ` so that numeric, text, array based, and other non-stringable types can be sent and then the value based lists will push that to ElasticSearch. This shifts as many corner cases and string/numeric coercions to ElasticSearch rather than Kibana client side code.
* Adds ability to handle arrays within arrays through a `flatten` call.
* Utilizes the `named queries` from ElasticSearch for the new API so that clients can get which parts matched and then use that for their exception list logic rather than in-memory string to string checks. This fixes CIDR and ranges as well as works with arrays.
For the backend exception lists that used value based lists:
* Broke down the `filterEventsAgainstList` function into a folder called `filters` and the functions into other files for better unit based testing.
* Changed the calls from `getListItemByValues` to `searchListItemByValues` which can return exactly what it matched against and this will not break anyone using the existing REST API for `getListItemByValues` since that REST API and client side API stays the same.
* Cleaned up extra promises being used in a few spots that async/await automatically will create.
* Removed the stringabilities and stringify in favor of just a simpler exact check using `JSON.stringify()`
For the tests:
* Adds unit tests to broken down functions
* Adds ip_array, keyword_array, text_array, FTR tests for the backend.
* Adds more CIDR and range based FTR tests for the backend.
* Unskips and fixes all the numeric tests and range tests that could not operate previously from bugs.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
* Separated out service layer for trusted apps.
* Improved the type structure a bit to avoid using explicit string literals and to add possibility to return OS specific parts of trusted app object in type safe manner.
* Added support for mapping of trusted app to exception item and back.
* Changed schema to support signer in the API.
* Renamed utils to mapping.
* Exported some types in lists plugin and used them in trusted apps.
* Added tests for mapping.
* Added tests for service.
* Switched deletion to use exceptions for not found case.
* Added resetting of the mocks in service layer tests.
* Added handlers tests.
* Refactored mapping tests to be more granular based on the case.
* Restored lowercasing of hash.
* Added schema tests for signer field.
* Removed the grouped tests (they were split into tests for separate concerns).
* Corrected the tests.
* Lowercased the hashes in the service test.
* Moved the lowercasing to the right location.
* Fixed the tests.
* Added test for lowercasing hash value.
* Introduced OperatingSystem enum instead of current types.
* Removed os list constant in favour of separate lists in places that use it (each place has own needs to the ordering).
* Fixed the missed OperatingSystem enum usage.
* bump version to 4.1.1-rc
* fix code to run kbn bootstrap
* fix errors
* DO NOT MERGE. mute errors and ping teams to fix them
* Address EuiSelectableProps configuration in discover sidebar
* use explicit type for EuiSelectable
* update to ts v4.1.2
* fix ts error in EuiSelectable
* update docs
* update prettier with ts version support
* Revert "update prettier with ts version support"
This reverts commit 3de48db3ec.
* address another new problem
Co-authored-by: Chandler Prall <chandler.prall@gmail.com>
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* ExceptionIdentifiers -> ExceptionListIdentifiers
* Pass refreshRule through to alert context menu
* Fix type errors and rename refreshRule to onRuleChange for consistency
* Finish adding .lower to exceptionable fields
* Add back migrations
* .lower -> .caseless
* Add separate field for os type
* updates
* Type updates
* Switch over to osTypes
* get rid of _tags
* Add tests for schema validation
* Remove remaining references to _tags
* Another round of test fixes
* DefaultArray tests
* More test fixes
* Fix remaining test failures
* types / tests
* more test updates
* lowercase os values
* Address feedback + fix test failure
* tests
* Fix integration test
* process.executable.path -> process.executable.caseless
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
