* Add API functions and hooks for reading and creating the lists index
* Ensure KibanaApiError extends the Error interface
It has a name, so we should type it as such. This way, we can use it
anywhere that an Error is accepted.
* Return an Error from validationEither and thus from our useAsync hooks
Because an io-ts pipeline needs a consistent type across its left value,
and validateEither was returning a string, we were forcing all our
errors to strings. In the case of an API error, however, this meant a
loss of data, since the original error's extra fields were lost.
By returning an Error from validateEither, we can now pass through Api
errors from useAsync and thus use them directly in kibana utilities like
toasts.addError.
* WIP: implements checking for and consequent creation of lists index
This adds most of the machinery that I think we're going to need. Not
featured here:
* lists privileges (stubbed out currently)
* handling when lists is disabled
* tests
* Add frontend plugin for lists
We need this to deteremine in security_solution whether lists is enabled
or not. There's no other functionality here, just boilerplate.
* Fix cross-plugin imports/exports
Now that lists has a client plugin, the optimizer cares about code
coming into and out of it.
By default, you cannot import another plugin's common/ folder into your
own common/ nor public/ folders. This is fixed by adding 'common' to
extraPublicDirs, however: extraPublicDirs need to resolve to modules.
Rather than adding each folder from which we export modules to
extraPublicDirs, I've added common/index.ts and exporting everything
through there.
By convention, I'm adding shared_exports.ts as an index of these exported modules,
and shared_imports.ts is used to import on the other end.
For now, I've left the ad hoc _deps files so as to limit the changes
here, but we should come back through and remove them at some point. NB
that I did remove lists_common_deps as it was only used in one or two
spots.
* Fix test failing due to lack of context
This component now uses useKibana indirectly through useListsConfig.
* Lists and securitySolution require each other's bundles
Without lists being a requiredBundle of securitySolution, we cannot
import its code when the plugin is disabled. The opposite is also true,
but there's no lists "app" to break.
* Fix logic in useListsConfig
Lists needs configuration if the index explicitly does not exist. If it
is true (already exists) or null (lists is disabled or we could not read
the index), we're good.
* useList* behavior when lists plugin is disabled
When the lists plugin is disabled, our calls in useListsIndex become no-ops so that:
* useListsIndex state does not change
* useListsConfig.needsConfiguration remains false as indexExists is
never non-null
This also removes use of our `useIsMounted` hook. Since the effects
we're consuming come from useAsync hooks, state will (already) not be
updated if the component is unmounted.
* Fix warning due to dynamic creation of a styled component
* Revert "Fix warning due to dynamic creation of a styled component"
This reverts commit 7124a8fbd9.
(This was already fixed on master)
* Check user's lists index privileges when determining configuration status
If there is no lists index and the user cannot create it, we will
display a configuration message in lieu of Detections
* Adds a lists hook to read privileges (missing schemae)
* Adds security hook useListsPrivileges to perform and parse the
privileges request
* Updates useListsConfig to use useListsPrivileges hook
* Move lists hooks to their own subfolder
* Redirect to main detections page if lists needs configuration
If:
* lists are enabled, and
* lists indexes DNE, and
* user cannot manage the lists indexes
Then they will be redirected to the main detections page where they'll
be instructed to configure detections. If any of the above is false,
things work as normal.
* Lock out of detections when user cannot write to value lists
Rather than add conditional logic to all our UI components dealing with
lists, we're going the heavy-handed route for now.
* Mock lists config hook in relevant Detections page tests
* Disable Detections when Lists is enabled
This refactors useListsConfig.needsConfiguration to mean:
* lists plugin is disabled, OR
* lists indexes DNE and can't be created, OR,
* user can't write to the lists index
In any of these situations, we want to disable detections, and so we
export that as a single boolean, needsConfiguration.
* Remove unneeded complexity exception
We refactored this to work 👍
* Remove outdated TODO
We link to our documentation, which will describe the lists aspects of
configuration.
## Summary
* Adds "wait_for" to all the create, update, patch, and delete endpoints
* Ran some quick tests against import and the performance still looks acceptable
* Updates the unit tests to reflect the addition
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
* Changes the value list imports to use a streaming in model
* Adds a custom light hand spun multi-part parser for the incoming text
* Adds a buffer pause and resume which continues to buffer the incoming data if an async event such as creating a list from the attachment file needs to happen but does not emit the lines until the resume continues.
* Adds a data slicing if the buffer becomes larger than the maximum so that if we begin buffering too quickly within memory we don't blow up the limit of Elastic Search.
* Adds unit tests
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* adds 2 menu items to alert page, progress on exception modal
* adds enriching
* remove unused useExceptionList()
* implements some types
* move add exception modal files
* Exception builder changes to support latest schema
* Changes to lists plugin schemas and fix api bug
Needed to make the schemas more forgiving. Before this change they required name,
description, etc for creation and update.
The update item API was using the wrong url.
* Adding and editing exceptions working
- Modifies add_exception_modal component
- Creates edit_exception_modal component
- Creates shared comments component
- Creates use_add_exception api hook for adding or editing exceptions
- Updates viewer code to support adding and editing exceptions
- Updates alerts table code to use updated version of add_exception_modal
* fixes duplicate types
* updates os tag input
* fixes comment style
* removes checkbox programatically
* grahpql updates to expose exceptions_list
* Add fetch_or_create_exception_list hook
* fixes data population
* refactor use_add_exception hook, add tests
* fix rebase issues, pending updates to edit modal
* fix edit modal and default endpoint exceptions
* adds second checkbox
* adds signal index stuff
* switches boolean logic
* fix some type errors
* remove unnecesary code
* fixes checkbox logic in edit modal
* fixes recursive prop passing
* addresses comments/fixes types
* Revert schema type changes
* type fixes
* fixes regular exception modal
* fix more type errors, remove console log
* fix tests
* move add exception hook, lint
* close alert checkbox closes alert
* address PR comments
* add type to patch rule call, fix ts errors
* fix lint
* fix merge problems after conflict
* Address PR comments
* undo graphql type change
Co-authored-by: Davis Plumlee <davis.plumlee@elastic.co>
## Summary
Adds these data types to the value based lists end points from [Elasticsearch field data types](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html):
Single value based list types:
* binary
* boolean
* byte
* date
* date_nanos
* date_range
* double
* float
* integer
* ip
* half_float
* keyword
* text
* long
* short
Range value based list types:
* double_range
* float_range
* integer_range
* ip_range
* long_range
Geo value based list types: (caveat is that you cannot query them using other geometry just yet ... you can only these and export them)
* geo_point
* geo_shape
* shape
For importing and exporting different values such as ranges, geo, or single values, this introduces a serialize and deserialize option for the endpoints.
For example if you want to serialize in an ip_range such as 192.168.0.1,192.168.0.3 which has a comma between the two would use the following:
```ts
POST /api/lists
{
"name": "List with an ip range",
"serializer": "(?<gte>.+),(?<lte>.+)",
"deserializer": "{{gte}},{{lte}}",
"description": "This list has ip ranges",
"type": "date_range"
}
```
If you want to serialize in keywords from a list that _only_ match a particular value you would use the following:
```ts
POST /api/lists
{
"id": "keyword_custom_format_list",
"name": "Simple list with a keyword using a custom format",
"description": "This parses the first found ipv4 only",
"serializer": "(?<value>((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))",
"deserializer": "{{value}}",
"type": "keyword"
}
```
The serializer is a [named capturing group](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/match) while the deserializer is using [MustacheJS](https://github.com/janl/mustache.js/). The range type, single value types, and geo types all have default captures for their serialize and default mustache templates if none are configured with an endpoint.
The default capture groups and mustache handles for each are:
* shape, geo_point, geo_shape: `(?<lat>.+),(?<lon>.+)`
* date_range: `(?<gte>.+),(?<lte>.+)|(?<value>.+)`
* other ranges are: `(?<gte>.+)-(?<lte>.+)|(?<value>.+)`
* all single data types: `(?<value>.+)`
For ranges you can use both `gte, lte`, and `value` together. If `gte` _and_ `lte` matches it will use that for the greater than, less than elastic range and ignore `value` even if `value` also matched. If _only_ `value` matches and `gte`, `lte` does not match then it will use `value` and put `value` as _both_ the `gte`, and `lte`.
For example, if you are serializing in a list of ip ranges as the list data type, `ip_range` and you have these 3 entries in the file:
```ts
127.0.0.1
127.0.0.2-5
```
The default `serializer` will use `(?<gte>.+)-(?<lte>.+)|(?<value>.+)` and you will get two elastic documents like so:
```ts
{
"_source" : {
"ip_range" : {
"gte" : "127.0.0.1",
"lte" : "127.0.0.1"
}
}
{
"_source" : {
"ip_range" : {
"gte" : "127.0.0.2",
"lte" : "127.0.0.5"
}
}
```
The default mustache handles for each are:
* shape, geo_point, geo_shape: `{{{lat}}},{{{lon}}}`
* date_range: `{{{gte}}},{{{lte}}}`
* other ranges are: `{{{gte}}}-{{{lte}}}`
* all values are: `{{{value}}}`
I use three instead of two handle bars (`{{{` vs.` {{`) so that HTML is not escaped for the lists. You can override and change it if you need or want the escaping.
If during the deserializer phase it detects that a `gte` and `lte` are exactly the same it will still output them as a two items and use the mustache deserialize value. Using the ip-range example above that will be outputted like so since it detects that the lte-gte are exactly the same value:
```ts
127.0.0.1-127.0.0.1
127.0.0.2-127.0.0.5
```
---
Interesting queries to run from the lists scripts folder for testing:
Load some small test files from `./lists/files` for example:
```ts
./import_list_items_by_filename.sh ip_range ./lists/files/ip_range_cidr.txt
./import_list_items_by_filename.sh ip_range ./lists/files/ip_range.txt
./import_list_items_by_filename.sh date ./lists/files/date.txt
./import_list_items_by_filename.sh ip_range ./lists/files/ip_range_mixed.txt
...
```
Export them
```ts
./export_list_items.sh ip_range_cidr.txt
./export_list_items.sh ip_range.txt
./export_list_items.sh date.txt
./export_list_items.sh ip_range_mixed.txt
...
```
Find on them
```ts
./find_list_items.sh ip_range_cidr.txt
./find_list_items.sh ip_range.txt
./find_list_items.sh date.txt
./find_list_items.sh ip_range_mixed.txt
...
```
Find specific values such as:
```ts
./get_list_item_by_value.sh ip_range_mixed.txt 192.168.0.1
./get_list_item_by_value.sh date.txt 2020-08-25T17:57:01.978Z
...
```
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
* Removes the feature flag and turns on lists by default
* Applies to both exception lists and value lists
* Removes all scary messages about having it enabled
* Updates the unit tests to work with it on
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
This PR tries to start to tie together the server and client changes for exceptions lists.
- Updates graphql types to allow UI access to a rule's `exceptions_list` property
- Updates the exception viewer component to now dynamically take the rule `exceptions_list`, up until now we just had an empty array in it's place
- Updates the viewer logic to check if a rule has an endpoint list associated with it. If it does, then it displays both detections and endpoint UIs (in the viewer), if it does not, then it only displays the detections UI
- Updates the viewer UI to better deal with spacing when an exception list item only has one or two entries (before the and badge with the antennas was stretching passed the exception items to fill the space)
- Updates the detections engine exceptions logic to fetch list items using an exception list's `id` as opposed to it's `list_id`, this now aligns with the UI using the same params on its end
- Adds exception list `type` to information kept by the rule for exception lists
- Updates the exception list type from `string` to `endpoint | detection`
- Updates the exception list _item_ type from `string` to `simple`
- Adds unit tests for the detection engine server side util that fetches the exception list items
* stub out task for the exceptions list packager
* Hits list code and pages
* refactor
* Begin adding saved object and type definitions
* Transforms to endpoint exceptions
* Get internal SO client
* update messaging
* cleanup
* Integrating with task manager
* Integrated with task manager properly
* Begin adding schemas
* Add multiple OS and schema version support
* filter by OS
* Fixing sort
* Move to security_solutions
* siem -> securitySolution
* Progress on downloads, cleanup
* Add config, update artifact creation, add TODOs
* Fixing buffer serialization problem
* Adding cleanup to task
* Handle HEAD req
* proper header
* More robust task management
* single -> agnostic
* Fix OS filtering
* Scaffolding digital signatures / tests
* Adds rotue for creating endpoint user
* Cleanup
* persisting user
* Adding route to fetch created user
* Addings tests for translating exceptions
* Adding test for download API
* Download tweaks + artifact generation fixes
* reorganize
* fix imports
* Fixing test
* Changes id of SO
* integration tests setup
* Add first integration tests
* Cache layer
* more schema validation
* Set up for manifest update
* minor change
* remove setup code
* add manifest schema
* refactoring
* manifest rewrite (partial)
* finish scaffolding new manifest logic
* syntax errors
* more refactoring
* Move to endpoint directory
* minor cleanup
* clean up old artifacts
* Use diff appropriately
* Fix download
* schedule task on interval
* Split up into client/manager
* more mocks
* config interval
* Fixing download tests and adding cache tests
* lint
* mo money, mo progress
* Converting to io-ts
* More tests and mocks
* even more tests and mocks
* Merging both refactors
* Adding more tests for the convertion layer
* fix conflicts
* Adding lzma types
* Bug fixes
* lint
* resolve some type errors
* Adding back in cache
* Fixing download test
* Changing cache to be sized
* Fix manifest manager initialization
* Hook up datasource service
* Fix download tests
* Incremental progress
* Adds integration with ingest manager for auth
* Update test fixture
* Add manifest dispatch
* Refactoring to use the same SO Client from ingest
* bug fixes
* build renovate config
* Fix endpoint_app_context_services tests
* Only index the fields that are necessary for searching
* Integ test progress
* mock and test city
* Add task tests
* Tests for artifact_client and manifest_client
* Add manifest_manager tests
* minor refactor
* Finish manifest_manager tests
* Type errors
* Update integ test
* Type errors, final cleanup
* Fix integration test and add test for invalid api key
* minor fixup
* Remove compression
* Update task interval
* Removing .text suffix from translated list
* Fixes hashes for unit tests
* clean up yarn.lock
* Remove lzma-native from package.json
* missed updating one of the tests
Co-authored-by: Alex Kahan <alexander.kahan@elastic.co>
### Summary
This PR creates the bulk functionality of the exception builder. The exception builder is the component that will be used to create exception list items. It does not deal with the actual API creation/deletion/update of exceptions, it does contain an `onChange` handler that can be used to access the exceptions. The builder is able to:
- accept `ExceptionListItem` and render them correctly
- allow user to add exception list item and exception list item entries
- accept an `indexPattern` and use it to fetch relevant field and autocomplete field values
- disable `Or` button if user is only allowed to edit/add to exception list item (not add additional exception list items)
- displays `Add new exception` button if no exception items exist
- An exception item can be created without entries, the `add new exception` button will show in the case that an exception list contains exception list item(s) with an empty `entries` array (as long as there is one exception list item with an item in `entries`, button does not show)
- debounces field value autocomplete searches
- bubble up exceptions to parent component, stripping out any empty entries
* Add wrapper function to make an AbortSignal arg optional
Components commonly do not care about aborting a request, but are
required to pass `{ signal: new AbortController().signal }` anyway. This
addresses that use case.
* Adds hook for retrieving the component's mount status
This is useful for dealing with asynchronous tasks that may complete
after the invoking component has been unmounted. Using this hook,
callbacks can determine whether they're currently unmounted, i.e.
whether it's safe to set state or not.
* Add our own implemetation of useAsync
This does not suffer from the Typescript issues that the react-use
implementation had, and is generally a cleaner hook than useAsyncTask as
it makes no assumptions about the underlying function.
* Update exported Lists API hooks to use useAsync and withOptionalSignal
Removes the now-unused useAsyncTask as well.
* Add some JSDoc for our new functions
* Add pure API functions and react hooks for value list APIs
This also adds a generic hook, useAsyncTask, that wraps an async
function to provide basic utilities:
* loading state
* error state
* abort/cancel function
* Fix type errors in hook tests
These were not caught locally as I was accidentally running typescript
without the full project.
* Document current limitations of useAsyncTask
* Defines a new validation function that returns an Either instead of a tuple
This allows callers to further leverage fp-ts functions as needed.
* Remove duplicated copyright comment
* WIP: Perform request/response validations in the FP style
* leverages new validateEither fn which returns an Either
* constructs a pipeline that:
* validates the payload
* performs the API call
* validates the response
and short-circuits if any of those produce a Left value.
It then converts the Either into a promise that either rejects with the
Left or resolves with the Right.
* Adds helper function to convert a TaskEither back to a Promise
This cleans up our validation pipeline considerably.
* Adds request/response validations to findLists
* refactors private API functions to accept the encoded request schema
(i.e. snake cased)
* refactors validateEither to use `schema.validate` instead of
`schema.decode` since we don't actually want the decoded value, we just
want to verify that it'll be able to be decoded on the backend.
* Refactor our API types
* Add request/response validation to import/export functions
* Fix type errors
* Continue to export decoded types without a qualifier
* pull types used by hooks from their new location
* Fix errors with usage of act()
* Attempting to reduce plugin bundle size
By pulling from the module directly instead of an index, we can
hopefully narrow down our dependencies until tree-shaking does this for
us.
* useAsyncFn's initiator does not return a promise
Rather than returning a promise and requiring the caller to handle a
rejection, we instead return nothing and require the user to watch the
hook's state.
* success can be handled with a useEffect on state.result
* errors can be handled with a useEffect on state.error
* Fix failing test
Assertion count wasn't updated following interface changes; we've now
got two inline expectations so this isn't needed.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* move last snapshot to inline
* move legacy files to legacy subfolder
* move request types out of legacy
* export Headers from http instead of elasticsearch
* renaming - first pass
* renaming - second pass
* fix core mocks
* adapt new calls
* update generated doc
* fix IT test mocks
* fix new usages
### Summary
This PR is a follow up to #68864 . That PR used a partial to differentiate between new and existing comments, this meant that comments could be updated when they shouldn't. It was decided in our discussion of exception list schemas that comments should be append only. This PR assures that's the case, but also leaves it open to editing comments (via API). It checks to make sure that users can only update their own comments.
* Updates list entry schema, exposes exception list client, updates tests
* create new de list schema and unit tests
* updated route unit tests and types to match new list schema
* updated existing DE exceptions code so it should now work as is with updated schema
* test and types cleanup
* cleanup
* update unit test
* updates per feedback
### Summary
This PR is meant to update the `ExceptionListItemSchema.entries` structure to align with the most recent conversations regarding the need for a more explicit depiction of `nested` fields. To summarize:
- Adds schema validation for requests and responses within `lists/public/exceptions/api.ts`. It was super helpful in catching existing bugs. Anyone that uses the api will run through this validation. If the client tries to send up a malformed request, the request will not be made and an error returned. If the request is successful, but somehow the response is malformed, an error is returned. There may be some UX things to figure out about how to best communicate these errors to the user, or if surfacing the raw error is fine.
- Updates `entries` structure in lists plugin api
- Updates hooks and tests within `lists/public` that make reference to new structure
- Updates and adds unit tests for updated schemas
- Removes unused temporary types in `security_solution/public/common/components/exceptions/` to now reference updated schema
- Updates UI tests
- Updates `lists/server/scripts`
## Summary
Fixes io-ts formatter bugs for REST and validation by:
* First trying to get the correct key from the io-ts context. If no keys are found, then it will fall back on trying to get the first name from the context.
* If the key is a value and an object then this will do a `JSON.stringify()` on the value object
* This fixes a few places where `formatError` was not being used within the code base resulting in `[object Object]` within the validations to show up.
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
### Summary
This is part of a series of upcoming changes to the exception list item structure. This PR focuses solely on updating exception_item.comment. The hope is to keep these PRs relatively small.
- Updates exception_item.comment structure which was previously a string to exception_item.comments which is an array of { comment: string; created_by: string; created_at: string; }
- Adds a few unit tests server side
- Fixes some minor misspellings
- Updates ExceptionViewer component in the UI to account for new structure
* add babel support for export type
* bump ts version to 3.9.3
* rebuild kbn-pm
* bump typescript-eslint
* fix error in security plugin UI
* check export as works
* fix app migration type
* use correct test subj attribute
* fix errors from the old PR
* embeddable is already passed in props
* explicitly define type of fetch
* add some types for viz
* fix fetch type p.2
* add null to allow spreading without type errors due to override
* add type guard to fix type error
* cast to any, since cannot assign unknown
* add timestamp to known types
* fix type error in fetch
* fix type error. id is always defined in attibutes
* declare a type
* move ts-ignore to the lines with errors
* declare tuple type explicitly
* mute type error. cannot assign unknown
* fix errors. id is always defined
* fix error type
* fix override errors. id is always defined
* fix error. extends any doesn't work anymore
* fix type error. type is always defined
* env doesn't always contain values
* fix type error
* cast to string
* add: logs is already declared in getNodeLogsUrl
* state is already passed in props
* fix some errors in timelion
* number of fragments is always defined
* 'absolute' is not just string, but value
* TEMP: option is always defined
* always true if cast to promise manually
* both props are always defined
* explicitly define returned SO type
* workaround type
* bump tslib to be compatible with ts v3.9
* test private property
* rebuild kbn-pm
* Fix ts errors for beats management
* Fix type inference broken by the TS 3.9 upgrade
* Fix ingest manager saved object attributes typings
* Fix TS errors in cross_cluster_replication and index_management.
* Fix TS error in Watcher.
* roll back colorRange wrong type
* fix security plugin types
* TypeScript 3.9 fixes for APM
* Fix ColorRange types.
* fix actions & alerts errors. ByGidi
* fix lists error
* More APM fixes
* Remove paramaterization from `removeEmpty in agent config SettingsPage component (it's only used there and doesn't need to be parameterized.)
* Add option chain for case in registerTransactionDurationAlertType
* Cast `overallValue` in transform_metrics_chart
* Use more specific type for custom link filters
* Add more option chaining for local UI filters buckets response
* Remove unused parameters from routes
* Fix getProjection type parameter
* Use destructuring in serviceNodesLocalFiltersRoute to hide `never` error
* Revert `UnionToIntersection` change in `AggregationResponseMap`
Fixes#67804.
* fix platform type error
* Fix visualizations types.
* Fix data plugin types.
* bump TS version to 3.9.5
* Fix telemetry TS errors
* Fix dashboard code
* Adding Canvas Fixes for TS 3.9
* Fix case and security_solution types
* roll back to the old export syntax. new one might cause problems in api-extractor
* update docs
* Fix timelion code
* Fix meta
* Fix types
* fix type errors om ingest_manager
* bump babel deps
* enable private props & methods syntax
* update kbn-pm dist
* whitelist 0BSD license
* use @babel/plugin-proposal-private-methods in default set as well
* disable new babel plugins
* Revert "disable new babel plugins"
This reverts commit 04d959431d.
* cleanup security_solution types
* Fixes type error for newer TypeScript
* update docs
Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
Co-authored-by: Felix Stürmer <stuermer@weltenwort.de>
Co-authored-by: CJ Cenizal <cj@cenizal.com>
Co-authored-by: Larry Gregory <larry.gregory@elastic.co>
Co-authored-by: Nathan L Smith <smith@nlsmith.com>
Co-authored-by: Walter Rafelsberger <walter@elastic.co>
Co-authored-by: Luke Elmers <luke.elmers@elastic.co>
Co-authored-by: Alejandro Fernández Haro <alejandro.haro@elastic.co>
Co-authored-by: Tim Roes <tim.roes@elastic.co>
Co-authored-by: Clint Andrew Hall <clint.hall@elastic.co>
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
Co-authored-by: FrankHassanabad <frank.hassanabad@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
### Summary
- Adds missing unit tests for relevant files missing them
- Changes filter search to fire request on 'Enter'
- Breaks out the main ExceptionViewer component into smaller components to make more readable and better tested
- Updates utility bar to have the specific list description text next to it as proposed by @spong in #68294 (comment)
- Adds loading state any time async request occurs
- Now fetches list on list type toggle (if user selects to view either only detections or endpoint items), before was simply filtering already fetched items
## Summary
* Smaller follow ups and bug fixes from: https://github.com/elastic/kibana/pull/68127
* Added unknown to `findDifferencesRecursive`
* Added linter rule to catch NodeJS code in the common folders for both `lists` and `security_solution`
* Removed the Hapi server type from the common folder of lists
### Checklist
* Added unknown to the correct locations
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
### Summary
This PR is a follow up to #68027 . It brings it all together to complete the exceptions viewer component. This component is meant to display all exception items and allow a user to create, edit, delete, and search these exception items.
- Moves ExceptionItem (from part 1) into its own folder
- Adds exceptions_viewer_header component that includes the search, list toggle, and add exception buttons
- Adds actual ExceptionViewer component
- Updates the useExceptionList hook refresh function logic. Noticed that the previous version was creating some issues
## Summary
* https://github.com/elastic/siem-team/issues/646
* Converts the detection rules and REST to use io-ts
* Removes their joi counterparts
* Updates all tests to use it
* Fixes a bug with the risk_score that was being sent in as a string from the UI instead of a number
* Fixes a bug within the exactCheck validating where it can now accept null value types for optional body messages.
* Fixes a bug in the FindRoute where it did not send down fields from REST
* Changes the lists plugin to utilize the io-ts types from siem rather than having them duplicated.
* Makes some stronger validations
* Adds a lot of codecs
**Things to look out for:**
* Generic testing to ensure I didn't break something that was not part of the tests.
* Fix for the risk_score from string to number is in:
```
x-pack/plugins/security_solution/public/alerts/components/rules/step_about_rule/index.test.tsx
```
* Fix for the exact check (unit tests are written and added)
```
x-pack/plugins/security_solution/public/alerts/components/rules/step_about_rule/index.test.tsx
```
* Within all the types I added are there any misspelled things or copy-pasta mistakes with strings:
x-pack/plugins/security_solution/common/detection_engine/schemas/types
* Fix for `find_rules_route.ts:58`
```
x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/find_rules_route.ts
```
**Follow on things that this PR doesn't do we need to:**
* Add linter rule to forbid NodeJS code within common section
* The `[object Object]` formatter issues seen in the code such as:
```
// TODO: Fix/Change the formatErrors to be better able to handle objects
'Invalid value "[object Object]" supplied to "note"',
```
* Formatter issues such as: `'Invalid value "" supplied to ""'`
* Remove the hapi server object from lists plugin
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* rename siem to security_solution
* rename siem to security solution inside of code
* rename translation keys
* fix snapshot
* replace siem for security solution in tutorial
* missing translation to be renamed
* fix types for api test integration
* updates runner file to match the new path
* change category for kibana settings
* miss renaming in advance settings
* fixes cypress tests
* fix api integration test
* fix new translation
* fix unit test
* update translation i18n
* update translation i18n II
Co-authored-by: Gloria Hornero <snootchie.boochies@gmail.com>
### Summary
- Updates exception list hooks to include filtering options and updates corresponding unit tests.
- Adds refreshList callback to hook that fetches the list and its items
- Updates hooks tests to test onError callback
- Updates tests to use type checking more effectively per feedback from @FrankHassanabad (thanks!)
* introduce lists plugin for use by executor
* adds getListClient function on setup
* refactors searchAfterBulkCreate to integrate with the lists plugin so we only generate signals from events not in the list
* fixes type check issues
* fixes unit tests, adds field and other parameters for using lists in executor.
* cleaning up types and exports, updates to match new contracts with lists client from master
* prior to this commit the refactored while loop was doing more search after loops than it needed to and this fixes two bugs in the list filter function where we were returning the wrong count, and we were not accessing the right field on the event
* exception lists are optional
* use exceptions list format, this works with given sample query in scripts
* updates tests and fixes type issues
* updates README doc in detection engine with example for rule with list exception
* adds one test and removes commented out code
* fix sample rule json from 30s to 5m
* fix sample rule json from 30s to 5m
* remove unused import
* more cleanup
* e2e test for prepackaged rules was failing because lists was undefined in the siem plugin and was preventing the registration of the rule alert type. I removed this but once lists is ready for prime time we should consider adding the null check back
* can't reuse the same env var since the tests are setting the ELASTIC_XPACK_SIEM_LISTS_FEATURE env var to true without enabling the lists plugin
* fixes from pr review, still needs more TLC
* exports listspluginsetup type from top-level in lists plugin, fixes logic for empty exceptions list, updates types
* utilize type.is to remove as casting, also do null checks and throw an error when exceptionItem is malformed. This will change in the very near future once the new json format for exception lists is incorporated
* fix type issues after merging master into branch
* update mock
* remove bad null check for ml plugin before registering rule alert type in siem plugin
* prettier linting
* adds test for filter events with list
* pr comments
* adds logic for included vs excluded and updates tests
* update test cases for search after bulk create to default to included for exception lists
* filter out non-list exception items from the loop
## Summary
Adds the REST and API routes for find and filter for exception lists and value lists
* Fixes bugs with string parameters for the _find with exception lists
* Adds the _find for the value based lists
* More scripts for how to filter things for both list values and exception lists
* Misc type script fixes
* Adds a cursor to move from the previous page to the next page
* Adds name space 'agnostic' vs. 'single' feature for exception_lists
**REST API's:**
```ts
POST /api/lists/_find
POST /api/lists/items/_find
POST /api/exception_lists/_find
POST /api/exception_lists/items/_find
```
**Parameters you can send:**
* sort
* sort_order
* filter
* page
* per_page
* list_id (for list items only and required)
* cursor (for finding the next page or advancing to deep pages)
**See test scripts below:**
```sh
find_exception_list_items_by_filter.sh
find_exception_lists_by_filter.sh
find_list_items.sh
find_list_items_with_cursor.sh
find_list_items_with_sort.sh
find_list_items_with_sort_cursor.sh
find_lists.sh
find_lists_with_cursor.sh
find_lists_with_filter.sh
find_lists_with_sort.sh
find_lists_with_sort_cursor.sh
```
### Checklist
Note: Unit tests are left out as this is blocking people but I will be adding tests as this is being reviewed unless someone needs these features now. This is still all behind a feature flag and considered to be in the area of proof of concept and not production ready until more tests and end to tests are added.
- [ ] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
Added some basic functionality to help exception list UI work move forward. Wired up to exception list api and created hooks. This PR includes:
- UI api functions for basic exception list and exception list item CRUD
- useExceptionList hook to fetch the list and its items
- usePersistExceptionList hook to create or update an exception list
- usePersistExceptionListItem hook to create or update an exception item
- list_plugin_deps.tsx in the siem folder to import the lists plugin hooks
## Summary
See for more details:
https://github.com/elastic/kibana/issues/65938
Adds pieces of the `exception list` and `exception list item` and refactors/cleans the code up where I had parts incorrect with little things such as the javascript library io-ts. Some unit tests were added but I am holding off until more of the operations solidify before adding the unit tests. Everything is still behind a feature flag that must be enabled and not advised still at this point to use so I feel ok pushing these parts forward.
Adds to the API:
- Create exception list
- Read exception list
- Update exception list
- Delete exception list (and exception list items that are associated with it)
- Create exception list item
- Find exception list (/_find)
- Read exception list item
- Update exception list item
- Delete exception list items individually
- Find exception list item (/_find)
What is still missing from the REST and client API?
- Patch exception list
- Patch exception list item
- Bulk versions of everything
- Import/Export options for these exception lists and list items
### Manual testing and REST API endpoints
Go here:
```sh
/projects/kibana/x-pack/plugins/lists/server/scripts
```
See the files:
```sh
delete_all_exception_lists.sh
delete_exception_list.sh
delete_exception_list_by_id.sh
delete_exception_list_item.sh
delete_exception_list_item_by_id.sh
exception_lists
find_exception_list_items.sh
find_exception_lists.sh
get_exception_list.sh
get_exception_list_by_id.sh
get_exception_list_item.sh
get_exception_list_item_by_id.sh
post_exception_list.sh
post_exception_list_item.sh
update_exception_list.sh
update_exception_list_item.sh
```
Ensure you first run:
```sh
./hard_reset
```
and ensure you have setup your kibana.dev.yml to have:
```yml
# Enable lists feature
xpack.lists.enabled: true
xpack.lists.listIndex: '.lists-frank'
xpack.lists.listItemIndex: '.items-frank'
```
Then you can use the above scripts to create, read, update, and delete exception list and exception list items as well as perform find commands against them all.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
Note: Some but limited unit tests at this point.
* line restricted zones for export exressions
* more robust rule
* fix or mute eslint errors
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Fixes up the API contracts to work outside of a request and as a regular plugin.
* Removes space and request stuff that is not needed
* Adds in plugin ability with space id and user name being pushed down
* Adds unit tests to the schema for input/output validation
* Changes the mocks to use a `file_name.mock.ts` pattern
* Introduces io-ts partials _carefully_ where I get both the partials and the required undefined in the types
* Introduces an Identity type to remove weird intersection types and make plain types when using io-ts.
* I Introduces a RequiredKeepUndefined in order to work with partials and keep the undefined as required for when the type is used directly within the code. This makes it simpler to force new functions/methods to have to push down `undefined`
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* Removes plugin dependencies for better integration outside of Requests such as alerting
* Adds more unit tests
* Fixes more TypeScript types to be more normalized
* Makes this work with the user 'elastic' if security is turned off
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
* Adds large list support using REST endpoints.
Status:
---
* Currently ready to be merged behind the feature flag of it being disabled with ongoing work happening after it is merged.
* REST Endpoints shouldn't have large refactoring at this point
* Team meeting occurred where the pieces were discussed in person.
What is left?
---
- [ ] Add other data types. At the moment `ip` and `keyword` are the two types of lists. See: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html
- [x] Unit tests
- [x] Lots of misc TODO's in the code base still
- [ ] Import loads everything into memory first when it should attempt streaming
- [ ] Add end to end backend tests
- [x] Add transform and io-ts validation for returns
Testing
---
Ensure you set this in your ENV before starting Kibana:
```ts
export ELASTIC_XPACK_SIEM_LISTS_FEATURE=true
```
Download or create a large list file such as this one filled with IP's:
https://cinsscore.com/list/ci-badguys.txt
Go to your REST endpoint folder of scripts:
```ts
cd kibana/x-pack/plugins/lists/server/scripts
```
Do a hard reset:
```ts
./hard_reset
```
Then import it as either a data type of `ip`:
```ts
./import_list_items_by_filename.sh ip ~/Downloads/ci-badguys-smaller.txt
```
Or as a `keyword`
```ts
./import_list_items_by_filename.sh keyword ~/Downloads/ci-badguys-smaller.txt
```
Then you can export it through:
```ts
./export_list_items.sh ci-badgusy-smaller.txt
```
For all the other endpoints and testing of the CRUD operations you have access to:
```ts
delete_all_lists.sh
delete_list.sh
delete_list_index.sh
delete_list_item.sh
delete_list_item_by_id.sh
delete_list_item_by_value.sh
export_list_items.sh
export_list_items_to_file.sh
get_list.sh
get_list_item_by_id.sh
get_list_item_by_value.sh
import_list_items.sh
import_list_items_by_filename.sh
lists_index_exists.sh
patch_list.sh
patch_list_item.sh
post_list.sh
post_list_index.sh
post_list_item.sh
```
### Checklist
Delete any items that are not applicable to this PR.
- [ ] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)
