### Summary
Addresses https://github.com/elastic/kibana/issues/124742
#### Issue TLDR
Import of rules that reference exception items with comments fail. Failure message states that comments cannot include `created_at`, `created_by`, `id`.
## Summary
Exposes the functionality of
* search_after
* point in time (pit)
From saved objects to the exception lists. This _DOES NOT_ expose these to the REST API just yet. Rather this exposes it at the API level to start with and changes code that had hard limits of 10k and other limited loops. I use the batching of 1k for this at a time as I thought that would be a decent batch guess and I see other parts of the code changed to it. It's easy to change the 1k if we find we need to throttle back more as we get feedback from others.
See this PR where `PIT` and `search_after` were first introduced: https://github.com/elastic/kibana/pull/89915
See these 2 issues where we should be using more paging and PIT (Point in Time) with search_after: https://github.com/elastic/kibana/issues/93770https://github.com/elastic/kibana/issues/103944
The new methods added to the `exception_list_client.ts` client class are:
* openPointInTime
* closePointInTime
* findExceptionListItemPointInTimeFinder
* findExceptionListPointInTimeFinder
* findExceptionListsItemPointInTimeFinder
* findValueListExceptionListItemsPointInTimeFinder
The areas of functionality that have been changed:
* Exception list exports
* Deletion of lists
* Getting exception list items when generating signals
Note that currently we use our own ways of looping over the saved objects which you can see in the codebase such as this older way below which does work but had a limitation of 10k against saved objects and did not do point in time (PIT)
Older way example (deprecated):
```ts
let page = 1;
let ids: string[] = [];
let foundExceptionListItems = await findExceptionListItem({
filter: undefined,
listId,
namespaceType,
page,
perPage: PER_PAGE,
pit: undefined,
savedObjectsClient,
searchAfter: undefined,
sortField: 'tie_breaker_id',
sortOrder: 'desc',
});
while (foundExceptionListItems != null && foundExceptionListItems.data.length > 0) {
ids = [
...ids,
...foundExceptionListItems.data.map((exceptionListItem) => exceptionListItem.id),
];
page += 1;
foundExceptionListItems = await findExceptionListItem({
filter: undefined,
listId,
namespaceType,
page,
perPage: PER_PAGE,
pit: undefined,
savedObjectsClient,
searchAfter: undefined,
sortField: 'tie_breaker_id',
sortOrder: 'desc',
});
}
return ids;
```
But now that is replaced with this newer way using PIT:
```ts
// Stream the results from the Point In Time (PIT) finder into this array
let ids: string[] = [];
const executeFunctionOnStream = (response: FoundExceptionListItemSchema): void => {
const responseIds = response.data.map((exceptionListItem) => exceptionListItem.id);
ids = [...ids, ...responseIds];
};
await findExceptionListItemPointInTimeFinder({
executeFunctionOnStream,
filter: undefined,
listId,
maxSize: undefined, // NOTE: This is unbounded when it is "undefined"
namespaceType,
perPage: 1_000,
savedObjectsClient,
sortField: 'tie_breaker_id',
sortOrder: 'desc',
});
return ids;
```
We also have areas of code that has perPage listed at 10k or a constant that represents 10k which this removes in most areas (but not all areas):
```ts
const items = await client.findExceptionListsItem({
listId: listIds,
namespaceType: namespaceTypes,
page: 1,
pit: undefined,
perPage: MAX_EXCEPTION_LIST_SIZE, // <--- Really bad to send in 10k per page at a time
searchAfter: undefined,
filter: [],
sortOrder: undefined,
sortField: undefined,
});
```
That is now:
```ts
// Stream the results from the Point In Time (PIT) finder into this array
let items: ExceptionListItemSchema[] = [];
const executeFunctionOnStream = (response: FoundExceptionListItemSchema): void => {
items = [...items, ...response.data];
};
await client.findExceptionListsItemPointInTimeFinder({
executeFunctionOnStream,
listId: listIds,
namespaceType: namespaceTypes,
perPage: 1_000,
filter: [],
maxSize: undefined, // NOTE: This is unbounded when it is "undefined"
sortOrder: undefined,
sortField: undefined,
});
```
Left over areas will be handled in separate PR's because they are in other people's code ownership areas.
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* validation for Pre GET one of host isolation exceptions.
* adjust checks for host isolation validation
* Add validation for import for all artifacts
* Validate host isolation exceptions exports
* Validate host isolation exceptions multi list find
* Validate host isolation exceptions single list find
* Validate host isolation exceptions Summary
* add FTR tests to validate authz
* Update all exception extension point handlers to use the ExceptionListClient passed in on context
* Refactored ExceptionListItemGenerator a bit and added methods to get Host Isolation exceptions
* Update handlers to immediately exit if the namespace_type is not `agnostic`
* Improved `log.info` messages in artifact and policy services
* Add `lists-summary` to Security solution `all` feature privilege (was missing)
* Add an instance of ExceptionListClient with server extension points turned off to the `context` provided to callbacks
* Unit test cases to validate context
* Don't show a default value '-' for emoty descriptions on artifacts list. Also removes empty spaces
* Update copy to say 'event filters' instead of 'exceptions'
* Decrease spacing between avatar and comments textbox
* Adds extra spacing between last exception builder field and the buttons group
* Reduces effect scope togle width to by dynamic depending on translations
* Makes effected policy button group persistent across different artifact forms
* Removes unused import
* Center button group for small devices
* update summary endpoint to use filters and use that for fleet event filters cards
fixes elastic/security-team/issues/2513
* update tests
fixes elastic/security-team/issues/2513
* update host isolation card to show total as the actual number of artifacts
fixes elastic/kibana/issues/121507
* fix types
missing merge updates
* use named constant for isolation exception list
review changes
* Update fleet_integration_event_filters_card.tsx
review changes
* fix the total on summary api
review suggestions
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* add extension point for import by stream
* add extension point for import by array
* Add mocks and tests for import
* adjust tests for import to use `ExceptionListClient#importExceptionListAndItems()`
* Export the additional Extension point Types from server
* Extension point for getting one exception item
* Extension point for single list `find*`
* Extension point for multi list `find*`
* extension point for export exceptions list
* extension point for get summary
* extension point for Delete exception item
## Lists Plugin changes:
- Modified ExceptionListClient to accept an optional KibanaRequest when instantiating a new instance of the class
- Changes the extension points callback argument structure to an object having context and data. Context provides to the callbacks the HTTP request so that additional validation can be performed (ex. Authz to certain features)
- ExtensionPointStorageClient#pipeRun() will now throw if an extension point callback also throws an error (instead of logging it and continuing on with callback execution)
- ErrorWithStatusCode was export'ed out of the server (as ListsErrorWithStatusCode) and available for use by dependent plugins
## Security Solution Plugin (endpoint) changes:
- Added new getEndpointAuthz(request) and getExceptionListsClient() methods to EndpointAppContextService
- Added new server lists integration modules. Registers extension points with the Lists plugin for create and update of exception items. Currently validates only Trusted Apps
- Added exception item artifact validators:
- a BaseValidator with several generic and reusable methods that can be applied to any artifact
- a TrustedAppValidator to specifically validate Trusted Applications
- Refactor:
- moved EndpointFleetServices to its own folder and also renamed it to include the word Factory (will help in the future if we create server-side service clients for working with Endpoint Policies)
- Created common Artifact utilities and const's for working with ExceptionListItemSchema items
* Lists plugin framework for registering extension points
* Support for two extension points for Exceptions List
* `ExceptionListClient` changed to executed extension points
* Security Solution: Change security solution `getExceptionListClient()` to use the Lists plugin factory
## Summary
Without the added overwrite support for exceptions separate from rules, unexpected user behavior experienced. This PR does the following:
- Updates the import rules modal text to account for exceptions
- Updates the import rules modal logic to account for the exceptions overwrite option
- Users can now select to overwrite rules, exceptions or both
- Updates the backend logic in the rules import route to batch checking if the exception lists referenced by the rules trying to be imported exist. If the list does not exist, it removes the reference before trying to import the rule. Previously, this check was being done one by one for each rule.
- Added effort to try to speed up the import after added exceptions logic from original PR slowed down functionality
## Summary
Addresses https://github.com/elastic/kibana/issues/92613 and https://github.com/elastic/kibana/issues/117399
Goal is to allow users to import their exception lists and items alongside their rules. This PR does not complete all the UI updates needed, but does tackle the majority of use cases. The bulk of the changes occur in `import_rules_route` and the new `import_exceptions_route`.
- Adds exceptions import endpoint in `lists` plugin
- Adds exceptions import logic in import rules route in `security_solution` plugin
- Adds integration tests for exception import endpoint
- Adds integration tests for rules import endpoint to account for new functionality
- Purposely not yet adding an import modal in the exceptions table UI until further list management features added (checked with product on this front)
## Summary
See: https://github.com/elastic/kibana/issues/110903
This removes the top level API `export *` spots from:
* `security_solution` plugin
by removing _all_ the exports from `security_solution/common/index.ts` since non of those were shared outside this plugin. Look at the metrics from the build below and you will see _huge_ drops off numbers across the board for required API documentation to the page load size.
In the file `security_solution/common/index.ts` I now put the advice of:
```
// Careful of exporting anything from this file as any file(s) you export here will cause your page bundle size to increase.
// If you're using functions/types/etc... internally it's best to import directly from their paths than expose the functions/types/etc... here.
// You should _only_ expose functions/types/etc... that need to be shared with other plugins here.
```
But really I doubt we will have to share anything from `security_solutions` plugin to another plugin or expose it for anyone else. So I think this is 👍 the way forward to not expose anything directly from `security_solution/common/index.ts` anymore.
## Summary
See: https://github.com/elastic/kibana/issues/110903
This removes the `export *` from:
* lists plugin
This also adds `import type` and `export type` in a few areas and fixes the `LicenseType` by changing it from `server` to using the version from `common` to remove the restricted paths. This extra addition prevents more memory leaks when we run jest.
## Summary
This removes all the areas marked as deprecated from `.../src/plugins/data/public` with their `@kbn/es-query` equivalent or it uses the directly exported version from `.../src/plugins/data/public`. Anywhere else this adds the `import type {` where it can to encourage the build system to do more type erasures.
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
## Summary
This addresses parts of https://github.com/elastic/kibana/issues/117255
By introducing top level mocks for:
* `core/server/index.ts`
* `task_manager/server/index.ts`
* `alerting/server/index.ts`
* `actions/server/index.ts`
These top level mocks add the few required functions we use sparingly and adds them from the "restricted zones" to avoid giant typescript imports from happening from the server side which also pulls in the memory leaks.
```ts
moduleNameMapper: {
'core/server$': '<rootDir>/x-pack/plugins/security_solution/server/__mocks__/core.mock.ts',
'task_manager/server$':
'<rootDir>/x-pack/plugins/security_solution/server/__mocks__/task_manager.mock.ts',
'alerting/server$': '<rootDir>/x-pack/plugins/security_solution/server/__mocks__/alert.mock.ts',
'actions/server$': '<rootDir>/x-pack/plugins/security_solution/server/__mocks__/action.mock.ts',
},
```
For testing this you can now run:
```sh
node --max-old-space-size=600 --expose-gc ./node_modules/.bin/jest --runInBand --logHeapUsage --detectOpenHandles --no-cache --config x-pack/plugins/security_solution/jest.config.dev.js x-pack/plugins/security_solution/server
```
And the server side tests will be able to complete in less than 600 megs of memory. The memory leaks and memory consumption issues are mitigated through the layers but this doesn't guarantee that in the future these won't show up again. The root of the issue(s) with the memory leaks from `core/server` aren't addressed here as those are separate concerns at this point but this at least mitigates the amount of leakage from our side for now.
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* bump to a pre-8.0 version
* export KibanaClient from /lib sub-folder
* workaround the problem of the absence of estypes
* update es client usage in pacakges
* export estypes from another path
* import errors from root
* import errors from root 2
* update transport import
* update import path for /api/types
* update import path for /api/types
* import errors from top export
* use TransportResult instead if ApiResponse
* fix errors in client_config
* fix src/core/server/saved_objects/migrationsv2/actions/integration_tests/actions.test.ts
* use KibanaClient in mock. we dont export the original Client
* fix client mocks
* fix errors on SO
* fix remaining core errors
* update estype import path
* fix errors in data plugin
* fix data_views
* fix es_ui_shared
* fix errors in interactive_setup
* fix errors in ./test folder
* add @elastic/transport to the runtime deps
* fix errors in packages
* fix erros in src/core
* fix errors in test/
* fix an error in actions plugin
* woraround and fix errors in APM plugin
* fix errors in canvas
* fix errors in event_log
* fix errors in fleet
* fix errors in ILM
* fix errors in infra
* fix errors in ingest_pipeline
* fix errors in lens
* fix errors in license_management
* fix errors in licensing
* fix errors in logstash
* fix errors in ml
* fix errors in monitoring
* fix errors in observability
* fix errors in rule_registry
* fix errors in reporting
* fix errors in rule_registry
* fix errors in security
* fix errors in security_solution
* fix errors in snapshot_restore
* fix errors in transform
* fix errors in UA
* fix errors in uptime
* fix errors in x-pack/test
* fix eslint errors
* fix new errors
* use default HTTP Connection. Undici does not support agent config options keepAlive and maxSockets
* create does not accept require_alias option
* update deps
* use transport types exported from ES client package
* fix ErrorCause | string errors
* do not use enum
* fix errors in data plugin
* update x-pack code
* fix transport
* fix apm search request
* do not crash on reporting
* fix kbn-test build
* mute reporting error to start
* fix ftr build
* another attempt
* update import path
* address or mute new errors
* REMOVE me. pin transport version temporarily.
* remove deep imports from transport package
* fix jest crash
* fix product check tests
* remove unnecessary ts-expect-error
* fix a few failed unit tests
* bump to canary 24
* remove unnecessary ts-expect-error
* remove dependency on transport
* fix types in tests
* mute errors in xpack tests
* product check doesn;t spam in logs anymore
* filterPath --> filter_path
* ignoreUnavailable --> ignore_unavailable
* ignoreUnavailable --> ignore_unavailable
* trackScores --> track_scores
* trackTotalHits --> track_total_hits
* fix es-arcives
* fix data plugin crashes
* fix watcher test utils
* rollback unnecessary changes
* fix another problem in es-archiver
* fix scroll. for whatever reason scroll fails when request scroll_id in body
* add meta: true in kbn-securitysolution-es-utils
* bump client to canary 25
* fix errors in accordance with the es client spec
* update securityscolution-es-utils
* unify scroll api in reporting and fix tests
* fix unit tests in watcher
* refactor APM to abort request with AbortController API
* fix missing es client calls in tests
* fix missing meta in detection engine FTR tests
* fix another bunch of errors in js tests
* fix wrong coercion
* remove test-grep pattern
* fix apm unit test
* rename terminateAfter to terminate_after in infra plugin
* rename terminateAfter to terminate_after in uptime plugin
* rename terminateAfter to terminate_after in apm plugin
* fix security roles FTR tests
* fix reference
* fix post_privilidges test
* fix post_privilidges
* bump client to 26
* add meta for index_management test helpers
* remove ts-expect-error caused by bad type in reason
* bump client to 27
* REMOVE me. workaround until fixed in the es client
* fix incorrect type casting
* swtich from camelCase params
* use `HttpConnection` for FTR-related clients
* bump client to 29
* Revert "REMOVE me. workaround until fixed in the es client"
This reverts commit c038850c09.
* fix new util
* revert repository changes
* do not crash if cannot store event_loop data
* fix new estypes imports
* fix more types
* fix security test types and add ts-ignore for custom ES client
* fix more estypes imports
* yet more ts violations
* line by line fixing is hard
* adapt `evaluateAlert` from infra as it's also used from FTR tests
* use convertToKibanaClient in FTR test instead of meta:true in plugin code
* migrate from deprecated API in fleet
* fix intergration tests
* fix fleet tests
* fix another fleet test
* fix more tests
* let's call it a day
* Removes custom header check on 404 responses, includes es client ProductNotSupportedError in EsUnavailableError conditional (#116029)
* Removes custom header check on 404 responses, includes es client ProductNotSupportedError in EsUnavailableError conditional
* Updates proxy response integration test
* disable APM until compatible with client v8
* skip async_search FTR test
* use kbnClient in integration tests
* bump version to 29
* bump to 30
* have configureClient return a KibanaClient instead of Client, remove resolved violations.
* bump to 31
* bump to 31
* Revert "bump to 31"
This reverts commit 5ac713e640.
* trigger stop to unusubscribe
* update generated docs
* remove obsolete test
* put "as" back
* cleanup
* skip test
* remove new type errors in apm package
* remove ErrorCause casting
* update a comment
* bump version to 32
* remove unnecessary ts-expect-error in apm code
* update comments
* update to client v33
* remove outdated type definition
* bump to 34 without params mutation
* unskip the test that should not fail anymore
* remove unnecessary ts-expect-error comments
* update to v35. body can be string
* move `sort` to body and use body friendly syntax
* fix a failing test. maps register the same SO that has been already registered by home
Co-authored-by: pgayvallet <pierre.gayvallet@gmail.com>
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
* Decode fileName when creating a list
* Return wait_for for delete list item
* Return back import
* Update x-pack/plugins/lists/server/services/items/write_lines_to_bulk_list_items.test.ts
Co-authored-by: Ryland Herrick <ryalnd@gmail.com>
* Use i18n for message
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Ryland Herrick <ryalnd@gmail.com>
* [eslint] add rule to prevent export* in plugin index files
* deduplicate export names for types/instances with the same name
* attempt to auto-fix duplicate exports too
* capture exported enums too
* enforce no_export_all for core too
* disable rule by default, allow opting-in for help fixing
* update tests
* reduce yarn.lock duplication
* add rule but no fixes
* disable all existing violations
* update api docs with new line numbers
* revert unnecessary changes to yarn.lock which only had drawbacks
* remove unnecessary eslint-disable
* rework codegen to split type exports and use babel to generate valid code
* check for "export types" deeply
* improve test by using fixtures
* add comments to some helper functions
* disable fix for namespace exports including types
* label all eslint-disable comments with related team-specific issue
* ensure that child exports of `export type` are always tracked as types
Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Update dependency @elastic/elasticsearch to ^8.0.0-canary.15
* update tests for new error message building mechanism
* fix integration tests
* fix functional test
* mute new type errors
* fix new type errors
* bump es client to canaary.16
* fix integration test
* fix type errors in infra plugin
* mute type error in ml plugin
* fix type errors in monitoring plugin
* fix and mute errors in security solution plugin
* bump version to canary.18
* remove an unnecessary change
Co-authored-by: Renovate Bot <bot@renovateapp.com>
Co-authored-by: restrry <restrry@gmail.com>
* [build_ts_refs] improve caches, allow building a subset of projects
* cleanup project def script and update refs in type check script
* rename browser_bazel config to avoid kebab-case
* remove execInProjects() helper
* list references for tsconfig.types.json for api-extractor workload
* disable composite features of tsconfig.types.json for api-extractor
* set declaration: true to avoid weird debug error
* fix jest tests
Co-authored-by: spalger <spalger@users.noreply.github.com>
* Remove legacy imports from 'elasticsearch' package
This prefers the newer types from '@elastic/elasticsearch'.
There was one instance where mock data was insufficient to satisfy the
newer analogous types; in all other cases this was just a find/replace.
* Fix type errors with a null guard
We know that this mock has hits with _source values, but we cannot
convey this to typescript as null assertions are disabled within this
project. This seems like the next best solution, preferable to a
@ts-expect-error.
* Fix a few more type errors
* Replace legacy type imports in integration tests
* refactors destructuring due to _source being properly declared as
conditional
* Update more integration tests to account for our optional _source
Changes here fall into one of two categories:
* If the test was making an assertion on a value from _source, we simply
null chain and continue to assert on a possibly undefined value.
* If the test logic depends on _source being present, we first assert that
presence, and exit the test early if absent.
* Fix more type errors
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
