**Part of: https://github.com/elastic/security-team/issues/6726**
## Summary
Migrates the prebuilt rules and timelines status API route schema to
OpenAPI. This is exploratory work to assess the level of effort required
to migrate API route schemas from `io-ts` to `zod` generated by OpenAPI
codegen.
**Summary of the changes:**
- Added a CI job that runs code generation in Security Solution and
comments change if there are any.
- Migrated the `/api/detection_engine/rules/prepackaged/_status` route
to use generated `zod` schemas
- Updated schema tests
- Adjusted the code generator templates to handle `strict` schemas,
i.e., schemas that do not allow any extra params
- Updated the error transformation code to work with zod errors.
Validation errors are converted to string representations, like the
following:
<img width="627" alt="image"
src="93002573-972f-42e1-901d-01a19937f568">
## Summary
Setting `xpack.alerting.enableFrameworkAlerts` to true by default. This
causes alerts-as-data resource installation to be handled by the
alerting plugin and not the rule registry. We're keeping the feature
flag in case we run into issues but eventually we'll clean up the code
to remove the feature flag and clean up the rule registry code that
relies on the feature flag. Changing this default setting early will
allow us to identify issues before the 8.8 FF where we can revert if
needed.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Dearest Reviewers 👋
I've been working on this branch with @mistic and @tylersmalley and
we're really confident in these changes. Additionally, this changes code
in nearly every package in the repo so we don't plan to wait for reviews
to get in before merging this. If you'd like to have a concern
addressed, please feel free to leave a review, but assuming that nobody
raises a blocker in the next 24 hours we plan to merge this EOD pacific
tomorrow, 12/22.
We'll be paying close attention to any issues this causes after merging
and work on getting those fixed ASAP. 🚀
---
The operations team is not confident that we'll have the time to achieve
what we originally set out to accomplish by moving to Bazel with the
time and resources we have available. We have also bought ourselves some
headroom with improvements to babel-register, optimizer caching, and
typescript project structure.
In order to make sure we deliver packages as quickly as possible (many
teams really want them), with a usable and familiar developer
experience, this PR removes Bazel for building packages in favor of
using the same JIT transpilation we use for plugins.
Additionally, packages now use `kbn_references` (again, just copying the
dx from plugins to packages).
Because of the complex relationships between packages/plugins and in
order to prepare ourselves for automatic dependency detection tools we
plan to use in the future, this PR also introduces a "TS Project Linter"
which will validate that every tsconfig.json file meets a few
requirements:
1. the chain of base config files extended by each config includes
`tsconfig.base.json` and not `tsconfig.json`
1. the `include` config is used, and not `files`
2. the `exclude` config includes `target/**/*`
3. the `outDir` compiler option is specified as `target/types`
1. none of these compiler options are specified: `declaration`,
`declarationMap`, `emitDeclarationOnly`, `skipLibCheck`, `target`,
`paths`
4. all references to other packages/plugins use their pkg id, ie:
```js
// valid
{
"kbn_references": ["@kbn/core"]
}
// not valid
{
"kbn_references": [{ "path": "../../../src/core/tsconfig.json" }]
}
```
5. only packages/plugins which are imported somewhere in the ts code are
listed in `kbn_references`
This linter is not only validating all of the tsconfig.json files, but
it also will fix these config files to deal with just about any
violation that can be produced. Just run `node scripts/ts_project_linter
--fix` locally to apply these fixes, or let CI take care of
automatically fixing things and pushing the changes to your PR.
> **Example:** [`64e93e5`
(#146212)](64e93e5806)
When I merged main into my PR it included a change which removed the
`@kbn/core-injected-metadata-browser` package. After resolving the
conflicts I missed a few tsconfig files which included references to the
now removed package. The TS Project Linter identified that these
references were removed from the code and pushed a change to the PR to
remove them from the tsconfig.json files.
## No bazel? Does that mean no packages??
Nope! We're still doing packages but we're pretty sure now that we won't
be using Bazel to accomplish the 'distributed caching' and 'change-based
tasks' portions of the packages project.
This PR actually makes packages much easier to work with and will be
followed up with the bundling benefits described by the original
packages RFC. Then we'll work on documentation and advocacy for using
packages for any and all new code.
We're pretty confident that implementing distributed caching and
change-based tasks will be necessary in the future, but because of
recent improvements in the repo we think we can live without them for
**at least** a year.
## Wait, there are still BUILD.bazel files in the repo
Yes, there are still three webpack bundles which are built by Bazel: the
`@kbn/ui-shared-deps-npm` DLL, `@kbn/ui-shared-deps-src` externals, and
the `@kbn/monaco` workers. These three webpack bundles are still created
during bootstrap and remotely cached using bazel. The next phase of this
project is to figure out how to get the package bundling features
described in the RFC with the current optimizer, and we expect these
bundles to go away then. Until then any package that is used in those
three bundles still needs to have a BUILD.bazel file so that they can be
referenced by the remaining webpack builds.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* [packages] add kibana.jsonc files
* auto-migrate to kibana.jsonc
* support interactive pkg id selection too
* remove old codeowners entry
* skip codeowners generation when .github/CODEOWNERS doesn't exist
* fall back to format validation if user is offline
* update question style
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* refact(NA): apply root_input_dir=src to each already created pkg
* refact(NA): update package generator
* fix(NA): correctly use rootDir
* fix(NA): use root input dir on latest introduced pkgs for jsts_transpiler macro
* chore(NA): merge with main
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
The lists plugin stores large value lists in two data indices - `.lists-*` and `.items-*`. These were still using the legacy ES template. This PR updates relevant routes to now use the new index templates.
- `createListsIndexRoute` now uses the new template routes and checks for legacy templates to delete them
- `deleteListsIndex` now uses up to date ES API
- Updates the templates to follow new structure
* [type-summarizer] reimplement for broader support
* Enable sourceMaps in all packages
* include naming collision in summarizePackage test
* fix readmes
* remove unnecessary transient dependency
* remove code that was commented out
* remove outdated todo comment
* ensure errors triggered by untyped-exports are ligible
* remove unused import
* break out snippet generation from AstIndexer
* refactor several massive files into smaller pieces and add more inline docs
* fix typos
* update jest snapshots
* add sections to readme that points people to the useful parts of the source code along with a high-level overview of how the type-summarizer works
* remove --dump flag, it doesn't work
* use decName instead of calling names.get a second time
* include `export` as invalid name
* Only apply field aliases to legacy .siem-signals indices
* Fix unit test mocks
* Add new function for special index existence check
* Actually add new function for special index existence check
* Undo getIndexVersion change
* Add basic integration tests for field alias logic
* Add back create_index to test list
* Add missing markdown to readme
* Revert change to delete_index_route
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* bump to a pre-8.0 version
* export KibanaClient from /lib sub-folder
* workaround the problem of the absence of estypes
* update es client usage in pacakges
* export estypes from another path
* import errors from root
* import errors from root 2
* update transport import
* update import path for /api/types
* update import path for /api/types
* import errors from top export
* use TransportResult instead if ApiResponse
* fix errors in client_config
* fix src/core/server/saved_objects/migrationsv2/actions/integration_tests/actions.test.ts
* use KibanaClient in mock. we dont export the original Client
* fix client mocks
* fix errors on SO
* fix remaining core errors
* update estype import path
* fix errors in data plugin
* fix data_views
* fix es_ui_shared
* fix errors in interactive_setup
* fix errors in ./test folder
* add @elastic/transport to the runtime deps
* fix errors in packages
* fix erros in src/core
* fix errors in test/
* fix an error in actions plugin
* woraround and fix errors in APM plugin
* fix errors in canvas
* fix errors in event_log
* fix errors in fleet
* fix errors in ILM
* fix errors in infra
* fix errors in ingest_pipeline
* fix errors in lens
* fix errors in license_management
* fix errors in licensing
* fix errors in logstash
* fix errors in ml
* fix errors in monitoring
* fix errors in observability
* fix errors in rule_registry
* fix errors in reporting
* fix errors in rule_registry
* fix errors in security
* fix errors in security_solution
* fix errors in snapshot_restore
* fix errors in transform
* fix errors in UA
* fix errors in uptime
* fix errors in x-pack/test
* fix eslint errors
* fix new errors
* use default HTTP Connection. Undici does not support agent config options keepAlive and maxSockets
* create does not accept require_alias option
* update deps
* use transport types exported from ES client package
* fix ErrorCause | string errors
* do not use enum
* fix errors in data plugin
* update x-pack code
* fix transport
* fix apm search request
* do not crash on reporting
* fix kbn-test build
* mute reporting error to start
* fix ftr build
* another attempt
* update import path
* address or mute new errors
* REMOVE me. pin transport version temporarily.
* remove deep imports from transport package
* fix jest crash
* fix product check tests
* remove unnecessary ts-expect-error
* fix a few failed unit tests
* bump to canary 24
* remove unnecessary ts-expect-error
* remove dependency on transport
* fix types in tests
* mute errors in xpack tests
* product check doesn;t spam in logs anymore
* filterPath --> filter_path
* ignoreUnavailable --> ignore_unavailable
* ignoreUnavailable --> ignore_unavailable
* trackScores --> track_scores
* trackTotalHits --> track_total_hits
* fix es-arcives
* fix data plugin crashes
* fix watcher test utils
* rollback unnecessary changes
* fix another problem in es-archiver
* fix scroll. for whatever reason scroll fails when request scroll_id in body
* add meta: true in kbn-securitysolution-es-utils
* bump client to canary 25
* fix errors in accordance with the es client spec
* update securityscolution-es-utils
* unify scroll api in reporting and fix tests
* fix unit tests in watcher
* refactor APM to abort request with AbortController API
* fix missing es client calls in tests
* fix missing meta in detection engine FTR tests
* fix another bunch of errors in js tests
* fix wrong coercion
* remove test-grep pattern
* fix apm unit test
* rename terminateAfter to terminate_after in infra plugin
* rename terminateAfter to terminate_after in uptime plugin
* rename terminateAfter to terminate_after in apm plugin
* fix security roles FTR tests
* fix reference
* fix post_privilidges test
* fix post_privilidges
* bump client to 26
* add meta for index_management test helpers
* remove ts-expect-error caused by bad type in reason
* bump client to 27
* REMOVE me. workaround until fixed in the es client
* fix incorrect type casting
* swtich from camelCase params
* use `HttpConnection` for FTR-related clients
* bump client to 29
* Revert "REMOVE me. workaround until fixed in the es client"
This reverts commit c038850c09.
* fix new util
* revert repository changes
* do not crash if cannot store event_loop data
* fix new estypes imports
* fix more types
* fix security test types and add ts-ignore for custom ES client
* fix more estypes imports
* yet more ts violations
* line by line fixing is hard
* adapt `evaluateAlert` from infra as it's also used from FTR tests
* use convertToKibanaClient in FTR test instead of meta:true in plugin code
* migrate from deprecated API in fleet
* fix intergration tests
* fix fleet tests
* fix another fleet test
* fix more tests
* let's call it a day
* Removes custom header check on 404 responses, includes es client ProductNotSupportedError in EsUnavailableError conditional (#116029)
* Removes custom header check on 404 responses, includes es client ProductNotSupportedError in EsUnavailableError conditional
* Updates proxy response integration test
* disable APM until compatible with client v8
* skip async_search FTR test
* use kbnClient in integration tests
* bump version to 29
* bump to 30
* have configureClient return a KibanaClient instead of Client, remove resolved violations.
* bump to 31
* bump to 31
* Revert "bump to 31"
This reverts commit 5ac713e640.
* trigger stop to unusubscribe
* update generated docs
* remove obsolete test
* put "as" back
* cleanup
* skip test
* remove new type errors in apm package
* remove ErrorCause casting
* update a comment
* bump version to 32
* remove unnecessary ts-expect-error in apm code
* update comments
* update to client v33
* remove outdated type definition
* bump to 34 without params mutation
* unskip the test that should not fail anymore
* remove unnecessary ts-expect-error comments
* update to v35. body can be string
* move `sort` to body and use body friendly syntax
* fix a failing test. maps register the same SO that has been already registered by home
Co-authored-by: pgayvallet <pierre.gayvallet@gmail.com>
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
* remove transport API in favour of typed public API
* put elasticsearch_client back
* fix create index call
* fix setpolicy
* fix unit tests in SecuritySolution
## Summary
Removes ElasticSearch deprecation and makes it harder to import it from the wrong package. I accidentally exposed a deprecated `ElasticSearch` from a package we do not want to expose and everyone's IDE is suggesting it rather than the correct one from Kibana core.
* Removes the type from the exports within the package
* Fixes the instance that is trying to import it in favor of the correct one.
* [build_ts_refs] improve caches, allow building a subset of projects
* cleanup project def script and update refs in type check script
* rename browser_bazel config to avoid kebab-case
* remove execInProjects() helper
* list references for tsconfig.types.json for api-extractor workload
* disable composite features of tsconfig.types.json for api-extractor
* set declaration: true to avoid weird debug error
* fix jest tests
Co-authored-by: spalger <spalger@users.noreply.github.com>
## Summary
Addressees https://github.com/elastic/kibana/issues/83910 by removing the elastic legacy client from:
* `lists` plugin
* `security_solution` plugin
* `kbn-securitysolution-es-utils` package
Removes found dead code in `security_solution` plugin:
* `server/lib/configuration/inmemory_configuration_adapter.ts`
* `server/lib/detection_engine/privileges/read_privileges.ts`
* `server/lib/configuration/index.ts`
* `server/lib/configuration/adapter_types.ts`
* `server/lib/compose/kibana.ts`
* `server/lib/ecs_fields/extend_map.test.ts`
* `server/lib/ecs_fields/extend_map.ts`
* `server/lib/index_fields/elasticsearch_adapter.ts`
* `server/lib/index_fields/index.ts`
* `server/lib/index_fields/mock.ts`
* `server/lib/index_fields/types.ts`
* `server/lib/source_status/elasticsearch_adapter.ts`
* `server/lib/source_status/index.ts`
* `server/lib/source_status/query.dsl.ts`
* `server/lib/source_status/types.ts`
* `server/lib/sources/configuration.test.ts`
* `server/lib/sources/configuration.ts`
* `server/lib/sources/index.ts`
* `server/lib/sources/types.ts`
Removes dead code in `lists` plugin:
* `server/schemas/common/get_call_cluster.mock.ts`
* `server/lib/ecs_fields/index.ts`
* `server/lib/framework/kibana_framework_adapter.ts`
Removes dead types from `security_solution` plugin:
* `server/lib/framework/types.ts`
* `server/lib/types.ts`
Removes dead functions from `security_solution` plugin:
* `server/utils/build_query/calculate_timeseries_interval.ts`
* `server/utils/runtime_types.ts`
### What to check as a reviewer
* Ensure that there is no left over words of `legacy` such as `legacy.something`
* Ensure there are no more `callAsCurrentUser` since that is all dead and gone
* Ensure anywhere you see `esClient.someThing` it returns the `.body` at the end or destructors it as in `{ body } = esClient.someThing`
### Risk Matrix
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Telemetry might stop working or have invalid values. | Med | High | We will have to manually test telemetry. Pinged people from telemetry for a code review |
| An REST route returns invalid values. | Med | High | e2e tests caught some of these already. The rest of the code was re-checked by hand |
| Deleted function/code might actually be still in use somewhere. | Low | High | e2e and unit tests should catch any of this. |
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
An MVP of the RBAC work required for the "alerts as data" effort. An example of the existing implementation for alerts would be that of the security solution. The security solution stores its alerts generated from rules in a single data index - .siem-signals. In order to gain or restrict access to alerts, users do so by following the Elasticsearch privilege architecture. A user would need to go into the Kibana role access UI and give explicit read/write/manage permissions for the index itself.
Kibana as a whole is moving away from this model and instead having all user interactions run through the Kibana privilege model. When solutions use saved objects, this authentication layer is abstracted away for them. Because we have chosen to use data indices for alerts, we cannot rely on this abstracted out layer that saved objects provide - we need to provide our own RBAC! Instead of giving users explicit permission to an alerts index, users are instead given access to features. They don't need to know anything about indices, that work we do under the covers now.
Co-authored-by: Yara Tercero <yctercero@users.noreply.github.com>
Co-authored-by: Yara Tercero <yara.tercero@elastic.co>
