In order to get us closer to the developer experience we want for
packages, we are trying to move package builds out of bazel and instead
we want to build files on demand. In the case of .peggy files this means
importing them directly and teaching babel/jest/webpack how to handle
these imports by automatically transpiling and caching the results.
This change does just that, adding a `@kbn/peggy` package which wraps
peggy for types, and also adds support for defining peggy config
adjacent to a peggy grammar file in a `${basename}.config.json` file.
This file will be parsed and used to configure things like
`allowedStartRules` as described in [the peggy
docs](https://peggyjs.org/documentation.html#generating-a-parser-javascript-api).
This PR also implements `@kbn/peggy-loader` which uses `@kbn/peggy` to
transpile peggy files in webpack, and a peggy transform for both Jest
and our custom babel register hook.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fix https://github.com/elastic/kibana/issues/144255
Create the `@kbn/core-root-server-internal` package, and move `Root`,
`Server` and `bootstrap` to it.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Moves the ownership for some of the appservices code to the
visualizations team:
- expressions plugin
- unified search
- data plugin, we decided for now to co-own it with the
@elastic/kibana-data-discovery team
## Summary
In order to reduce the number of irrelevant PR review requests, this PR
updates the CODEOWNERS file and removes the
@elastic/security-detections-response team from code owners of the
`rule_registry` plugin. This affects the following sub-teams:
- @elastic/security-detections-response-alerts. Although this team owns
the alerts generation and indexing of the Detection Engine rule types
(and this code depends on the `RuleDataClient`), they prefer not to be a
code owner, because they keep getting pinged for PRs that don't need
their review.
- @elastic/security-detections-response-rules area team. This team
doesn't depend on the rule registry and doesn't do any work related to
alerts-as-data.
- @elastic/security-solution-platform. This team won't be doing anything
in the rule registry for the foreseeable future.
https://github.com/elastic/kibana/pull/144326 attempted to change the
owners for packages, but because of a misconfiguration in CI the correct
codeowners file wasn't determined until after the PR was merged. This
fixed the issue by modifying the kibana.jsonc files of the packages to
use the correct owner, which then ends up in the codeowners file.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* [Fleet] Update GH Projects automation
Update GH projects automation for issues labeled with `Team:Fleet` to be automatically added to the Ingest Dev project with the proper `Area` property set.
* Update add-to-fleet-project.yml
* Rename add-to-fleet-project.yml to add-fleet-issues-to-ingest-project.yml
**Partially addresses:** https://github.com/elastic/kibana/issues/138600, https://github.com/elastic/kibana/issues/92169, https://github.com/elastic/kibana/issues/138606
**Addresses:** https://github.com/elastic/kibana/issues/136957, https://github.com/elastic/kibana/issues/136962, https://github.com/elastic/kibana/issues/138614
## Summary
In this PR we are:
- Splitting the Detection Engine into subdomains ([ticket](https://github.com/elastic/kibana/issues/138600)). Every subdomain got its own folder under `detection_engine`, and we moved some (not all) code into them. More on that is below. New subdomains introduced:
- `fleet_integrations`
- `prebuilt_rules`
- `rule_actions_legacy`
- `rule_exceptions`
- `rule_management`
- `rule_preview`
- `rule_schema`
- `rule_creation_ui`
- `rule_details_ui`
- `rule_management_ui`
- `rule_exceptions_ui`
- Updating the CODEOWNERS file accordingly.
- Refactoring the Rule Management page and the Rules table. Our main focus was on the way how we communicate with the API endpoints, how we cache and invalidate the fetched data, and how this code is organized in the codebase. More on that is below.
- Increasing the bundle size limit. This is going to be decreased back in a follow-up PR ([ticket](https://github.com/elastic/kibana/issues/143532))
## Restructuring folders into subdomains
For the background and problem statement, please refer to https://github.com/elastic/kibana/issues/138600
We were focusing on code that is closely related to the Rules area: either owned by us de facto (we work on it) or owned by us de jure (according to the CODEOWNERS file). Or goal was to explicitly extract code that we don't own de facto into separate subdomains, transfer ownership to other area teams, and reflect this in the CODEOWNERS file. On the other hand, we wanted the code that we own to also be organized in clear subdomains that we could easily own via CODEOWNERS. We didn't touch the code that is already explicitly owned by other area teams, e.g. `x-pack/plugins/security_solution/server/lib/detection_engine/rule_types`.
This is a draft "domain map" - an architectural diagram that shows how the Detection Engine _could_ be split into subdomains. It's more a TO-BE idea/aspiration rather than an AS-IS statement. Any feedback, critiques, and suggestions would be extremely appreciated!
<img width="2592" alt="Screenshot 2022-10-18 at 16 08 40" src="https://user-images.githubusercontent.com/7359339/196453965-b65f5b49-9a33-4d90-bb48-1347e9576223.png">
It shows the flow of dependencies between subdomains and proposes some rules:
- The whole graph of dependencies between all subdomains should be a DAG. There should not be bi-directional or circular dependencies between them.
- **Generic subdomains** represent some general knowledge that can be used/applied outside of the Detection Engine.
- Can depend on some generic kbn packages, npm packages or utils.
- Can't depend on any other Detection Engine subdomains.
- **Crosscutting subdomains** represent some code that can be common to / shared between many other subdomains. This could be some very common domain models and API schemas.
- Can depend on generic subdomains.
- Can depend on other crosscutting subdomains (dependencies between them must form a DAG).
- Can't depend on core or UI subdomains.
- **Core subdomains** contain most of the "meat" of the Detection Engine: domain models, server-side and client-side business logic, server-side API endpoints, client-side UI (potentially shareable between several pages).
- Can depend on crosscutting and generic subdomains.
- Can depend on other core subdomains (dependencies between them must form a DAG).
- Can't depend on UI subdomains.
- **UI subdomains** contain the implementation of pages related to the Detection Engine. Every page can easily depend on several core subdomains, so these subdomain are on top of everything.
- Can depend on any other subdomains. Dependencies must form a DAG.
Dashed lines show some existing dependencies that we think should be eliminated.
Ownership TO-BE is color-coded. We updated the CODEOWNERS file according to the new folders.
The folder restructuring is not 100% finished but we did a big part of it. Most of the FE code continues to live in legacy folders, e.g. see `x-pack/plugins/security_solution/public/detections`. So this work is to be continued...
## Refactoring of Rule Management FE
- [x] https://github.com/elastic/kibana/issues/136957 For effective HTTP requests caching and deduplication, we've migrated all data fetching logic to `useQuery` and `useMutation` hooks from `react-query`. That allowed us to introduce the following improvements to our codebase:
* All outgoing HTTP requests are now automatically deduplicated. That means that data fetching hooks like `useRule` could be used on any level in the component tree to access response data directly. So, no need to put the hook on the top level anymore and use prop-drilling to make the response data available to all children components that require it.
* All HTTP responses are now cached with the default TTL of 5 minutes—no more redundant requests. With a hot cache, transitions to some pages now happen immediately.
- [x] https://github.com/elastic/kibana/issues/136962 Data fetching hooks of the Rules Area are now organized in one place. `security_solution/public/detection_engine/rule_management/api/hooks` contains abstraction layer on top of the Kibana's HTTP client. All data fetching should happen exclusively through that layer to ensure that:
* Mutation queries automatically invalidate associated cache entries.
* Optimistic updates or updates from mutation responses could be implemented centrally where possible.
- [x] https://github.com/elastic/kibana/issues/92169 From some of the Rule Management components, logic was extracted to hooks located in `security_solution/public/detection_engine/rule_management/logic`.
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* [Discover] Create unifiedHistogram plugin
* [Discover] Move discover resizable panels to unifiedHistogram
* [Discover] Replace DiscoverPanels with unifiedHistogram Panels
* [CI] Auto-commit changed files from 'node scripts/build_plugin_list_docs'
* [Discover] Fix types and limtis.yml for unifiedHistogram
* [Discover] Begin migrating layout and chart to unified_histogram
* [Discover] Update i18n keys from discover to unifiedHistogram
* [Discover] Update data-test-subj tags from discover to unifiedHistogram
* [Discover] Update classNames, ids, and scss to change discover to unifiedHistogram
* [Discover] Remove more references to discover from unifiedHistogram
* [Discover] Replace DiscoverServices with UnifiedHistogramServices
* [Discover] Replacing CHART_HIDDEN_KEY with chartHiddenKey prop
* [Discover] Add missing tsconfig references
* [Discover] Remove remaining references to discover from unifiedHistogram
* [Discover] Migrate HitsCounter to unifiedHistogram
* [Discover] Continue removing discover dependencies from unifiedHistogram
* [Discover] Replace SCSS with emotion
* [Discover] Changing PANELS_MODE to be internal
* [Discover] Clean up types
* [Discover] Clean up props and types
* [Discover] Update layout to use Chart component
* [Discover] Update discover_main_content
* [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'
* [Discover] Update discover_main_content to use UnifiedHistogramLayout, clean up unifiedHistogram implementation and props, add missing bundles
* [Discover] Fix missing styles in unifiedHistogram
* [Discover] Fix issue where mouse can get out of sync with the resize handle with the Discover resizable layout
* [Discover] Fix some Jest tests
* [Discover] Update discoverQueryHits to unifiedHistogramQueryHits in tests
* [Discover] Finish decoupling discover_main_content from unified histogram layout
* [Discover] Create useDiscoverHistogram hook and remove old histogram dependencies from Discover
* [Discover] Move functions to create chart data from discover to unifiedHistogram
* [Discover] Continue fixing broken Jest tests
* Revert unifiedHistogram.reloadSavedSearchButton removal
* [Discover] Add missing type export and a better suspense fallback
* [Discover] Make callback names consistent
* [Discover] Continue cleanup and add documentation to unifiedHistogram
* [Discover] Update genChartAggConfigs to take object
* [Discover] Update UnifiedHistogramHitsContext.number to total
* [Discover] Cleanup imports
* [Discover] Add support for hiding the entire top panel in the unified histogram by leaving all context props undefined
* [Discover] Fix broken discover_layout unit tests
* [Discover] Clean up naming in discover_main_content
* [Discover] Continue fixing Jest tests and adding new tests
* [Discover] Finish writing Jest tests
* [Discover] Fix conflicts with getVisualizeInformation and triggerVisualizeActions after merge
* [Discover] Fix hiding reset chart height button when default chart height
* [Discover] Update CODEOWNER file
* [Discover] Removed types for @link comments
* [Discover] Fix broken discover_layout.test.tsx file
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
