mirror of
https://github.com/elastic/kibana.git
synced 2025-04-20 16:03:20 -04:00
fa23a90d80
### Background This PR is a proposal for a different way to structure the Streams code flow based on some challenges faced while working on https://github.com/elastic/streams-program/issues/26 and discussed [here](https://github.com/elastic/streams-program/discussions/147) and [here](https://github.com/elastic/streams-program/discussions/55), mainly around finding it difficult to decide where to place certain validations that need access to the state as a whole. It is also in response to some expressed difficulty about how to add new stream types into the code base. It aims to achieve 3 goals: 1. It is easy to add new stream types and there is a clear place where changes (new validation, new logic) for existing stream types happen, making the code easier to evolve over time 2. It is easier to improve the robustness of the system because there are clear phases where problems can be caught, fixed and rolled back 3. It lays some ground work for features such as [bulk changes](https://github.com/elastic/streams-program/issues/125), [dry runs](https://github.com/elastic/streams-program/discussions/138) and a [health endpoint](https://github.com/elastic/streams-program/discussions/139) In the future, this will most likely be handled by Elasticsearch to a large degree, as imagined in https://github.com/elastic/streams-program/discussions/30 The solution takes inspiration from the reconciliation / controller pattern that Kubernetes uses, where users specify a desired state and the system takes action towards reaching that step. But it is also somewhat more similar to how React's Virtual DOM works in that it happens in a single iteration. Another key pattern is the [Active Record pattern](https://www.martinfowler.com/eaaCatalog/activeRecord.html), we let each stream class contain all the logic for how to validate and modify that stream in Elasticsearch. The client and `State` class simply orchestrate the flow but defer all actual work and decision making to the stream classes. **Note:** This PoC ignores the management of assets ### Summary The process takes the following steps: 1. A route accepts a request (upsert / delete) and translates it into one or more (for bulk) `StreamChange` objects before passing these to `State.applyChanges` method (which also takes a toggle for dry runs) 2. The current state of Streams is loaded by using the `State` class 3. The changes are then applied to the current state to derive the desired state [1] 4. The desired state is then validated, this is done by asking each individual stream if given the desired state and starting state, from the perspective of that individual stream, is it in a valid state (upserted or deleted correctly) 5. If the state is invalid, we return those errors and stop 6. Else we continue, if it's a dry run, we ask the desired state object for what has changed and report that in the shape of the Elasticsearch actions that would be attempted 7. Else we proceed to commit the changes to Elasticsearch by asking each changed stream to determine which Elasticsearch actions need to be performed to reach the desired state 8. These actions are then combined and sent to the `ExecutionPlan` class which does planning (mainly for actions around Unwired streams) and then handles executing the actions in the most parallel way but in the safe order 9. If any error happens, we attempt to revert back to the starting state by taking the changed streams and marking each stream as created based on the starting state and then getting the Elasticsearch actions for that and applying those This PR also changes our `resync` endpoint to make use of the same rough strategy (load current state, mark all as created, get Elasticsearch actions and apply). [1] Applying changes: 1. The current state is first cloned 2. Then for each change we see if it is a deletion or an upsert 3. Based on this we either mark existing streams for deletion or create/update existing streams 10. When creating a new stream instance we use the helper `streamFromDefinition` which is the only mapping between the definition documents and the Active Record-style stream type classes 11. As part of this, each stream that changes is marked in the desired state 12. The stream is passed the desired and current state and should update itself based on the change 13. The stream can return a set of cascading changes (taking the same format as the requested changes) which are executed directly after but we have a limit for how many rounds of cascading changes can happen to avoid infinite loops ### Adding new stream types Key in all of this is that the client and `State` classes don't know anything about any of the specific stream types, they know only of the `StreamActiveRecord` interface. When adding a new stream type you need to implement this interface and update `streamFromDefinition` to create the right class for your new definition. Streams of different types should only interact with each other by creating cascading changes. ### Possible follow up tasks - Introduce a lazy Elasticsearch cluster state cache because multiple places in the code access the same stuff over and over again - Make API endpoints the consume `attemptChanges` pass back the `DesiredState` and planned `ElasticsearchActions` as debug information based on a flag (maybe also all cascading changes) - Don't run cascading changes by default but run them if _some_ flag is submitted based on https://github.com/elastic/streams-program/discussions/230 - Wrap `attemptChanges` and `resync` with the new LockManager https://github.com/elastic/kibana/pull/216397 - Unit test WiredStream, UnwiredStream and GroupStream - Clean up old sync helpers - Wrap ES calls to get better stack traces for errors ### Out of scope - Asset linking and content pack installation (it's probably okay for these to continue to use the asset client directly since there is less domain logic and no cascading changes involved) --------- Co-authored-by: Joe Reuter <johannes.reuter@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
196 lines
6.3 KiB
JSON
196 lines
6.3 KiB
JSON
{
|
|
"extends": "../../tsconfig.base.json",
|
|
"compilerOptions": {
|
|
"outDir": "target/types",
|
|
"types": ["node", "@kbn/ambient-ftr-types"],
|
|
// there is still a decent amount of JS in this plugin and we are taking
|
|
// advantage of the fact that TS doesn't know the types of that code and
|
|
// gives us `any`. Once that code is converted to .ts we can remove this
|
|
// and allow TS to infer types from any JS file imported.
|
|
"allowJs": false
|
|
},
|
|
"include": [
|
|
"**/*",
|
|
"./api_integration/apis/logstash/pipeline/fixtures/*.json",
|
|
"./api_integration/apis/logstash/pipelines/fixtures/*.json",
|
|
"./api_integration/apis/telemetry/fixtures/*.json",
|
|
"./monitoring_api_integration/fixtures/**/*.json",
|
|
"../../typings/**/*",
|
|
"../../src/platform/packages/shared/kbn-test/types/ftr_globals/**/*"
|
|
],
|
|
"exclude": [
|
|
"security_solution_cypress/cypress/**/*",
|
|
"target/**/*",
|
|
"*/plugins/**/*",
|
|
"*/packages/**/*",
|
|
"*/*/packages/**/*",
|
|
"security_solution_api_integration/**/*",
|
|
"security_solution_endpoint/**/*"
|
|
],
|
|
"kbn_references": [
|
|
"@kbn/test-suites-src",
|
|
"@kbn/core",
|
|
"@kbn/data-plugin",
|
|
"@kbn/kibana-usage-collection-plugin",
|
|
"@kbn/share-plugin",
|
|
"@kbn/telemetry-collection-manager-plugin",
|
|
"@kbn/telemetry-plugin",
|
|
"@kbn/actions-plugin",
|
|
"@kbn/alerting-plugin",
|
|
"@kbn/apm-plugin",
|
|
"@kbn/cases-plugin",
|
|
"@kbn/fleet-plugin",
|
|
"@kbn/global-search-plugin",
|
|
"@kbn/features-plugin",
|
|
"@kbn/encrypted-saved-objects-plugin",
|
|
"@kbn/event-log-plugin",
|
|
"@kbn/features-plugin",
|
|
"@kbn/global-search-plugin",
|
|
"@kbn/index-management-plugin",
|
|
"@kbn/infra-plugin",
|
|
"@kbn/licensing-plugin",
|
|
"@kbn/ml-plugin",
|
|
"@kbn/monitoring-plugin",
|
|
"@kbn/observability-plugin",
|
|
"@kbn/security-plugin",
|
|
"@kbn/security-solution-plugin",
|
|
"@kbn/snapshot-restore-plugin",
|
|
"@kbn/spaces-plugin",
|
|
"@kbn/task-manager-plugin",
|
|
"@kbn/telemetry-collection-xpack-plugin",
|
|
"@kbn/transform-plugin",
|
|
"@kbn/triggers-actions-ui-plugin",
|
|
"@kbn/upgrade-assistant-plugin",
|
|
"@kbn/remote-clusters-plugin",
|
|
"@kbn/cross-cluster-replication-plugin",
|
|
"@kbn/synthetics-plugin",
|
|
"@kbn/global-search-test-plugin",
|
|
"@kbn/test",
|
|
"@kbn/repo-info",
|
|
"@kbn/tooling-log",
|
|
"@kbn/dev-utils",
|
|
"@kbn/dev-proc-runner",
|
|
"@kbn/ftr-common-functional-services",
|
|
"@kbn/securitysolution-list-constants",
|
|
"@kbn/expect",
|
|
"@kbn/dev-cli-errors",
|
|
"@kbn/ci-stats-reporter",
|
|
"@kbn/std",
|
|
"@kbn/apm-synthtrace",
|
|
"@kbn/core-saved-objects-base-server-internal",
|
|
"@kbn/rule-data-utils",
|
|
"@kbn/maps-plugin",
|
|
"@kbn/test-subj-selector",
|
|
"@kbn/rison",
|
|
"@kbn/reporting-plugin",
|
|
"@kbn/aiops-plugin",
|
|
"@kbn/ml-agg-utils",
|
|
"@kbn/logging",
|
|
"@kbn/utility-types",
|
|
"@kbn/data-views-plugin",
|
|
"@kbn/datemath",
|
|
"@kbn/safer-lodash-set",
|
|
"@kbn/es-archiver",
|
|
"@kbn/config-schema",
|
|
"@kbn/es-query",
|
|
"@kbn/session-view-plugin",
|
|
"@kbn/ml-is-populated-object",
|
|
"@kbn/ml-string-hash",
|
|
"@kbn/data-visualizer-plugin",
|
|
"@kbn/visualizations-plugin",
|
|
"@kbn/rule-registry-plugin",
|
|
"@kbn/controls-plugin",
|
|
"@kbn/core-saved-objects-server",
|
|
"@kbn/core-provider-plugin",
|
|
"@kbn/user-profile-components",
|
|
"@kbn/apm-synthtrace-client",
|
|
"@kbn/utils",
|
|
"@kbn/journeys",
|
|
"@kbn/alerting-api-integration-helpers",
|
|
"@kbn/cloud-security-posture-plugin",
|
|
"@kbn/cloud-integration-saml-provider-plugin",
|
|
"@kbn/security-api-integration-helpers",
|
|
"@kbn/alerts-as-data-utils",
|
|
"@kbn/discover-plugin",
|
|
"@kbn/files-plugin",
|
|
"@kbn/shared-ux-file-types",
|
|
"@kbn/guided-onboarding-plugin",
|
|
"@kbn/field-formats-plugin",
|
|
"@kbn/ml-anomaly-utils",
|
|
"@kbn/ml-data-frame-analytics-utils",
|
|
"@kbn/data-forge",
|
|
"@kbn/observability-shared-plugin",
|
|
"@kbn/maps-vector-tile-utils",
|
|
"@kbn/server-route-repository",
|
|
"@kbn/core-http-common",
|
|
"@kbn/lens-plugin",
|
|
"@kbn/logs-shared-plugin",
|
|
"@kbn/telemetry-tools",
|
|
"@kbn/profiling-plugin",
|
|
"@kbn/observability-onboarding-plugin",
|
|
"@kbn/uptime-plugin",
|
|
"@kbn/ml-category-validator",
|
|
"@kbn/observability-ai-assistant-plugin",
|
|
"@kbn/stack-connectors-plugin",
|
|
"@kbn/stack-alerts-plugin",
|
|
"@kbn/profiling-utils",
|
|
"@kbn/profiling-data-access-plugin",
|
|
"@kbn/es",
|
|
"@kbn/metrics-data-access-plugin",
|
|
"@kbn/dataset-quality-plugin",
|
|
"@kbn/reporting-export-types-csv-common",
|
|
"@kbn/reporting-export-types-pdf-common",
|
|
"@kbn/reporting-export-types-png-common",
|
|
"@kbn/reporting-common",
|
|
"@kbn/io-ts-utils",
|
|
"@kbn/security-plugin-types-common",
|
|
"@kbn/slo-schema",
|
|
"@kbn/typed-react-router-config",
|
|
"@kbn/ftr-common-functional-ui-services",
|
|
"@kbn/infra-forge",
|
|
"@kbn/slo-plugin",
|
|
"@kbn/aiops-test-utils",
|
|
"@kbn/observability-ai-assistant-app-plugin",
|
|
"@kbn/aiops-log-rate-analysis",
|
|
"@kbn/apm-data-view",
|
|
"@kbn/core-saved-objects-api-server",
|
|
"@kbn/search-types",
|
|
"@kbn/alerting-comparators",
|
|
"@kbn/alerting-state-types",
|
|
"@kbn/reporting-server",
|
|
"@kbn/data-quality-plugin",
|
|
"@kbn/observability-synthetics-test-data",
|
|
"@kbn/openapi-common",
|
|
"@kbn/securitysolution-lists-common",
|
|
"@kbn/securitysolution-exceptions-common",
|
|
"@kbn/securitysolution-endpoint-exceptions-common",
|
|
"@kbn/entityManager-plugin",
|
|
"@kbn/osquery-plugin",
|
|
"@kbn/entities-schema",
|
|
"@kbn/actions-simulators-plugin",
|
|
"@kbn/cases-api-integration-test-plugin",
|
|
"@kbn/security-solution-plugin/public/management/cypress",
|
|
"@kbn/management-settings-ids",
|
|
"@kbn/mock-idp-utils",
|
|
"@kbn/cloud-security-posture-common",
|
|
"@kbn/saved-objects-management-plugin",
|
|
"@kbn/alerting-types",
|
|
"@kbn/ai-assistant-common",
|
|
"@kbn/core-deprecations-common",
|
|
"@kbn/usage-collection-plugin",
|
|
"@kbn/palettes",
|
|
"@kbn/sse-utils-server",
|
|
"@kbn/gen-ai-functional-testing",
|
|
"@kbn/automatic-import-plugin",
|
|
"@kbn/core-elasticsearch-server",
|
|
"@kbn/streams-schema",
|
|
"@kbn/server-route-repository-utils",
|
|
"@kbn/streams-plugin",
|
|
"@kbn/response-ops-rule-params",
|
|
"@kbn/scout-info",
|
|
"@kbn/inference-common",
|
|
"@kbn/apm-sources-access-plugin",
|
|
"@kbn/aiops-change-point-detection",
|
|
"@kbn/es-errors",
|
|
]
|
|
}
|