kibana/packages
Yngrid Coello 511f77c231
[Dataset quality] Failure store support (#206758)
Closes https://github.com/elastic/logs-dev/issues/183,
https://github.com/elastic/logs-dev/issues/184 and
https://github.com/elastic/logs-dev/issues/185.

## Summary
This PR aims to support failure store in dataset quality page. The
following acceptance criteria items were resolved

### Dataset quality page
- [x] A column for Failed docs is included in the table
- [x] A tooltip is placed in the title of the column
- [x] A % of documents inside Failure store is calculated for every
dataStream
- [x] If % is lesser than 0.0001 but greater than 0 we should show ⚠
symbol next to the ~0 value (as we do with degraded docs)
- [x] Failed docs percentages greater than 0 should link to discover

 🎥 Demo 


https://github.com/user-attachments/assets/6d9e3f4c-02d9-43ab-88cb-ae70716b05d9

### Dataset details page
- [x] A metric, Failed docs, is included in the Overview panel under
Data set quality. This metric includes the number of documents inside
the failure store for the specific dataStream.
- [x] A tooltip is placed in the title of the Failed docs metric with
message: `The percentage of docs sent to failure store due to an issue
during ingestion.`
- [x] Degraded docs graph section is transformed to Document trends
allowing the users to switch between Degraded docs and Failed docs
trends over time.
- [x] A new chart for failed documents is created with links to
discover/Logs explorer using the right dataView

 🎥 Demo 


https://github.com/user-attachments/assets/6a3a1f09-2668-4e83-938e-ecdda798c199

### Failed docs ingestion issue flyout

- [x] Whenever documents are found in failure store we should list
Document indexing failed in Quality issues table
- [x] User should be able to expand Document indexing failed and see
more information in the flyout
- [x] The flyout will show Docs count, an aggregation of the number of
documents inside failure store for the selected timeframe
- [x] The flyout will show Last ocurrence, the datetime registered for
the most recent document in the failure store.
- [x] The flyout will contain a section called Error messages where a
list of unique error messages should be shown, exposing Content (error
message) and Type (Error Type).
- [x] Type should contain a tooltip where message (`Error message
category`) explain users how we are categorising the errors.
- [x] Other issues inside Quality issues table will be appended by field
ignored and the field will be shown in bold.


https://github.com/user-attachments/assets/94dc81f0-9720-4596-b256-c9d289cefd94

Note: This PR was reconstructed from
https://github.com/elastic/kibana/pull/199806 which it supersedes.

## How to test

1. Execute `failed_logs` synthtrace scenario
2. Open dataset quality page

## Follow ups
- Enable in serverless
- Deployment agnostic tests cannot be added until we enable this in
serverless
- FTR tests will be added as part of
https://github.com/elastic/logs-dev/issues/182

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2025-01-23 09:13:28 +01:00
..
core Replace Borealis isServerless conditional with YML config (#206690) 2025-01-20 16:38:50 +01:00
kbn-ambient-common-types Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-ambient-ftr-types Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-ambient-storybook-types Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-ambient-ui-types Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-apm-synthtrace [Dataset quality] Failure store support (#206758) 2025-01-23 09:13:28 +01:00
kbn-apm-synthtrace-client [One Discover] Display stacktrace in the logs overview tab (#204521) 2025-01-22 18:06:14 +02:00
kbn-axe-config Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-babel-preset [Security Solution] - remove styled-components and cleanup for event viewer and data table components (#206523) 2025-01-15 09:18:19 -06:00
kbn-babel-register Sustainable Kibana Architecture: Move modules owned by @elastic/kibana-operations (#202739) 2024-12-31 13:47:59 +01:00
kbn-babel-transform Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-bazel-runner Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-capture-oas-snapshot-cli Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-check-mappings-update-cli [Fleet] Send Agentless API resources (#206042) 2025-01-19 12:52:10 +02:00
kbn-check-prod-native-modules-cli Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-ci-stats-core Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-ci-stats-performance-metrics Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-ci-stats-reporter Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-ci-stats-shipper-cli Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-cli-dev-mode Sustainable Kibana Architecture: Move modules owned by @elastic/kibana-presentation (#204843) 2024-12-23 12:32:11 -06:00
kbn-code-owners Fix CODEOWNERS (#205302) 2025-01-02 08:36:55 -06:00
kbn-cypress-config Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-dependency-ownership Add check to fail CI if any dependencies are unowned (#206679) 2025-01-16 09:59:04 -05:00
kbn-dependency-usage Sustainable Kibana Architecture: Move modules owned by @elastic/kibana-security (#202748) 2025-01-05 12:57:01 +01:00
kbn-dev-cli-errors Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-dev-cli-runner Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-dev-proc-runner Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-dev-utils Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-docs-utils Sustainable Kibana Architecture: Move modules owned by @elastic/kibana-visualizations (#202754) 2025-01-08 12:19:24 +01:00
kbn-es [SecuritySolution] Breaking out timeline & note privileges (#201780) 2025-01-20 14:09:16 +01:00
kbn-es-archiver [ES body removal] @elastic/appex-qa (#204878) 2024-12-19 12:06:42 +01:00
kbn-eslint-config Sustainable Kibana Architecture: Move modules owned by @elastic/kibana-core (#201653) 2025-01-04 11:47:24 -07:00
kbn-eslint-plugin-css Resolve false positives with eslint no color rule (#204848) 2024-12-19 10:26:48 +00:00
kbn-eslint-plugin-disable Add ESLINT constraints to detect inter-group dependencies (#194810) 2024-10-22 06:34:19 -05:00
kbn-eslint-plugin-eslint [Authz] Added section for migration routes created by utility function (#198401) 2024-10-30 13:05:38 -05:00
kbn-eslint-plugin-i18n Sustainable Kibana Architecture: Move modules owned by @elastic/kibana-visualizations (#202754) 2025-01-08 12:19:24 +01:00
kbn-eslint-plugin-imports Sustainable Kibana Architecture: Move modules owned by @elastic/kibana-operations (#202739) 2024-12-31 13:47:59 +01:00
kbn-eslint-plugin-telemetry Sustainable Kibana Architecture: Move modules owned by @elastic/obs-ux-infra_services-team (#202830) 2024-12-29 09:58:37 +01:00
kbn-expect Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-failed-test-reporter-cli [kbn-scout-reporting] add failed test reporter (#205096) 2025-01-15 18:40:34 +00:00
kbn-find-used-node-modules Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-ftr-common-functional-services [FTR] Fixup Retry Logging (#205894) 2025-01-16 17:03:16 +00:00
kbn-ftr-common-functional-ui-services [Data Usage] functional tests (#203166) 2024-12-12 08:08:08 -05:00
kbn-ftr-screenshot-filename Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-gen-ai-functional-testing Add base FTR test coverage for inference APIs (#198000) 2024-12-04 06:39:45 -06:00
kbn-generate SKA: Cleanup and follow-up tasks (1 of many) (#206420) 2025-01-14 19:03:58 +01:00
kbn-generate-console-definitions SKA: Categorise remaining packages (#205875) 2025-01-08 22:20:14 +01:00
kbn-get-repo-files Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-import-locator Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-import-resolver Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-io-ts-utils [Perfomance] Track time range picker with onPageReady function (#202889) 2024-12-13 07:11:44 -06:00
kbn-jest-serializers Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-journeys Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-json-ast Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-kibana-manifest-schema Kibana Sustainable Architecture: Force visibility: 'private' for solutions in manifest (#199452) 2024-11-08 15:36:07 +01:00
kbn-lint-packages-cli [codeowners] Filter kibanamachine (#199404) 2024-11-12 12:39:17 -06:00
kbn-lint-ts-projects-cli Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-managed-vscode-config Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-managed-vscode-config-cli Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-management Sustainable Kibana Architecture: Move the rest of shared-ux modules (#205924) 2025-01-10 11:30:37 +00:00
kbn-manifest Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-mock-idp-plugin Preparation for High Contrast Mode, Security domains (#202609) 2024-12-09 13:03:23 -07:00
kbn-mock-idp-utils Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-openapi-bundler Updated js-yaml to v4 (#190678) 2024-09-19 12:25:03 +02:00
kbn-openapi-generator Sustainable Kibana Architecture: Move modules owned by @elastic/security-solution (#202851) 2024-12-16 22:55:27 -06:00
kbn-optimizer [controls] lazy load control actions (#206876) 2025-01-20 14:00:48 -07:00
kbn-optimizer-webpack-helpers Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-peggy Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-peggy-loader Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-performance-testing-dataset-extractor [ES body removal] @elastic/appex-qa (#204878) 2024-12-19 12:06:42 +01:00
kbn-picomatcher Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-plugin-check SKA: Categorise remaining packages (#205875) 2025-01-08 22:20:14 +01:00
kbn-plugin-generator Sustainable Kibana Architecture: Update tooling to take into account the new folders (#202327) 2024-12-02 04:04:35 -06:00
kbn-plugin-helpers Sustainable Kibana Architecture: Move CodeEditor related packages #205587 (#205738) 2025-01-08 15:25:47 +01:00
kbn-relocate SK: Relocate Script v7.2 (#207081) 2025-01-20 15:43:32 +01:00
kbn-repo-file-maps Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-repo-linter Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-repo-path Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-repo-source-classifier Sustainable Kibana Architecture: Move modules owned by @elastic/kibana-visualizations (#202754) 2025-01-08 12:19:24 +01:00
kbn-repo-source-classifier-cli Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-scout [kbn-scout-reporting] add failed test reporter (#205096) 2025-01-15 18:40:34 +00:00
kbn-scout-info [kbn-scout] Custom event-oriented test reporter & persistence (#202906) 2024-12-09 14:34:25 +00:00
kbn-scout-reporting [kbn-scout-reporting] escape html characters in html report (#206987) 2025-01-17 10:16:15 +01:00
kbn-set-map SKA: Categorise remaining packages (#205875) 2025-01-08 22:20:14 +01:00
kbn-some-dev-log Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-sort-package-json Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-stdio-dev-helpers Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-storybook Sustainable Kibana Architecture: Move plugins owned by @elastic/appex-sharedux (#204959) 2025-01-03 05:38:57 -06:00
kbn-styled-components-mapping-cli Update styled_components_files.js to include all files that import styled-components (#205011) 2025-01-05 16:54:17 +01:00
kbn-telemetry-tools Fix telemetry CLI to write empty properties collection (#205746) 2025-01-07 09:46:43 -07:00
kbn-test [React@18] fix outstanding easy unit tests (#206917) 2025-01-17 11:55:08 +00:00
kbn-test-eui-helpers Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-test-jest-helpers Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-test-subj-selector [scout] migrate more Discover tests (#201842) 2024-12-02 20:57:29 +01:00
kbn-tooling-log Sustainable Kibana Architecture: Categorise straightforward packages (#199630) 2024-11-22 10:33:25 +01:00
kbn-ts-projects Dependency usage CLI (#198920) 2024-11-25 14:07:40 +01:00
kbn-ts-type-check-cli Remove references to old type check script (#202825) 2024-12-18 11:38:30 -08:00
kbn-validate-next-docs-cli [ci] Work around docosaurus errors (#206097) 2025-01-09 16:33:30 -06:00
kbn-web-worker-stub Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-whereis-pkg-cli Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
kbn-yarn-lock-validator Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
serverless/storybook/config Sustainable Kibana Architecture: Move the rest of shared-ux modules (#205924) 2025-01-10 11:30:37 +00:00
README.md Sustainable Kibana Architecture: Move modules owned by @elastic/kibana-core (#201653) 2025-01-04 11:47:24 -07:00

Kibana-related packages

This folder contains packages that are intended for use in Kibana and Kibana plugins.

tl;dr:

  • Don't publish to npm registry
  • Always use the @kbn namespace
  • Always set "private": true in package.json

Using these packages

We no longer publish these packages to the npm registry. Now, instead of specifying a version when including these packages, we rely on yarn workspaces, which sets up a symlink to the package.

For example if you want to use the @kbn/i18n package in Kibana itself, you can specify the dependency like this:

"@kbn/i18n": "1.0.0"

However, if you want to use this from a Kibana plugin, you need to use a link: dependency and account for the relative location of the Kibana repo, so it would instead be:

"@kbn/i18n": "link:../../kibana/src/platform/packages/shared/kbn-i18n"

then run yarn kbn bootstrap from the plugin directory.

Creating a new package

Run the following command from the root of the Kibana repo:

node scripts/generate package @kbn/<PACKAGE_NAME> --web --owner @elastic/<TEAM_NAME>

Unit tests for a package

Currently there is only one tool being used in order to test packages which is Jest. Below we will explain how it should be done.

Jest

A package should follow the pattern of having .test.js files as siblings of the source code files, and these run by Jest.

A package using the .test.js naming convention will have those tests automatically picked up by Jest and run by the unit test runner, currently mapped to the Kibana test script in the root package.json.

  • yarn test runs all unit tests.
  • yarn jest runs all Jest tests in Kibana.

In order for the plugin or package to use Jest, a jest.config.js file must be present in it's root. However, there are safeguards for this in CI should a test file be added without a corresponding config file.


Each package can also specify its own test script in the package's package.json, for cases where you'd prefer to run the tests from the local package directory.