df0a21cac1
Closes https://github.com/elastic/kibana/issues/170220.
### Changes
- New endpoint added to query malformed docs in elasticsearch `GET
/internal/dataset_quality/data_streams/malformed_docs`
- Decoded response from apis in `data_streams_stats_client.ts` as
suggested by @tonyghiani in
https://github.com/elastic/kibana/pull/171777.
- New synthtrace scenario, malformed logs, where we ingest documents
that will have `_ignored` properties.
- Malformed Docs column was added to `columns.tsx`.
#### Demo
|
||
|---|---|---|
| .. | ||
| common | ||
| public | ||
| scripts | ||
| server | ||
| jest.config.js | ||
| kibana.jsonc | ||
| README.md | ||
| tsconfig.json | ||
Dataset Quality
In order to make ongoing maintenance of log collection easy we want to introduce the concept of dataset quality, where users can easily get an overview on the datasets they have with information such as integration, size, last activity, among others.
Development
Tests
Unit tests
Kibana primarily uses Jest for unit testing. Each plugin or package defines a jest.config.js that extends a preset provided by the @kbn/test package. The following command runs all Dataset quality unit tests:
yarn jest --config x-pack/plugins/dataset_quality/jest.config.js
You can also run a specific test by passing the filepath as an argument, e.g.:
yarn jest --config x-pack/plugins/dataset_quality/jest.config.js x-pack/plugins/dataset_quality/server/routes/data_streams/get_data_streams/get_data_streams.test.ts
API integration tests
| Option | Description |
|---|---|
| --server | Only start ES and Kibana |
| --runner | Only run tests |
| --grep | Specify the specs to run |
| --grep-files | Specify the files to run |
| --inspect | Add --inspect-brk flag to the ftr for debugging |
| --times | Repeat the test n number of times |
The API tests are located in x-pack/test/dataset_quality_api_integration/.
Start server and run test (single process)
node x-pack/plugins/dataset_quality/scripts/api [--help]
The above command will start an ES instance on http://localhost:9220, a Kibana instance on http://localhost:5620 and run the api tests. Once the tests finish, the instances will be terminated.
Start server and run test (separate processes)
# start server
node x-pack/plugins/dataset_quality/scripts/api --server
# run tests
node x-pack/plugins/dataset_quality/scripts/api --runner --grep-files=error_group_list