## Summary
This PR adds the [High Cardinality
Indexer](https://github.com/elastic/high-cardinality-cluster) to Kibana
as a new package called `kbn-data-forge`. It also replaces
`kbn-infra-forge` usage in the test and is the preferred way to generate
data for Observability use cases, specifically for SLO testing.
### Todo
- [x] Replace `kbn-infra-forge` usage
- [x] Create convenience functions for testing (`generate` and
`cleanup`)
- [x] Make the logger (`LoggingTool`) configurable as an injected
dependency
- [x] Make the Elasticsearch client (`Client`) configurable as an
injected dependency
- [x] Fix the ECS Generate commands
- [x] Add CLI options via Commander
### CLI Help Screen
```
Usage: data_forge.js [options]
A data generation tool that will create realistic data with different scenarios.
Options:
--config <filepath> The YAML config file
--lookback <datemath> When to start the indexing (default: "now-15m")
--events-per-cycle <number> The number of events per cycle (default: 1)
--payload-size <number> The size of the ES bulk payload (default: 10000)
--concurrency <number> The number of concurrent connections to Elasticsearch (default: 5)
--index-interval <milliseconds> The interval of the data in milliseconds (default: 60000)
--dataset <dataset> The name of the dataset to use. Valid options: "fake_logs", "fake_hosts", "fake_stack" (default: "fake_logs")
--scenario <scenerio> The scenario to label the events with (default: "good")
--elasticsearch-host <address> The address to the Elasticsearch cluster (default: "http://localhost:9200")
--elasticsearch-username <username> The username to for the Elasticsearch cluster (default: "elastic")
--elasticsearch-password <password> The password for the Elasticsearch cluster (default: "changeme")
--elasticsearch-api-key <key> The API key to connect to the Elasticsearch cluster
--kibana-url <address> The address to the Kibana server (default: "http://localhost:5601")
--kibana-username <username> The username for the Kibana server (default: "elastic")
--kibana-password <password> The password for the Kibana server (default: "changeme")
--install-kibana-assets This will install index patterns, visualizations, and dashboards for the dataset
--event-template <template> The name of the event template (default: "good")
--reduce-weekend-traffic-by <ratio> This will reduce the traffic on the weekends by the specified amount. Example: 0.5 will reduce the traffic by half (default: 0)
--ephemeral-project-ids <number> The number of ephemeral projects to create. This is only enabled for the "fake_stack" dataset. It will create project IDs that will last 5 to 12 hours. (default: 0)
-h, --help output usage information
```
### Testing an Example
Run the following command against a clean Kibana development enviroment:
```
node x-pack/scripts/data_forge.js --events-per-cycle 200 --lookback now-1h --install-kibana-assets --ephemeral-project-ids 10 --dataset fake_stack
```
This should install a handful of DataViews (Admin Console, Message
Processor, Nginx Logs, Mongodb Logs) along with a few dashboards and
visualizations.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
