mirror of
https://github.com/elastic/kibana.git
synced 2025-06-27 18:51:07 -04:00
b76b5c3b64
## Summary With the merge of https://github.com/elastic/kibana/pull/174631, we've need of a follow up pr, to extend the performance override, to the command line interface. Also, modify the type of the performance option override, by making both the batch size and the concurrency optional. - With this change, if one or the other of the two options (batch size & concurrency) are not provided, the defaults are used. ### How to Test locally Start the server in one terminal, such as: `$ node scripts/functional_tests_server.js --config test/functional/apps/console/config.ts` Then in a second terminal: `$ node scripts/es_archiver.js load x-pack/test/functional/es_archives/logstash_functional --batch-size 300 --concurrency 2` #### Additional Testing (regarding only using one option or the other) _Just batch size_ `$ node scripts/es_archiver.js load x-pack/test/functional/es_archives/logstash_functional --batch-size 300` _Just concurrency_ `$ node scripts/es_archiver.js load x-pack/test/functional/es_archives/logstash_functional --concurrency 2` #### To Slow the Load Action Down (good for testing) Load an archive with very low numbers `node scripts/es_archiver.js load x-pack/test/functional/es_archives/logstash_functional --concurrency 1 --batch-size 10` You should see the progress indicator due to the forced latency ``` info [x-pack/test/functional/es_archives/logstash_functional] Created index "logstash-2015.09.20" info [x-pack/test/functional/es_archives/logstash_functional] Deleted existing index "logstash-2015.09.21" info [x-pack/test/functional/es_archives/logstash_functional] Created index "logstash-2015.09.21" info progress: 2940 info progress: 6040 info progress: 9380 info progress: 12451 info [x-pack/test/functional/es_archives/logstash_functional] Indexed 4634 docs into "logstash-2015.09.22" info [x-pack/test/functional/es_archives/logstash_functional] Indexed 4757 docs into "logstash-2015.09.20" info [x-pack/test/functional/es_archives/logstash_functional] Indexed 4614 docs into "logstash-2015.09.21" ``` --------- Co-authored-by: Robert Oskamp <traeluki@gmail.com> |
||
|---|---|---|
| .. | ||
| src | ||
| index.ts | ||
| jest.config.js | ||
| kibana.jsonc | ||
| package.json | ||
| README.mdx | ||
| tsconfig.json | ||
---
id: kibDevDocsOpsEsArchiver
slug: /kibana-dev-docs/ops/es-archiver
title: "ES Archiver"
description: A tool which helps developers capture and restore ES indexes
date: 2024-01-11
tags: ['kibana', 'dev', 'contributor', 'operations', 'ci', 'es-archiver', 'load', 'es-archiver-load-action', 'performance']
---
The ES Archiver is a service primarily used by the Functional Tests to load up ES indexes using the bulk API which makes the archives more resilient to ES upgrades and easier to inspect/edit locally because they are just plain text files containing newline-delimited JSON (though they are sometimes compressed).
## CLI
This tool also has a CLI which can be used to save indexes to new archives or load additional archives into a specific ES instance.
To teach the ES Archiver how to talk to ES and Kibana it is ideal to start ES and Kibana using `node scripts/functional_test_servers --config some/config/file.ts` and then use the same `--config` flag when running `node scripts/es_archiver` so that it can access the location and authorization information about the ES instace from the FTR config file.
Additional information about what functionality the CLI provides can be found by running `node scripts/es_archiver --help`
## Performance Option Parameter
We now have a performance parameter for the es-archiver#load(), entry-point
function.
This parameter is optional, with defaults:
- Batch size: 5000
- Concurrency (maximum number of bulk requests that can be active in parallel): 4
According to our benchmarks, these default settings are giving the best results in terms of loading time for
the majority of the archives.
However, there might be cases where different settings are needed, so they can be overridden when loading an archive.
### How to override the default performance settings in test files
To control the batch size and concurrency
#### Example
```typescript
await esArchiver.load('x-pack/test/functional/es_archives/getting_started/shakespeare', {
performance: {
batchSize: 300,
concurrency: 1,
},
});
```