mirror of
https://github.com/elastic/kibana.git
synced 2025-04-24 17:59:23 -04:00
Your window into the Elastic Stack
64426b2b4d
## Summary
Implements an exponential backoff retry strategy when the LLM API throws
rate limit (`429`) errors.
### Backoff implementation
- The `run` method from the `RuleMigrationsTaskClient` has been moved to
the new `RuleMigrationTaskRunner` class.
- The settings for the backoff are defined in this class with:
```ts
/** Exponential backoff configuration to handle rate limit errors */
const RETRY_CONFIG = {
initialRetryDelaySeconds: 1,
backoffMultiplier: 2,
maxRetries: 8,
// max waiting time 4m15s (1*2^8 = 256s)
} as const;
```
- Only one rule will be retried at a time, the rest of the concurrent
rule translations blocked by the rate limit will await for the API to
recover before attempting the translation again.
```ts
/** Executor sleep configuration
* A sleep time applied at the beginning of each single rule translation in the execution pool,
* The objective of this sleep is to spread the load of concurrent translations, and prevent hitting the rate limit repeatedly.
* The sleep time applied is a random number between [0-value]. Every time we hit rate limit the value is increased by the multiplier, up to the limit.
*/
const EXECUTOR_SLEEP = {
initialValueSeconds: 3,
multiplier: 2,
limitSeconds: 96, // 1m36s (5 increases)
} as const;
```
### Migration batching changes
```ts
/** Number of concurrent rule translations in the pool */
const TASK_CONCURRENCY = 10 as const;
/** Number of rules loaded in memory to be translated in the pool */
const TASK_BATCH_SIZE = 100 as const;
```
#### Before
- Batches of 15 rules were retrieved and executed in a `Promise.all`,
requiring all of them to be completed before proceeding to the next
batch.
- A "batch sleep" of 10s was executed at the end of each iteration.
#### In this PR
- Batches of 100 rules are retrieved and kept in memory. The execution
is performed in a task pool with a concurrency of 10 rules. This ensures
there are always 10 rules executing at a time.
- The "batch sleep" has been removed in favour of an "execution sleep"
of rand[1-3]s at the start of each single rule migration. This
individual sleep serves two goals:
- Spread the load when the migration is first launched.
- Prevent hitting the rate limit consistently: The sleep duration is
increased every time we hit a rate limit.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
|
||
|---|---|---|
| .buildkite | ||
| .devcontainer | ||
| .github | ||
| api_docs | ||
| config | ||
| dev_docs | ||
| docs | ||
| examples | ||
| kbn_pm | ||
| legacy_rfcs | ||
| licenses | ||
| oas_docs | ||
| packages | ||
| plugins | ||
| scripts | ||
| src | ||
| test | ||
| typings | ||
| x-pack | ||
| .backportrc.json | ||
| .bazelignore | ||
| .bazeliskversion | ||
| .bazelrc | ||
| .bazelrc.common | ||
| .bazelversion | ||
| .browserslistrc | ||
| .editorconfig | ||
| .eslintignore | ||
| .eslintrc.js | ||
| .gitattributes | ||
| .gitignore | ||
| .i18nrc.json | ||
| .node-version | ||
| .npmrc | ||
| .nvmrc | ||
| .prettierignore | ||
| .prettierrc | ||
| .puppeteerrc | ||
| .stylelintignore | ||
| .stylelintrc | ||
| .telemetryrc.json | ||
| .yarnrc | ||
| BUILD.bazel | ||
| catalog-info.yaml | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| FAQ.md | ||
| fleet_packages.json | ||
| github_checks_reporter.json | ||
| kibana.d.ts | ||
| LICENSE.txt | ||
| NOTICE.txt | ||
| package.json | ||
| preinstall_check.js | ||
| README.md | ||
| renovate.json | ||
| RISK_MATRIX.mdx | ||
| run_fleet_setup_parallel.sh | ||
| SECURITY.md | ||
| sonar-project.properties | ||
| STYLEGUIDE.mdx | ||
| tsconfig.base.json | ||
| tsconfig.browser.json | ||
| tsconfig.browser_bazel.json | ||
| tsconfig.json | ||
| TYPESCRIPT.md | ||
| updatecli-compose.yaml | ||
| versions.json | ||
| WORKSPACE.bazel | ||
| yarn.lock | ||
Kibana
Kibana is your window into the Elastic Stack. Specifically, it's a browser-based analytics and search dashboard for Elasticsearch.
- Getting Started
- Documentation
- Version Compatibility with Elasticsearch
- Questions? Problems? Suggestions?
Getting Started
If you just want to try Kibana out, check out the Elastic Stack Getting Started Page to give it a whirl.
If you're interested in diving a bit deeper and getting a taste of Kibana's capabilities, head over to the Kibana Getting Started Page.
Using a Kibana Release
If you want to use a Kibana release in production, give it a test run, or just play around:
- Download the latest version on the Kibana Download Page.
- Learn more about Kibana's features and capabilities on the Kibana Product Page.
- We also offer a hosted version of Kibana on our Cloud Service.
Building and Running Kibana, and/or Contributing Code
You might want to build Kibana locally to contribute some code, test out the latest features, or try out an open PR:
- CONTRIBUTING.md will help you get Kibana up and running.
- If you would like to contribute code, please follow our STYLEGUIDE.mdx.
- For all other questions, check out the FAQ.md and wiki.
Documentation
Visit Elastic.co for the full Kibana documentation.
For information about building the documentation, see the README in elastic/docs.
Version Compatibility with Elasticsearch
Ideally, you should be running Elasticsearch and Kibana with matching version numbers. If your Elasticsearch has an older version number or a newer major number than Kibana, then Kibana will fail to run. If Elasticsearch has a newer minor or patch number than Kibana, then the Kibana Server will log a warning.
Note: The version numbers below are only examples, meant to illustrate the relationships between different types of version numbers.
| Situation | Example Kibana version | Example ES version | Outcome |
|---|---|---|---|
| Versions are the same. | 7.15.1 | 7.15.1 | 💚 OK |
| ES patch number is newer. | 7.15.0 | 7.15.1 | ⚠️ Logged warning |
| ES minor number is newer. | 7.14.2 | 7.15.0 | ⚠️ Logged warning |
| ES major number is newer. | 7.15.1 | 8.0.0 | 🚫 Fatal error |
| ES patch number is older. | 7.15.1 | 7.15.0 | ⚠️ Logged warning |
| ES minor number is older. | 7.15.1 | 7.14.2 | 🚫 Fatal error |
| ES major number is older. | 8.0.0 | 7.15.1 | 🚫 Fatal error |
Questions? Problems? Suggestions?
- If you've found a bug or want to request a feature, please create a GitHub Issue. Please check to make sure someone else hasn't already created an issue for the same topic.
- Need help using Kibana? Ask away on our Kibana Discuss Forum and a fellow community member or Elastic engineer will be glad to help you out.