Your window into the Elastic Stack
Find a file
Walter Rafelsberger c8c27c4820
[ML] AIOps: Chunk groups of field candidates into single queries. (#188137)
## Summary

Part of #187684.

So far we ran individual queries for each field candidate to get
significant items. The historic reason is that we've been uber cautious
not to run into issues with the `max_buckets` setting. But since we
fetch the top 1k items and the `max_buckets` default is 65k it should be
safe to change that.

This PR updates fetching significant items to combine multiple field
candidates within one query using multiple aggs. The current setting in
this PR is now to add up to 50 field candidates into a single query.
This will result in up to ~50k buckets (50 x 1k buckets for the sig
terms agg plus 50 buckets for the cardinality aggs). If there's more
field candidates, we'll still make use of the async queue where we do up
to 5 queries in parallel.

The result is that for example for 200 field candidates we'll just do 4
queries instead of 200 previously.

Previous:

<img width="1624" alt="image"
src="https://github.com/user-attachments/assets/1e11ff1c-a0c2-4dcf-9399-27456439faad">


![aiops-log-rate-analysis-apm-0001](https://github.com/user-attachments/assets/67b6337e-a406-45bc-bb49-85ad047fcbe8)

After:

<img width="1554" alt="image"
src="https://github.com/user-attachments/assets/33ccb9ef-fe5b-4945-a87f-77347ba097ea">


### Checklist

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
2024-07-17 18:04:39 +02:00
.buildkite [Security Solution] Enable Detections API bundling in build pipeline (#188436) 2024-07-17 14:04:50 +01:00
.github [i18n][system upgrade] Upgrade i18n tooling (#186519) 2024-07-16 21:47:54 +01:00
api_docs [api-docs] 2024-07-17 Daily api_docs build (#188499) 2024-07-17 05:08:27 +00:00
config [Index Management] Disable data stream stats in serverless (#186420) 2024-06-21 13:45:23 -07:00
dev_docs [Docs] Added callout to docs team to OAS tutorial (#187750) 2024-07-08 17:30:10 +02:00
docs [Fields Metadata] Add metadata fields static source (#188453) 2024-07-17 11:47:40 +02:00
examples Improve features plugin's contract type names (#187944) 2024-07-17 15:33:32 +01:00
kbn_pm chore(NA): remove usage of re2 and replace it with a non native module (#188134) 2024-07-15 20:33:28 +01:00
legacy_rfcs rename @elastic/* packages to @kbn/* (#138957) 2022-08-18 08:54:42 -07:00
licenses build: remove requirement to clone open-source repo (#180715) 2024-04-15 15:10:46 -05:00
oas_docs [OAS] Better support for enums (#188198) 2024-07-16 10:49:48 +01:00
packages [Discover] Add log level badge cell renderer for logs profile (#188281) 2024-07-17 12:46:02 -03:00
plugins
scripts [i18n][system upgrade] Upgrade i18n tooling (#186519) 2024-07-16 21:47:54 +01:00
src [Discover] Add log level badge cell renderer for logs profile (#188281) 2024-07-17 12:46:02 -03:00
test [Discover] Add log level badge cell renderer for logs profile (#188281) 2024-07-17 12:46:02 -03:00
typings Remove legacy kibana react code editor (#171047) 2024-01-05 14:35:09 +01:00
x-pack [ML] AIOps: Chunk groups of field candidates into single queries. (#188137) 2024-07-17 18:04:39 +02:00
.backportrc.json chore(NA): adds 8.16 into backportrc (#187530) 2024-07-04 19:09:25 +01:00
.bazelignore Remove references to deleted .ci folder (#177168) 2024-02-20 19:54:21 +01:00
.bazeliskversion chore(NA): upgrade bazelisk into v1.11.0 (#125070) 2022-02-09 20:43:57 +00:00
.bazelrc chore(NA): use new and more performant BuildBuddy servers (#130350) 2022-04-18 02:01:38 +01:00
.bazelrc.common Transpile packages on demand, validate all TS projects (#146212) 2022-12-22 19:00:29 -06:00
.bazelversion chore(NA): revert bazel upgrade for v5.2.0 (#135096) 2022-06-24 03:57:21 +01:00
.browserslistrc Add Firefox ESR to browserlistrc (#184462) 2024-05-29 17:53:18 -05:00
.editorconfig .editorconfig MDX files should follow the same rules as MD (#96942) 2021-04-13 11:40:42 -04:00
.eslintignore [ES|QL] New @kbn/esql-services package (#179029) 2024-03-27 14:39:48 +01:00
.eslintrc.js [GenAI][Integrations] UI for the custom integration creation with AI (#186304) 2024-06-21 19:19:05 +02:00
.gitattributes
.gitignore [Moving] Move APM and APM_Data_Access folders into /x-pack/observability_solution/ (#177433) 2024-02-23 09:56:21 -07:00
.i18nrc.json [ES|QL] Rename text-based-languages pugin to esql plugin (#187520) 2024-07-10 00:16:32 +10:00
.node-version Upgrade Node.js to 20.15.1 (#187791) 2024-07-15 12:34:07 -05:00
.npmrc [npmrc] Fix puppeteer_skip_download configuration (#177673) 2024-02-22 18:59:01 -07:00
.nvmrc Upgrade Node.js to 20.15.1 (#187791) 2024-07-15 12:34:07 -05:00
.prettierignore
.prettierrc
.puppeteerrc Add .puppeteerrc (#179847) 2024-04-03 09:14:39 -05:00
.stylelintignore chore(NA): stop grouping bazel out symlink folders (#96066) 2021-04-01 14:16:14 -05:00
.stylelintrc Bump stylelint to ^14 (#136693) 2022-07-20 10:11:00 -05:00
.telemetryrc.json [Telemetry] Fix telemetry-tools TS parser for packages (#149819) 2023-01-31 04:09:09 +03:00
.yarnrc chore(NA): manage npm dependencies within bazel (#92864) 2021-03-03 12:37:20 -05:00
BUILD.bazel Transpile packages on demand, validate all TS projects (#146212) 2022-12-22 19:00:29 -06:00
catalog-info.yaml [CI] Remove kme leftovers (take 2) (#187947) 2024-07-10 16:01:06 +02:00
CODE_OF_CONDUCT.md
CONTRIBUTING.md Update doc slugs to improve analytic tracking, move to appropriate folders (#113630) 2021-10-04 13:36:45 -04:00
FAQ.md Fix small typos in the root md files (#134609) 2022-06-23 09:36:11 -05:00
fleet_packages.json [main] Sync bundled packages with Package Storage (#186399) 2024-06-18 11:41:16 -07:00
github_checks_reporter.json
kibana.d.ts fix all violations 2022-04-16 01:37:30 -05:00
LICENSE.txt
nav-kibana-dev.docnav.json Adds link to ESO developer documentation in nav (#187867) 2024-07-10 11:50:26 +02:00
NOTICE.txt Copy assets from appropriate directory for kbn-monaco (#178669) 2024-03-21 16:29:20 +01:00
package.json [Security Solution] Adds diff algorithm and unit tests for multi-line string fields (#188022) 2024-07-17 11:45:36 -04:00
preinstall_check.js Always throw error objects - never strings (#171498) 2023-11-20 09:23:16 -05:00
README.md [README] Update version Compatibility with Elasticsearch (#116040) 2022-01-10 10:31:21 -05:00
renovate.json chore(NA): remove usage of re2 and replace it with a non native module (#188134) 2024-07-15 20:33:28 +01:00
RISK_MATRIX.mdx Add "Risk Matrix" section to the PR template (#100649) 2021-06-02 14:43:47 +02:00
run_fleet_setup_parallel.sh [Fleet] Prevent concurrent runs of Fleet setup (#183636) 2024-05-31 16:38:51 +02:00
SECURITY.md
sonar-project.properties [ci] Run sonarqube daily (#173961) 2024-01-03 15:43:29 -06:00
STYLEGUIDE.mdx [styleguide] update path to scss theme (#140742) 2022-09-15 10:41:14 -04:00
tsconfig.base.json [Security Solution] Add missing Exceptions API OpenAPI specifications (#185951) 2024-07-15 13:12:56 +02:00
tsconfig.browser.json
tsconfig.browser_bazel.json [build_ts_refs] improve caches, allow building a subset of projects (#107981) 2021-08-10 22:12:45 -07:00
tsconfig.json Transpile packages on demand, validate all TS projects (#146212) 2022-12-22 19:00:29 -06:00
TYPESCRIPT.md Fix small typos in the root md files (#134609) 2022-06-23 09:36:11 -05:00
versions.json chore(NA): update versions after v8.14.4 bump (#188120) 2024-07-11 16:49:56 +01:00
WORKSPACE.bazel chore(NA): remove usage of re2 and replace it with a non native module (#188134) 2024-07-15 20:33:28 +01:00
yarn.lock [Security Solution] Adds diff algorithm and unit tests for multi-line string fields (#188022) 2024-07-17 11:45:36 -04:00

Kibana

Kibana is your window into the Elastic Stack. Specifically, it's a browser-based analytics and search dashboard for Elasticsearch.

Getting Started

If you just want to try Kibana out, check out the Elastic Stack Getting Started Page to give it a whirl.

If you're interested in diving a bit deeper and getting a taste of Kibana's capabilities, head over to the Kibana Getting Started Page.

Using a Kibana Release

If you want to use a Kibana release in production, give it a test run, or just play around:

Building and Running Kibana, and/or Contributing Code

You might want to build Kibana locally to contribute some code, test out the latest features, or try out an open PR:

Documentation

Visit Elastic.co for the full Kibana documentation.

For information about building the documentation, see the README in elastic/docs.

Version Compatibility with Elasticsearch

Ideally, you should be running Elasticsearch and Kibana with matching version numbers. If your Elasticsearch has an older version number or a newer major number than Kibana, then Kibana will fail to run. If Elasticsearch has a newer minor or patch number than Kibana, then the Kibana Server will log a warning.

Note: The version numbers below are only examples, meant to illustrate the relationships between different types of version numbers.

Situation Example Kibana version Example ES version Outcome
Versions are the same. 7.15.1 7.15.1 💚 OK
ES patch number is newer. 7.15.0 7.15.1 ⚠️ Logged warning
ES minor number is newer. 7.14.2 7.15.0 ⚠️ Logged warning
ES major number is newer. 7.15.1 8.0.0 🚫 Fatal error
ES patch number is older. 7.15.1 7.15.0 ⚠️ Logged warning
ES minor number is older. 7.15.1 7.14.2 🚫 Fatal error
ES major number is older. 8.0.0 7.15.1 🚫 Fatal error

Questions? Problems? Suggestions?

  • If you've found a bug or want to request a feature, please create a GitHub Issue. Please check to make sure someone else hasn't already created an issue for the same topic.
  • Need help using Kibana? Ask away on our Kibana Discuss Forum and a fellow community member or Elastic engineer will be glad to help you out.