@maximpn brought up the issues caused by the types required by the
rison-node package, which attempted to communicate that "encoded values
must be primitive values, or recursive arrays/object of primitive
values". This isn't actually expressible in TypeScript, which lead to
many instances of `rison.encode(value as unknown as RisonValue)` which
is useless. Additionally, the rison-node library actually supports any
value and will either produce valid rison or `undefined` for that value.
To address this I'm adding a wrapper function which accepts `any` and
returns a `string`. If rison-node is totally unable to produce any rison
for the value (because the value is `undefined` or some other type like
Symbol or BigInt) the `encode()` function will throw. If you're
accepting arbitrary input you can use the `encodeUnknown()` function,
which will return a string or undefined, if the value you provided has
zero rison representation.
Like JSON.stringify() any non-circular primitive, object, or array can
be encoded with either function. If the values within those objects are
not encodable (functions, RegExps, etc) then they will be skipped. Any
object/array with the `toJSON()` method will be converted to JSON first,
and if the prototype of the object has the `encode_rison()` method it
will be used to convert he value into rison.
The changes in this PR are mostly updating usage of rison-node to use
`@kbn/rison` (which is also enforced by eslint). There are also several
changes which remove unnecessary casting.
The analysis can be long running and in cases can time out depending on
server/proxy settings. This update allows a user to try to continue the
analysis if it failed half way through.
Adds random sampling as an option to `fetchFieldHistograms()`. Since this already supported normal sampling, this also adds a check if only one of the two options is used. Pending a future update we might remove support for normal sampling in a follow up. The Explain Log Rate Spikes analysis makes use of the new option and applies random sampling for the mini histogram data it fetches.
The queries for p-values and histograms were done in chunks of 10 parallel queries. The drawback with this approach was that if just one of these 10 queries was a lot slower, we'd still have to wait for it to finish before we could start the next chunk. This PR replaces the chunking approach with an async concurrent queue of up to 10 queries. The difference is that as soon as the first of the 10 first queries finishes, we can start another query and don't have to wait for the slower ones to finish.
For this PR the `async` library is added to `package.json`, however it's not a completely new library being added since it was already used as a dependency of other packages we use in Kibana.
Applying the overall params like the time range to the histogram queries was missing. This fixes it by creating getHistogramQuery that can be applied when fetching the histogram for overall data, individual field/value histograms and group histograms.
Renames `fetchFieldCandidates()` to `fetchIndexInfo()` which now returns total document count and sampling probability in addition to field candidates. Previously sampling probability was only determined within the `frequent_items` fetcher. This refactor makes it reusable for other queries and we can apply it to the `p-value` aggregations. Adding it to fetching the histogram data will be done in another follow up.
Limits the fields we pass on to the frequent_items aggregation to 15. This is a trade off between speed and quality of the grouping result. The amount of fields we pass on to the agg grow the time it takes to get frequent items more than linearly and we risk timeouts with more fields.
So far we passed on the abort signal from the client to possibly cancel the analysis, but the signal was not passed on to the ES queries to cancel those. That means the analysis could be cancelled after each step but it did not cancel ES queries that were already running. This PR takes the already existing abort signal and passes it on to all ES queries.
This surfaced an issue with running too many queries in parallel: We didn't have a limit so far when fetching the histogram data. With using the abort signals, Kibana would report a warning if more than 10 queries were run at once. The PR updates fetching histogram data to also do it in chunks of 10 queries like we already do for the p-value aggregation. So the larger bulk of the file diff is the result of wrapping the histogram queries inside a for of to iterate over the chunks of queries.
In the analysis part we created chunks of 10 calls for the p-value aggregations, but individual calls of these chunks were then still called one after the other. This PR changes the behaviour to run calls for each chunk in parallel.
This wraps the analysis part in a try/catch block. We had individual try/catch blocks already to handle ES queries. Because the analysis is run async, if another error happens it would trigger an unresolved promise error and could crash Kibana server. Adding this additional try/catch block avoids this.
* fix chart showing as empty when filter matches field/value pair in hovered row
* use both overall and split buckets to get timerange
* always pass along split stats as they are up to date
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
- Adds a flag for `compressResponse` and `flushFix` to the request body to be able to overrule compression settings inferred from headers.
- Updates the developer examples with a toggle to run requests with compression enabled or disabled.
- Adds support for backpressure handling for response streams.
- The backpressure update includes a fix where uncompressed streams would never start streaming to the client.
- The analysis endpoint for Explain Log Rate Spikes now includes a ping every 10 seconds to keep the stream alive.
- Integration tests were updated to test both uncompressed and compressed streaming.
- Fixes error handling that before was not providing enough information for debugging purposes and support. This will now output more fine grained error information to the Kibana server log. The analysis is now more resilient to errors for individual queries. For example, we don't stop the analysis anymore if individual queries for p-values or histograms fail.
- Moves the error callout above all other possible elements like empty prompts when the analysis doesn't return results.
Fix loading state messages for grouping:
- Fixes the progress bar messages to include the grouping step.
- If progress stalls, it might look to the user like the analysis got stuck when there are steps like the grouping that take a longer time. This updates the progress bar to show an animated background as long as the analysis is running. When the analysis finishes or gets canceled the animated background gets disabled and shows a static background.
* [ML] Improving log pattern analysis messaging
* moving to separate component
* translations
* fixing unselection when time range changes
* changes based on review
- Fixes styling of hovered and pinned rows to use EUI provided variables.
- The above was also done for the Log Pattern Analysis page to fix an issue with dark theme.
- Fixes unpinning a row for field/value pairs.
- Fixes pinning/unpinning for groups.
* add hover main chart sync at group level
* [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'
* fix overall request with groups
* [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'
* ensure all pairs in group are considered in query
* add alignment to log rate column
* fix types
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
- Change point groups might miss individual change points that were not returned by the `frequent_items` agg as part of groups. This PR now adds each missing one as an individual additional group.
- Only return groups if there's at least one group with more than one item, otherwise fall back to basic table with significant terms.
- Changes the UI behaviour to show the regular table by default and the grouping switch set to off.
- Adds `p-value` column to grouped table and defaults to sorting by that column similar to table with indidivual items.
This fixes some inconsistencies in versions used for `d3-*` and `@types/d3-*`. As part of this, related packages have been updated to the latest non-ESM-based versions. (The very latest d3 package versions were updated to be ESM only which we cannot use yet in the current Kibana server side setup).
Note that while this includes major version bumps, the d3 packages were bumped because of changes in JS language support and because they dropped IE support, not because of breaking API changes in the modules themselves.
- Updates `d3-array` from `1.2.4` to `2.12.1`. Removed usage of the package's `min/max` function to use native `Math.min/Math.max` instead.
- Fix `@types/d3-interpolate` to match version `3.0.1`.
- Updates `d3-scale` from `2.2.2` to `3.3.0`. Fixes version mismatch with `@types/d3-scale`.
- Updates `d3-shape` from `1.1.0` to `2.1.0`. Fixes version mismatch with `@types/d3-shape`.
- Updates `d3-time` from `1.1.0` to `2.1.1`. Fixes version mismatch with `@types/d3-time`.
- Removes `@types/d3-time-format` because the package `d3-time-format` is no longer used.
* [ML] Log caegorization UI POC
* improvements
* code structure changes
* further refactoring
* fixes after merge with main
* fixes after merging with main
* adding table pagination
* catching category request errors
* small refactor
* fixes after merge with main
* further fixes are merge with main
* showing errors in toast
* updating breadcrumbs
* translations and removing unused files
* changing case of side nav items
* translations
* updating actions
* fixing unused variable
* translations
* adding comment
* small changes based on review
* improving search request type
* capitalizing discover
* changes based on review
* fixing chart tooltip
* fixing examples from fields containing dots
* fixing breadcrumbs
* [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
- Tweaks the `frequent_items` agg config. `should_minimum_match:2` for the terms should clause improves getting back groups of results. `minimum_set_size:2` will avoid single item frequent sets.
- Passes on a possible filter bar query to the `frequent_items` agg.
* wip: create groups table
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* show significant terms data in groups expanded row
* update mock data to reflect new data format
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* sort group fields alphabetically
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* replace mock data with api data
* fix functional test and remove commented code
* update types
* update functional tests
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This is a temporary fix for response streaming. The current cloud environment buffers each stream with chunks up to 4KB. To force trigger flushing, we send along a 4KB dummy payload to trigger an update. This fixes a stale loading bar for Explain Log Rate Spikes. Once the cloud environment's proxy has been updated to support flushing below the 4KB threshold, we can remove this fix again.
Extends the `/internal/aiops/explain_log_rate_spikes` with an option to extend the analysis and summarize significant field/value pairs into groups using the frequent_items aggregation.
Improves dependency handling. Previously we had a mix of plugin dependencies and kibana context values passed on from the ML plugin. Now we have a single context that gets dependencies passed on from the parent container in the ML plugin.
Fixes histogram styling, some alignments with discover chart.
- Switch BarSeries to HistogramBarSeries.
- Fix too generic IDs.
- Align y axis ticks settings with Discover chart.
- Fixes horizontal margins.
- Fixes x domain to avoid empty buckets at start and end of histogram.
* Update all usages of EuiPageTemplate to EuiPageTemplate_Deprecated
* EuiPageContent_Deprecated as EuiPageContent
* EuiPageContentBody_Deprecated as EuiPageContentBody
* EuiPageContentHeader_Deprecated as EuiPageContentHeader
* EuiPageContentHeaderSection_Deprecated as EuiPageContentHeaderSection
* EuiPageSideBar_Deprecated as EuiPageSideBar
* EuiPageContent__Deprecated to EuiPageContent_Deprecated
* Fix rogue semi-colons
* WIP: NoDataConfigPage & NoDataPage converted to new template
- `withSolutionNav` not yet handled
* WIP: KibanaPageTemplateInner converted to use new template
- Pushes existing `pageHeader` prompts onto created EuiPageTemplate.PageHeader
- Uses `isEmptyState` to push `pageHeader` props to EuiPageTemplate.EmptyPrompt instead (if `children` are not supplied)
* WIP: `withSolutionNav` now renders the sidebar content properly
- Collapsing isn’t working (minWidth isn’t updating)
* Fixing stickiness of sidebar
* [Security] Fixed SecuritySolutionTemplateWrapper’s usage
- Moved `bottomBar` to EuiPageTemplate.BottomBar (now contained in just the page contents)
- Change EuiPanel children wrapper with EuiPageTemplate.Section
* [O11y] Wrap `children` with EuiPageTemplate.Section
* Fix getting_started usage
* WIP: Fixing types
* Removing `template` pass through
* Set EUI to 63.0.0
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* More import fixes
* Sidebar component update
* Expand `KibanaPageTemplate` to all namespaced EUI counterparts
- Updated `docs/tutorials` mdx page
- Fixed SolutionNav prop types
* Updated the tutorial mdx page
* [Stack Management] Updated app layout to new template
- Some temporary props applied for BWC until all other pages can be converted
- Converted `API Keys` page’s layout (and especially prompt usags) to new paradigm
* Fix circular dep
* Fix new circular dependency
- copying and pasting types from KibanaPageTemplateProps, but ah well
* [Security Solution] Remove `template` prop - no longer a prop on Kibana/EuiPageTemplate
* [O11y] Allow customizing EuiPageTemplate.Section wrapper
- converts pageBodyProps
- fixes non-centered loading template
* [Enterprise Search] Update page templates
- fix layouts by auto-wrapping an EuiPageSection for padding, while adding a `customPageSections` prop for more custom sections/layouts
- re-center 404 errors
- update tests
* Update KibanaPageTemplate tests
* Update snapshots
* Fix FTR test with removed EUI classNames
* Fix FTR tests with changed kbn classNames
* Update failing dashboard snapshots
- drop shadow changed slightly in EUI
* Fix failing Security Cypress test
* [O11y] Fix Inventory page using deprecated CSS hooks
* [O11y][Uptime] Fix missing bottom bars
- Modifies ObservabilityPageTemplate to accept a `bottomBar` prop (a la the old EuiPageTemplate behavior)
NOTE: This opinionated page layout structure is starting to feel like it could be potentially limiting / have all the same pitfalls the previous EuiPageTemplate did. If so, consider something closer to the Enterprise Search page template conversion (`customPageSections`).
- Misc cleanup: Use `KibanaPageTemplate` over `EuiPageTemplate`
* [O11y] Fix route template typing
- Since theObservabilityPageTemplate is using the new Eui/KibanaPageTemplate, its child templates and types need to be updated accordingly
* Fix broken minWidth behavior
- was an EUI issue that required a patch release
+ update snapshots
* [Security Solution] Type fixes, restore empty state
- Fix empty state logic removed in a previous commit
- bogarts KibanaPageTemplate's `isEmptyState` prop instead of using `template="noData"`
- extend template wrappers to past ...rest to underlying Kibana/EuiPageTemplate
+ replace EuiPageTemplate with KibanaPageTemplate for consistency
* Fix failing synthetics selector
* Grab EUI v63.0.6
- for deprecation tags and section tag support
* Fix Kibana Overview plugin layout
- needs to use KibanaPageTemplate.Section to get padding back
- use `bottomBorder` prop over horizontal rules
- restore previous page color via panelled=false
* Convert Home plugin to new KibanaPageTemplate
- use KibanaPageTemplate.Section instead to preserve page width/paddings
- use `bottomBorder` instead of `EuiHorizontalRule`
- NOTE: This causes margins to decrease slightly from xxl to xl (largest padding available for EuiPageSection) - this can be restored by CSS overrides if desired
- update CSS to preserve previous looks, + convert to logical properties
* [O11y] Fix non-centered empty/loading states
* [O11y] Restore subdued background on various empty state prompts
* [O11y] Fix all instances of views that require a scrollable full-height child
+ restore comment for inventory view
* [O11y][ux] Fix broken sidebar
- The entire app was missing a wrapping EuiProvider, and as such breakpoint utils were not working, and the sidebar was missing
+ misc cleanup
- remove unnecessary fragment
- remove role="main" attr - now that EuiPageTemplate sets a `main` tag, they'll conflict
- add isEmptyState to center loading component
* [APM Cypress tests] harden flaky test
* [APM Cypress tests] Fix failing Cypress test, again
Co-authored-by: cchaos <caroline.horn@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Adds first functional tests for Explain Log Rate Spikes. The test clicks the menu item, selects an index, clicks the "Use full data" button and asserts the page's elements.
