Improves the `flushFix` behaviour for Log Rate Analysis. Previously the
setting would add a 4KB size additional dummy payload to each object
returned as ndjson. For the dataset used for testing this, this would
result in an overall response payload of ˜900Kbytes. For comparison,
without `flushFix` the response size would be ˜40Kbytes in this case.
This PR changes the behaviour to only send a dummy payload every 500ms
if the real data sent in the last 500ms wasn't bigger than 4Kbytes.
Depending on the speed of the response, this can bring down the overall
response payload to ˜300Kbytes (Cloud uncached), ˜150Kbytes (Cloud
cached) or even ˜70Kbytes (local cluster) for the same dataset.
## Summary
This PR adds UI tracking for Change Point Detection for AIOps.
- tracks type of analysis and source (where the analysis is being run
from)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Quynh Nguyen (Quinn) <43350163+qn895@users.noreply.github.com>
## Summary
<img width="973" alt="Screenshot 2023-10-17 at 11 13 20"
src="ff08efbe-dec7-4a45-ad25-17debb679864">
Passes `locale={i18n.getLocale()}` to <Settings/> Component for elastic
charts. For now it only matters for translating months, days etc in time
histogram chart, but I'd rather pass it everywhere in case if we
introduce some copy in the future.
Fixes https://github.com/elastic/kibana/issues/166240
This updates the artificial dataset generator for log rate analysis to
allow to create variants including text fields.
The artificial dataset is now used for 4 variants of functional and API
integration tests: Testing spike and dip with both with and without a
text field.
The new tests surfaced some issues that were fixed as part of this PR:
- Getting the counts of log patterns in combination with individual
significant terms ended up with to granular groups. This PR adds
additional queries to get counts for log patterns in combination with
item sets already derived from significant terms.
- The `support` value is returned by the frequent item sets agg and is
used as a threshold whether to include an item set for grouping. This
was missing from significant log patterns and is fixed by this PR.
- Adds a check to not get frequent item sets for log patterns if there
are no significant terms.
- The way we fetched log patterns using a time filter that spans the
whole of the baseline start to the deviation end caused problems with
analysing dips. This PR updates those queries to only fetch the actual
baseline and deviation time range.
- The integration tests caught an issue where we'd still fetch the
histogram for log patterns even if we'd request grouping information
only.
Fixing two issues with the log pattern analysis charts:
Occasionally the count of docs in a category can be slightly greater
than the doc count which causes the chart to render negative values when
hovering over a category in the table.
Broken:
Fixed:
---
The sparkles were incorrectly not subtracting the difference from the
doc count. So they always rendered the pattern count on top of the doc
count.
Broken:
Fixed:
Part of #167467.
Adds support for text fields in log pattern analysis. Text fields will
be analysed using log categorization, similar to log pattern analysis.
Significant log patterns will be identified using the `chi2test`
package, similar to how we detect data drifts.
We register our cases integration after checking that the license level
is platinum or trial, however if the license changes from platinum ->
basic and then back again, we attempt to register them again. This
throws an error during kibana start up.
To test, change the license level platinum/trial -> basic and then basic
-> platinum.
There will now be a warning the log `ML failed to register cases
persistable state for ml_anomaly_swimlane`
Also updates the AIOPs change point cases registration which also should
only be registered when running with a platinum or trial license.
Closes https://github.com/elastic/kibana/issues/164893
### Background
"is partial" has 2 meanings
1) Results are incomplete because search is still running
2) Search is finished. Results are incomplete because there are shard
failures (either in local or remote clusters)
[async
search](https://www.elastic.co/guide/en/elasticsearch/reference/current/async-search.html)
defines 2 flags.
1) `is_running`: Whether the search is still being executed or it has
completed
2) `is_partial`: When the query is no longer running, indicates whether
the search failed or was successfully completed on all shards. While the
query is being executed, is_partial is always set to true
**note**: there is a bug in async search where requests to only local
clusters return `is_partial:false` when there are shard errors on the
local cluster. See
https://github.com/elastic/elasticsearch/issues/98725. This should be
resolved in 8.11
Kibana's existing search implementation does not align with
Elasticsearch's `is_running` and `is_partial` flags. Kibana defines "is
partial" as definition "1)". Elasticsearch async search defines "is
partial" as definition "2)".
This PR aligns Kibana's usage of "is partial" with Elasticsearch's
definition. This required the following changes
1) `isErrorResponse` renamed to `isAbortedResponse`. Method no longer
returns true when `!response.isRunning && !!response.isPartial`. Kibana
handles results with incomplete data. **Note** removed export of
`isErrorResponse` from data plugin since its use outside of data plugin
does not make sense.
2) Replace `isPartialResponse` with `isRunningResponse`. This aligns
Kibana's definition with Elasticsearch async search flags.
3) Remove `isCompleteResponse`. The word "complete" is ambiguous. Does
it mean the search is finished (no longer running)? Or does it mean the
search has all results and there are no shard failures?
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Jatin Kathuria <jatin.kathuria@elastic.co>
Co-authored-by: Patryk Kopyciński <contact@patrykkopycinski.com>
## Summary
This PR adds tracking for Log Rate Analysis and Log Pattern Analysis
endpoints for AIOps.
- tracks type of analysis and source (where the analysis is being run
from)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Implements #153108.
This enables the
`@kbn/telemetry/event_generating_elements_should_be_instrumented` eslint
rule for the `aiops` plugin to enforce `data-test-subj` attributes on
actionable EUI components so they are auto-instrumented by telemetry.
The ids were first auto-created using `node scripts/eslint --fix
x-pack/plugins/aiops` and then adapted.
Follow up to #167237.
Part of #167467.
We plan to reuse some of the queries log pattern analysis does to use
via log rate analysis too. Log pattern analysis mostly does queries from
the client side, late rate analysis has its own API endpoint and does ES
queries via Kibana server. In preparation for the use via log rate
analysis, this moves the code we need to have available server side for
log rate analysis to the `common` area of the plugin so it can be used
both on server/client.
Removed `isServerless` flag which lived in our global context and had to
be passed about to the various components which create their own version
of the context using `getMlGlobalServices`
This PR adds a new context which contains flags for all of the features
which can be toggled when in serverless mode.
Flags added:
```
showNodeInfo
showMLNavMenu
showLicenseInfo
isADEnabled
isDFAEnabled
isNLPEnabled
```
The enabled features flags are now read from the config file client
side, rather than using capabilities.
Additional changes:
- Changes the wording of the awaiting ML node callout in serverless.
- In the search project, the default ML page is the trained models list
and not Overview
- Reenables the Memory Usage page for all projects
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes#166079.
If a user picked a saved search to investigate, the log pattern analysis
page would freeze with an infinite render loop; the log rate analysis
pate wouldn't freeze but repeatedly query for new data.
This PR fixes the issue by memoizing the queries derived from the saved
search information to avoid it being a new instance every time.
### Checklist
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
**ML Management page**
- Fixes general page loading issues.
- Ensures only enabled features are shown as tabs
- Ensures only jobs for enabled features can be exported and imported.
- Ensures only enabled features are listed in the saved object sync
output.
- On trained models tab:
- Only lists DFA models if NLP is disabled.
- Only lists non-DFA models if DFA is disabled.
**Anomaly Detection**
- Hides node information in anomaly detection jobs list.
- Hides the Exclude frozen data option in the Use full time range
selector in job wizards.
**Data frame analytics**
- Hides all node and license level information.
**Trained models**
- Only lists DFA models if NLP is disabled.
- Only lists non-DFA models if DFA is disabled.
- Hides all node and license level information.
- Hides DFA nodes
**Notifications and memory usage**
- Ensures only enabled features are mentioned. Including selectable
types in the search bar filters.
**Integrations with other plugins**
- Changes registration for integrations into other plugins so they only
happen if the relevant feature is enabled.
- Client side: UI actions, cases, embeddables, alerts, maps.
- Server side: Sample data sets, cases
**AIOPS**
- Hides the Exclude frozen data option in the Use full time range
selector on all pages
**Notes for non ML team reviewers**
**response-ops**
I've divided the
[persistable_state.ts](https://github.com/elastic/kibana/pull/163724#diff-e02dc0b6cb5b63965372b1f4a84d2287cba31a15ab525ab7983f02d09f23879f)
test into basic and trial version.
The ML cases attachments should only be registered if anomaly detection
is available in a trial or platinum license. This was a bug which I
noticed when making serverless changes.
**Observability**
I've made a few minor changes to the nav menu, fixing names of ML
features and adding the missing Change point detection AIOPs page.
**Security solution**
I've made a few minor changes to the nav menu, fixing names of ML
features and adding some missing ML features.
I think the icons being used will need to be revisited before release as
we have [official ML
icons](https://elastic.github.io/eui/#/display/icons#apps) but not for
every page. So we should probably either have new icons created or all
agree on which standard non-ML icons should be used for the ones which
are missing.
**Search**
The NLP feature is currently disabled in main, I believe this was an
attempt to stop ML anomaly detection alert rules from being registered.
I've reenabled NLP and changed the way we're registering the alerts.
They will now only be registered if the anomaly detection feature is
enabled.
Fixes https://github.com/elastic/kibana/issues/163372
`data.search` uses
[poll_search](https://github.com/elastic/kibana/blob/main/src/plugins/data/common/search/poll_search.ts)
to fetch data. `poll_search` checks for `isErrorResponse` and throws
when`isErrorResponse` is true. **Note** `searchSource.search` uses
`data.search` to perform searches.
```javascript
return from(search()).pipe(
expand(() => {
const elapsedTime = Date.now() - startTime;
return timer(getPollInterval(elapsedTime)).pipe(switchMap(search));
}),
tap((response) => {
if (isErrorResponse(response)) {
throw response ? new Error('Received partial response') : new AbortError();
}
}),
takeWhile<Response>(isPartialResponse, true),
takeUntil<Response>(aborted$)
);
```
Subscribers to `data.search` and `searchSource.search` do not need to
check for `isErrorResponse` in `next`. `poll_search` has already thrown
then the response is `isErrorResponse` so these checks are dead code
blocks that are never executed.
This PR removes these dead code blocks. Breaking this change out of
https://github.com/elastic/kibana/pull/164506 so that it can be reviewed
in isolated.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
A recent EUI change has caused a problem with the theme when using the
deprecated `toMountPoint` inside `overlays.openFlyout` to create
flyouts.
This causes the contents of the rendered flyout to not know the current
theme, this is obvious when running in dark mode.
The fix is to switch to the non-deprecated version of `toMountPoint`.
Flyouts:
Create anomaly detection job from Lens flyout in Dashboard.
Anomaly swim lane and anomaly chart job embeddables job selection flyout
in Dashboard.
Log pattern analysis flyout in Discover.
Modals:
Trained models start deployment modal.
Trained models force stop deployment modal.
Trained models stop deployment modal when there are multiple
deployments.
Misc:
Page not found banner.
Jobs list header, which contains the settings button.
DFA clone job warning toast when the original data view no longer
exists.
Components in ml's date picker package
Fixes https://github.com/elastic/kibana/issues/164379
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Dima Arnautov <arnautov.dima@gmail.com>
This updates log rate analysis to be able to auto-detect whether the
selected deviation is a spike or dip compared to the baseline time
range. To achieve this, we compare the median bucket size of the two
selections. If a dip gets detected, the analysis will then switch the
window parameters sent to the API endpoint to run the analysis.
An info callout points out the auto-selected analysis type and explains
to which time range the analysis results refer to. We need to do this to
make it clear that for dip analysis the significant terms and their doc
counts refer to the baseline time range and vice versa for spike
analysis.
Improves the table hovering behavior for log rate analysis:
- When no row is hovered, it falls back to set the first row of the
current page to a hovered state. The result is that users will always
see a comparison view in the main document count chart.
- When a row gets pinned, the hovering of the other rows will be
disabled, so the comparison view in the main document count chart gets
locked on the pinned row.
Adds support for analysing dips for log rate analysis on Observability's
alert details page.
- This removes the filter against
`Comparator.GT/Comparator.GT_OR_EQ` which hides log rate analysis for
log threshold alerts. Instead an `analysisType` is passed on to log rate
analysis: When analysing dips the baseline and deviation ranges will be
switched for both the analysis and the brush labels on the document
count chart.
- Updates all references to "spikes" only in UI text, e.g. the AI
Assistant prompt.
Uses the recently created [category validation
package](https://github.com/elastic/kibana/pull/161261) to perform
validation on the field selected for pattern analysis.
If the field is considered unsuitable for categorization, a warning
callout is displayed which lists the reasons it is unsuitable.
If the field is suitable, no callout is displayed.
Other changes:
- Adds the selected field to the URL state, so it is remembered on page
refresh.
- If no field is in the URL, it will look for a field called `message`
in the data view and auto select it.
- renames the ML route `/jobs/categorization_field_examples` to
`/jobs/categorization_field_validation` as it is a more accurate name
and it's consistent with the newly added route in AIOPs.
**Log Pattern Analysis page in ML**
**Log Pattern Analysis flyout in Discover**
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
- Originally Kibana's `http` service did not support receiving streams,
that's why we used plain `fetch` for this. This has been fixed in
#158678, so this PR updates the streaming helpers to use Kibana's `http`
service from now on.
- The PR also breaks out the response stream code into its own package
and restructures it to separate client and server side code. This brings
down the `aiops` bundle size by `~300KB`! 🥳
- The approach to client side throttling/buffering was also revamped:
There was an issue doing the throttling inside the generator function,
it always waited for the timeout. The buffering is now removed from
`fetchStream`, instead `useThrottle` from `react-use` is used on the
reduced `data` in `useFetchStream`. Loading log rate analysis results
got a lot snappier with this update!
## Summary
fixes https://github.com/elastic/kibana/issues/161441
fixes https://github.com/elastic/kibana/issues/161464
The recent `EuiButtonEmpty`/`EuiButtonIcon` Emotion conversions have
highlighted a CSS order/specificity flaw in Kibana compared to EUI - EUI
orders its Sass _after_ its Emotion styles (see
https://elastic.github.io/eui/#/), and Kibana orders Sass _before_
Emotion styles.
I'm not totally sure why Greg set up Kibana's style order the way he did
in https://github.com/elastic/kibana/pull/134919, but at this point, EUI
has enough of its baseline atom components converted to Emotion that
remaining components in Sass are all generally molecules or higher level
components that need to come after Emotion.
### QA
- [x] Test as many pages in Kibana as possible to ensure no visual
regressions 🤞
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Adds functional tests for the Log Pattern Analysis AIOPs page and
Discover integration.
ML app:
Selects a message field, checks category counts are correct, selects
bother "filter in" and "filter out" buttons which redirect to Discover
and apply a filter. Checks the total doc counts in discover match the
selected category.
In Discover:
Selects a message field which opens the flyout, checks category counts
are correct, selects bother "filter in" and "filter out" buttons which
close the flyout and apply a filter. Checks the total doc counts in
discover match the selected category.
- Closes https://github.com/elastic/kibana/issues/149336
## Summary
This PR converts `unifiedFieldList` plugin into a new
`@kbn/unified-field-list` package.
Had to also move some deps:
- from `uiActions` plugin to the existing `@kbn/ui-actions-browser`
package
- from `data` plugin to a new `@kbn/data-service` package
Please test that Field Stats from the package are still working on your
pages.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Adds the check for non-time series data views to the Log Pattern
Analysis.
As all three AIOPs features now have the same check, I moved it to a
common component.
Part of https://github.com/elastic/kibana/issues/159355
Adds a secondary sort by `doc_count` only if the results table is sorted
by `p-value`. This is useful in cases when there are lots of items with
the same `p-value`. The combination of high significance (low `p-value`)
and sorted by `doc_count` ascending will surface the most unique
field/value combinations.
To allow for more fine grained control of the baseline and deviation
time ranges for `autoAnalysisStart` this PR allows to pass in a
`WindowParameters` object as an alternative to the plain timestamp. When
more useful metadata is available this might lead to better selections
then the default one provided by `getWindowParameters`.
## Summary
- Saved searches use content management api
- Saved search plugin provides proper plugin api instead of static
exports
- Discover no longer re-exports static content from saved search plugin
- ML no longer works directly with the saved search saved object
- saved object conflicts are thrown via the api, rather than the api
consumer
- Content management api logs failed requests a bit better, helpful for
tests failing in CI
Closes https://github.com/elastic/kibana/issues/157078
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Adds versioning to the AIOps API.
Versions are added to the server side routes and to the client side
functions which call the routes.
Updates API tests to add the API version to the request headers.
The single API endpoint is already internal and now has been given the
version '1'.
**Internal APIs**
`/internal/aiops/explain_log_rate_spikes`
## Summary
With the release of Per User Dark Mode, code should no longer rely on
calling `uiSettings` to determine which theme Kibana is displayed with.
With theme settings now configurable from User Profiles and Adv.
Settings, the code that was calling uiSettings to determine the Kibana
theme will not take into account which theme is currently being
displayed.
Applies an appropriate EUI theme based on the profile settings. In
particular for the following components:
- Anomaly swim lane (Fixes#158155 )
- Job tree map view (Fixes
https://github.com/elastic/kibana/issues/158304)
- Charts-related theme settings, e.g. the Single Metric Viewer
## Summary
`eui@77.2.2` ⏩ `eui@79.0.1`
🦴 The primary changes in this upgrade are around the deprecated
`EuiLoadingContent` being removed in favor of `EuiSkeletonText`.
- Most instances have been a [direct swap of
usage](327626a7be),
but [some replacements were a bit more
opinionated](e6ceb36a75)
as I saw them as potential to take advantage of `EuiSkeletonText`'s
syntactical sugar and screen reader announcements for when state
switches to loaded.
---
## [`79.0.1`](https://github.com/elastic/eui/tree/v79.0.1)
**Bug fixes**
- Fixed broken push `EuiFlyout` behavior
([#6764](https://github.com/elastic/eui/pull/6764))
## [`79.0.0`](https://github.com/elastic/eui/tree/v79.0.0)
- Updated all `EuiSkeleton` components with new props that allow for
more control over screen reader live announcements:
`announceLoadingStatus`, `announceLoadedStatus`, and `ariaLiveProps`
([#6752](https://github.com/elastic/eui/pull/6752))
- Improved keyboard accessibility in `EuiPageHeader` by ensuring the
right side menu items come into focus from left to right.
([#6753](https://github.com/elastic/eui/pull/6753))
**Breaking changes**
- Removed deprecated `EuiLoadingContent`. Use the `EuiSkeleton`
components instead. ([#6754](https://github.com/elastic/eui/pull/6754))
## [`78.0.0`](https://github.com/elastic/eui/tree/v78.0.0)
- Improved the contrast ratio of `EuiCheckbox`, `EuiRadio`, and
`EuiSwitch` in their unchecked states to meet WCAG AA guidelines.
([#6729](https://github.com/elastic/eui/pull/6729))
- Added React Testing Library `*ByTestSubject` custom commands to
`within()`. RTL utilities can be imported from
`@elastic/eui/lib/test/rtl`.
([#6737](https://github.com/elastic/eui/pull/6737))
- Updated `EuiAvatar` to support a new letter `casing` prop that allow
customizing text capitalization
([#6739](https://github.com/elastic/eui/pull/6739))
- Updated `EuiFocusTrap` to support the `gapMode` prop configuration
(now defaults to `padding`)
([#6744](https://github.com/elastic/eui/pull/6744))
**Bug fixes**
- Fixed inconsistency in `EuiSearchBar`'s AND/OR semantics between DSL
and query string generation
([#6717](https://github.com/elastic/eui/pull/6717))
- Fixed `EuiFieldNumber`'s native browser validity detection causing
extra unnecessary rerenders
([#6741](https://github.com/elastic/eui/pull/6741))
- Fixed the `scrollLock` property on `EuiFocusTrap` (and other
components using `EuiFocusTrap`, such as `EuiFlyout` and `EuiModal`) to
no longer block scrolling on nested portalled content, such as combobox
dropdowns ([#6744](https://github.com/elastic/eui/pull/6744))
**Breaking changes**
- `EuiAvatar`s with the default `user` type will now default to
capitalizing all initials in uppercase
([#6739](https://github.com/elastic/eui/pull/6739))
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
