## Summary
This PR enhances support for ES|QL data visualizer. Changes include:
- Add an Update button that when clicked, will update and run the query.
This is to complement the current cmd + Enter keyboard short cut.
5ca3ac0b-782e-404c-a04b-330c8eea6ab7
- Improve logic to no longer fetch total count & document count if only
the limit size is updated (so changing the limit size, but not the query
or time, will not refresh the count chart again)
- Remove dependency from data view's field format
- Refactor into a data fetching & processing into common hook to be used
for embeddable
- Support ES|QL in Field stats embeddable
- Fix count % of documents where field exists is > 100% when there are
multi-field values. (E.g. when row is an array of values like ["a", "b",
"c"], the count is much higher than the total number of rows)
<img width="1492" alt="Screenshot 2024-02-09 at 12 48 13"
src="c437e4f9-10e4-4d26-b00a-57277c5e1287">
- Add support for `geo_point` and `geo_shape` field types
<img width="1492" alt="Screenshot 2024-02-09 at 12 47 37"
src="d72e1e73-9880-4a12-be65-29b569d80694">
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR adds support for ES|QL queries in Data visualizer.
<img width="1695" alt="Screenshot 2024-01-26 at 17 07 59"
src="8a54b859-60d6-4c47-b3dd-e5f3ed43b6b0">
<img width="1695" alt="Screenshot 2024-01-26 at 17 12 39"
src="32fd08e4-7f3b-43e6-81a7-7ec4e777bac0">
a3f540e9-461d-4ebc-bd69-de4ffa2bc554
### Changes:
- Add a new card from the Data visualizer main page
- Add a link from the ML navigation
<img width="1717" alt="Screenshot 2024-01-08 at 18 03 50"
src="832f7890-4ce6-44c1-ab87-cde01f4bf1c0">
- Added a new button to Use ES|QL
<img width="1714" alt="Screenshot 2024-01-09 at 11 23 09"
src="a38a9360-6691-4f3b-a824-8481ab543250">
- Support for **keyword**, **text**, **numeric**, **boolean**, **date**,
and **ip** fields
<img width="1714" alt="Screenshot 2024-01-09 at 11 24 38"
src="b122ee5c-1500-4e2b-9434-e64b0b6ea3be">
<img width="1441" alt="Screenshot 2024-01-09 at 11 25 25"
src="eb35ee78-8a34-467e-84da-2026b01fcda1">
<img width="969" alt="Screenshot 2024-01-09 at 11 44 02"
src="d0f9947d-2b2c-4c14-89ba-9fc5d0a2bf64">
<img width="981" alt="Screenshot 2024-01-10 at 12 01 42"
src="aa5a8d44-7447-41fc-a544-d1b626bf8bce">
- Default to user's fieldFormats for fields that are dynamic generated
by ES|QL, else use Data view's format
- Default to Data view's setting (e.g. type `bytes` in this case for
field `bytes_normal_counter`)
<img width="1037" alt="Screenshot 2024-01-10 at 12 10 38"
src="9fb7e31c-f397-4209-a463-e1a43fe27ffd">
- Default to user's fieldFormats formatting for dynamically generated
fields (e.g. type `number` in this case for field `avg_price`)
<img width="1283" alt="Screenshot 2024-01-10 at 12 01 03"
src="acc25358-50bb-4237-9476-86067ef0badf">
- Add a new UI control to allow users to limit analysis to 5,000 -
10,000 - 100,000 - 1,000,000, rows. This speeds up fetching of the stats
for big data sets and avoid potential circuit breaking exceptions.
- Break overall stats request into smaller parallel requests (which
prevent time out or payload too big due by too many fields), at 10
requests at a time
- Break field stats for individual fields into more efficient batches
(which prevent time out or payload too big due by too many fields), at
10 requests at at ime
- Improve error handling by propagating up the error AND the ES|QL
request in both the UI and the developer's console (for better
debugging)
- Improve error handling in field stats rows: If one field, or a group
of fields, say 'keyword' fields fail to fetch for some reasons, it will
show error for that field but not affect all other fields.
<img width="1690" alt="Screenshot 2024-01-26 at 16 04 28"
src="6e240e12-76b4-42d6-b3be-c05342d76df9">
- Add deep linking in the top search bar
<img width="1185" alt="Screenshot 2024-01-26 at 16 56 49"
src="4f24df68-edc5-41c5-b2ed-d6150ba1e20b">
- More robust support for keyword fields with geo data
<img width="1438" alt="Screenshot 2024-01-26 at 16 55 01"
src="3b97925b-ca28-4952-8082-8d3242e3cb3f">
### Todos:
- [x] Add earliest/latest for date time fields -> Current blocker:
escape special characters in esql variable names
- [x] Fix formatting of numbers for dynamic query, where we don't know
the formatting based on the data view
- [x] Fix date time 'Update' not updating until Refresh is clicked
- [x] Better optimization to not fetch distribution & expanded row
content for pages that are not visible
### Good to have:
- [ ] Investigate bringing back the +/- filter buttons (either by
modifying the ES|QL query directly or by adding separate DSL filters?)
------------
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Several enhancements to the error toast notifications in the ML plugin
to improve error reporting.
The bulk of the changes are to add 'See the full error' buttons to the
toasts allowing the user to see further details on the error that has
occurred. Also makes minor edits to some of the error messages to
improve clarity.
Also closes#171839 by changes to
`x-pack/plugins/ml/public/application/jobs/jobs_list/components/utils.js`
so that the the 'Jobs started successfully' toast is only shown if 1 or
more jobs have been started successfully.
Fixes#171839
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
- Closes https://github.com/elastic/kibana/issues/172570
## Summary
This PR creates a new package `@kbn/visualization-utils` and moves
`getTimeZone` helper into it. Also the PR removes duplicates of other
similar helpers.
And the histogram in the field popover has now the same time zone
configuration as the the main hits histogram:
<img width="500" alt="Screenshot 2023-12-06 at 18 46 25"
src="2d350c91-1a41-419a-9d80-f203f1c90327">
## For testing
Change `dateFormat:tz` in Advanced Settings and check if histograms are
rendered accordingly.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes https://github.com/elastic/kibana/issues/171243. This PR adds
field `_tier` to the list of omit fields to not show or display. This is
especially relevant when `_tier` is added in the list of meta fields in
Kibana.
Steps to reproduce:
1. In Advanced settings, add `_tier` to the list of meta fields. This
will show _tier as a field across Kibana if data has a tier applied.
<img width="976" alt="image"
src="86ecbbba-c574-42f6-97cf-c465ec334d7e">
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Consolidates UI elements and backend code to create/delete data views
and destination indices related to transforms and data frame analytics.
We ended up with two different approaches for creating data views in the
wizards for transforms and data frame analytics, the original reason was
we were not aware of the `allowNoIndex: true` setting and worked around
that in different ways.
This PR aligns UI workflows and moves related code to a new package
`@kbn/ml-data-view-utils` for data views and
`@kbn/ml-creation-wizard-utils` for the destination index form. The
latter might be used for other shared components across wizard..
In Data Frame Analytics, the checkbox to create a data view was removed
from the last "Create" step, instead the option to create a data view
was moved to the "Details" step.
In Transforms, the UI component to create the destination index was
brought over from DFA where there is a switch option to automatically
use the job ID as the name for the destination index by default.
Moves the categorize field uiAction trigger and action and related items
to the AIOps/ML uiActions package.
ML and AIOps are adding more and more uiActions, and so it's nicer to
have them all in one package.
Also cleans up the registration of the uiActions in the AIOps plugin
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
- Renames references to index patterns to data views in function and
variable names.
- Some inconsistent naming of schemas for data frame analytics was
cleaned up as part of this PR.
- Note this doesn't cover the whole ml owned codebase but just code
related to data frame analytics.
Support to restore baseline/deviation time ranges from url state on full
page refresh. Also updates functional tests to include a full page refresh after the
first analysis run for each dataset.
Adds the ability to quickly create a categorisation anomaly detection
job from the pattern analysis flyout.
Adds a new `created_by` ID `categorization-wizard-from-pattern-analysis`
which can be picked up by telemetry.
Creates a new package for sharing our AIOPs ui actions IDs. I think we
should move the pattern analysis ID to this package too, but that can be
done in a separate PR.
51349f93-f072-4983-85f0-98741902fb5a
6e618581-8916-4e63-930f-945c96c25e6c
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This refactors the route handler of the log rate analysis API endpoint.
So far this route handler contained a lot of logic and was growing past
900+ lines with every new feature we worked on. This PR changes it so
the route handler can walk through the analysis steps on a higher level.
`define_route.ts:defineRoute()` is the outer most wrapper that's used to
define the route and its versions. It calls
`route_handler_factory:routeHandlerFactory()` for each version.
The route handler sets up
`response_stream_factory:responseStreamFactory()` to create the response
stream and then walks through the steps of the analysis.
The response stream factory acts as a wrapper to set up the stream
itself, the stream state (for example to set if it's running etc.), some
custom actions on the stream as well as analysis handlers that fetch
data from ES and pass it on to the stream.
Log rate analysis now supports both keywords and log patterns derived
from text fields. The type `SignificantTerm` originally was used for the
results of the `significant_terms` agg to get the p-values for keyword
fields. Since it's now used for both cases (keyword fields and log
patterns) this PR renames the type and related variables etc. to
`SignificantItem` (we used the wording `item` already in some cases in
the context of groups).
This PR uses [conditional
types](https://www.typescriptlang.org/docs/handbook/2/conditional-types.html)
to allow the handling of both multiple API versions within one route handler. The more tricky bit turned out
to be not the updated request body, but the response since it is an
NDJSON stream where some messages were updated. In this case also the
functions that create these messages were updated with conditional types
to be able to create a message that fits the definition of the API
version.
The API integration tests originally had these message identifiers in
the `expected` section of their `testData`. I changed that to use helper
functions that retrieve the expected messages from the stream according
to the expected version. All API integration tests are run on both
versions. The functional tests are run only on the newer version since
the UI is expected to work with version `2` only.
Improves the `flushFix` behaviour for Log Rate Analysis. Previously the
setting would add a 4KB size additional dummy payload to each object
returned as ndjson. For the dataset used for testing this, this would
result in an overall response payload of ˜900Kbytes. For comparison,
without `flushFix` the response size would be ˜40Kbytes in this case.
This PR changes the behaviour to only send a dummy payload every 500ms
if the real data sent in the last 500ms wasn't bigger than 4Kbytes.
Depending on the speed of the response, this can bring down the overall
response payload to ˜300Kbytes (Cloud uncached), ˜150Kbytes (Cloud
cached) or even ˜70Kbytes (local cluster) for the same dataset.
## Summary
<img width="973" alt="Screenshot 2023-10-17 at 11 13 20"
src="ff08efbe-dec7-4a45-ad25-17debb679864">
Passes `locale={i18n.getLocale()}` to <Settings/> Component for elastic
charts. For now it only matters for translating months, days etc in time
histogram chart, but I'd rather pass it everywhere in case if we
introduce some copy in the future.
Fixes https://github.com/elastic/kibana/issues/166240
Part of #167467.
Adds support for text fields in log pattern analysis. Text fields will
be analysed using log categorization, similar to log pattern analysis.
Significant log patterns will be identified using the `chi2test`
package, similar to how we detect data drifts.
## Summary
This PR adds tracking for Log Rate Analysis and Log Pattern Analysis
endpoints for AIOps.
- tracks type of analysis and source (where the analysis is being run
from)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
The `chi2test` utils so fare were only used within data comparison view.
We plan to use it with other plugins, so moving it so a separate package
in this PR. `SIGNIFICANCE_LEVELS` was updated to include some more
digits.
Removed `isServerless` flag which lived in our global context and had to
be passed about to the various components which create their own version
of the context using `getMlGlobalServices`
This PR adds a new context which contains flags for all of the features
which can be toggled when in serverless mode.
Flags added:
```
showNodeInfo
showMLNavMenu
showLicenseInfo
isADEnabled
isDFAEnabled
isNLPEnabled
```
The enabled features flags are now read from the config file client
side, rather than using capabilities.
Additional changes:
- Changes the wording of the awaiting ML node callout in serverless.
- In the search project, the default ML page is the trained models list
and not Overview
- Reenables the Memory Usage page for all projects
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR adds data drift detection workflow from Trained models to Data
comparison view. It also renames Data comparison to Data Drift.
**From the new map view in Trained model list:**
- Clicking on the index icon in the map view will give an option/action
to Analyze data drift
a68163ab-8a83-4378-8cf3-ea49f4480a06
- If model has detected related indices, it will also give an option to
Analyze data drift in the Transform actions
**From the data comparison/drift page:**
- Default screen with list of available data views and saved search will
be shown
<img width="1470" alt="Screen Shot 2023-09-07 at 00 22 01"
src="db13b8b7-9d90-4220-b03e-9f9d12ab53e9">
- But can also customize index patterns for the data sets to analyze.
Upon 'analyzing', a new data view will be created if needed (either
permanently or temporarily).
<img width="1271" alt="Screen Shot 2023-08-29 at 16 56 57"
src="e000e920-162b-4369-8762-70b6244e50e7">
<img width="1470" alt="Screen Shot 2023-09-07 at 00 22 49"
src="6577a530-c3b0-4ab9-95e4-d1d8fd1c9f0a">
- If there exists a data view with exact combination of index patterns
and time field, it will use that data view
- If there exists a data view with the same index patterns but different
time field, it will create a new data view with name
`{referencePattern},{comparisonPattern}-{timeField}`
- If no data view exists that matches, it will create a new data view
with name `{referencePattern},{comparisonPattern}`
## For reviewers:
- **appex-sharedux**: [Small change in the exported type interface for
BaseSavedObjectFinder](https://github.com/elastic/kibana/pull/162853/files#diff-5e2e62df8aba5ac9445962bfa00eee933a386110d0a24dfe6ac0f300a796ccc3)
to correctly list `children` as an accepted prop. This prop which is
used for the `toolsRight`.
- **security-solution**: Renaming of `Data comparison` to `Data Drift`
## Tests:
[Flaky test suite runner with Data Drift
test](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3216#018accc2-d33b-4cd6-a178-589e6698b675)
... successful after 50 runs✅
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Adds a shared service for elastic curated models. The first use case is
to provide a default/recommended ELSER version based on the hardware of
the current cluster.
#### Why?
In 8.11 we'll provide a platform-specific version of the ELSER v2
alongside the portable one. At the moment several solutions refer to
ELSER for download/inference purposes with a `.elser_model_1` constant.
Starting 8.11 the model ID will vary, so using the `ElastcModels`
service allows retrieving the recommended version of ELSER for the
current cluster without any changes by solution teams in future
releases. It is still possible to request an older version of the model
if necessary.
#### Implementation
- Adds a new Kibana API endpoint `/trained_models/model_downloads` that
provides a list of model definitions, with the `default` and
`recommended` flags.
- Adds a new Kibana API endpoint `/trained_models/elser_config` that
provides an ELSER configuration based on the cluster architecture.
- `getELSER` method is exposed from the plugin `setup` server-side as
part of our shared services and plugin `start` client-side.
### Checklist
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
**ML Management page**
- Fixes general page loading issues.
- Ensures only enabled features are shown as tabs
- Ensures only jobs for enabled features can be exported and imported.
- Ensures only enabled features are listed in the saved object sync
output.
- On trained models tab:
- Only lists DFA models if NLP is disabled.
- Only lists non-DFA models if DFA is disabled.
**Anomaly Detection**
- Hides node information in anomaly detection jobs list.
- Hides the Exclude frozen data option in the Use full time range
selector in job wizards.
**Data frame analytics**
- Hides all node and license level information.
**Trained models**
- Only lists DFA models if NLP is disabled.
- Only lists non-DFA models if DFA is disabled.
- Hides all node and license level information.
- Hides DFA nodes
**Notifications and memory usage**
- Ensures only enabled features are mentioned. Including selectable
types in the search bar filters.
**Integrations with other plugins**
- Changes registration for integrations into other plugins so they only
happen if the relevant feature is enabled.
- Client side: UI actions, cases, embeddables, alerts, maps.
- Server side: Sample data sets, cases
**AIOPS**
- Hides the Exclude frozen data option in the Use full time range
selector on all pages
**Notes for non ML team reviewers**
**response-ops**
I've divided the
[persistable_state.ts](https://github.com/elastic/kibana/pull/163724#diff-e02dc0b6cb5b63965372b1f4a84d2287cba31a15ab525ab7983f02d09f23879f)
test into basic and trial version.
The ML cases attachments should only be registered if anomaly detection
is available in a trial or platinum license. This was a bug which I
noticed when making serverless changes.
**Observability**
I've made a few minor changes to the nav menu, fixing names of ML
features and adding the missing Change point detection AIOPs page.
**Security solution**
I've made a few minor changes to the nav menu, fixing names of ML
features and adding some missing ML features.
I think the icons being used will need to be revisited before release as
we have [official ML
icons](https://elastic.github.io/eui/#/display/icons#apps) but not for
every page. So we should probably either have new icons created or all
agree on which standard non-ML icons should be used for the ones which
are missing.
**Search**
The NLP feature is currently disabled in main, I believe this was an
attempt to stop ML anomaly detection alert rules from being registered.
I've reenabled NLP and changed the way we're registering the alerts.
They will now only be registered if the anomaly detection feature is
enabled.
Fixes https://github.com/elastic/kibana/issues/163372
A recent EUI change has caused a problem with the theme when using the
deprecated `toMountPoint` inside `overlays.openFlyout` to create
flyouts.
This causes the contents of the rendered flyout to not know the current
theme, this is obvious when running in dark mode.
The fix is to switch to the non-deprecated version of `toMountPoint`.
Flyouts:
Create anomaly detection job from Lens flyout in Dashboard.
Anomaly swim lane and anomaly chart job embeddables job selection flyout
in Dashboard.
Log pattern analysis flyout in Discover.
Modals:
Trained models start deployment modal.
Trained models force stop deployment modal.
Trained models stop deployment modal when there are multiple
deployments.
Misc:
Page not found banner.
Jobs list header, which contains the settings button.
DFA clone job warning toast when the original data view no longer
exists.
Components in ml's date picker package
Fixes https://github.com/elastic/kibana/issues/164379
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Dima Arnautov <arnautov.dima@gmail.com>
## Summary
There's a few EUI imports out there that folks are reaching into
`@elastic/eui/lib/` for (which doesn't contain any types - something I'm
looking into separately) that could instead be imported at the top
`@elastic/eui` level, which is properly typed.
### Checklist
N/A - types only
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This updates log rate analysis to be able to auto-detect whether the
selected deviation is a spike or dip compared to the baseline time
range. To achieve this, we compare the median bucket size of the two
selections. If a dip gets detected, the analysis will then switch the
window parameters sent to the API endpoint to run the analysis.
An info callout points out the auto-selected analysis type and explains
to which time range the analysis results refer to. We need to do this to
make it clear that for dip analysis the significant terms and their doc
counts refer to the baseline time range and vice versa for spike
analysis.
Adds support for analysing dips for log rate analysis on Observability's
alert details page.
- This removes the filter against
`Comparator.GT/Comparator.GT_OR_EQ` which hides log rate analysis for
log threshold alerts. Instead an `analysisType` is passed on to log rate
analysis: When analysing dips the baseline and deviation ranges will be
switched for both the analysis and the brush labels on the document
count chart.
- Updates all references to "spikes" only in UI text, e.g. the AI
Assistant prompt.
Fixes a race condition in the case where a response stream finishes and
sets `isRunning` to `false`, but `useThrottle` didn't trigger it's last
update yet within the refresh rate. In the case of log rate analysis,
`isRunning` could be set to `false` too early and the UI wouldn't
consider later throttled updates (for example, setting `loaded=1` which
would result in inconsistent UI state).
The fix in this case is to return the unthrottled raw data instead of
the throttled one as soon as the stream finished.
Uses the recently created [category validation
package](https://github.com/elastic/kibana/pull/161261) to perform
validation on the field selected for pattern analysis.
If the field is considered unsuitable for categorization, a warning
callout is displayed which lists the reasons it is unsuitable.
If the field is suitable, no callout is displayed.
Other changes:
- Adds the selected field to the URL state, so it is remembered on page
refresh.
- If no field is in the URL, it will look for a field called `message`
in the data view and auto select it.
- renames the ML route `/jobs/categorization_field_examples` to
`/jobs/categorization_field_validation` as it is a more accurate name
and it's consistent with the newly added route in AIOPs.
**Log Pattern Analysis page in ML**
**Log Pattern Analysis flyout in Discover**
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
- Originally Kibana's `http` service did not support receiving streams,
that's why we used plain `fetch` for this. This has been fixed in
#158678, so this PR updates the streaming helpers to use Kibana's `http`
service from now on.
- The PR also breaks out the response stream code into its own package
and restructures it to separate client and server side code. This brings
down the `aiops` bundle size by `~300KB`! 🥳
- The approach to client side throttling/buffering was also revamped:
There was an issue doing the throttling inside the generator function,
it always waited for the timeout. The buffering is now removed from
`fetchStream`, instead `useThrottle` from `react-use` is used on the
reduced `data` in `useFetchStream`. Loading log rate analysis results
got a lot snappier with this update!
