Closes#188641
## Summary
This PR adds new formulas for rx and tx metrics for hosts. In inventory
we show the old metrics as legacy and the new ones with the old metrics
labels (this affects only hosts):
<img width="1788" alt="image"
src="https://github.com/user-attachments/assets/d3e5bf26-e521-4ff8-b00b-1d78eebd56f9">
All old alerts should work - The only difference is that it will show
the metric as "Legacy" and it still can be used in the rules. The hosts
view and the lens charts are using a new formula
## Testing
- Check the network metrics in the inventory / alert flyout (both the
new ones and the old ones)
- Check the network metrics and charts in the hosts view (only the new
ones should be available)
https://github.com/user-attachments/assets/886fd5a0-858c-458b-9025-eb55913b1932https://github.com/user-attachments/assets/7752939f-f693-4021-bf23-89e264ef0c2d
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Carlos Crespo <crespocarlos@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Part of #187684.
So far the popover to filter fields was only available when grouping was
enabled. This PR updates the behavior so it's available all the time and
can be used to exclude field candidates from the analysis. If we detect
the index to be based on an ECS schema, we auto-select a set of
predefined fields.
Changes in this PR:
- Creates a new route
`/internal/aiops/log_rate_analysis/field_candidates` to be able to fetch
field candidates independent of the main streaming API call.
- Fixes the code to consider "remaining" field candidates to also
consider text field candidates. This was originally developed to allow
to continue an analysis that errored for some reason. We use that option
to also pass on the custom field list from the field selection popover.
- Fetching the field candidates is done in a new redux slice
`logRateAnalysisFieldCandidatesSlice` using an async thunk.
- Filters the list of field candidates by a predefined field of allowed
fields when an ECS schema gets detected.
- Renames `fieldCandidates` to `keywordFieldCandidates` for clearer
distinction against `textFieldCandidates`.
- Refactors `getLogRateAnalysisTypeForCounts` args to a config object.
- Bump the API version for the full log rate analysis to version 3. We
missed bumping the version in
https://github.com/elastic/kibana/pull/188648. This update manages
proper versioning between v2 and v3, also the API integration tests
cover both versions.
[aiops-log-rate-analysis-fields-filter-0001.webm](https://github.com/user-attachments/assets/e3ed8d5b-f01c-42ef-8033-caa7135b8cc0)
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
At the moment, our package generator creates all packages with the type
`shared-common`. This means that we cannot enforce boundaries between
server-side-only code and the browser, and vice-versa.
- [x] I started fixing `packages/core/*`
- [x] It took me to fixing `src/core/` type to be identified by the
`plugin` pattern (`public` and `server` directories) vs. a package
(either common, or single-scoped)
- [x] Unsurprisingly, this extended to packages importing core packages
hitting the boundaries eslint rules. And other packages importing the
latter.
- [x] Also a bunch of `common` logic that shouldn't be so _common_ 🙃
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
closes [#189064](https://github.com/elastic/kibana/issues/189064)
## Summary
This PR changes Lens embeddable to support creating custom error
messages by using the new `customBadgeMessages` prop.
<img width="500px" alt="image"
src="https://github.com/user-attachments/assets/d48d86fa-b4a7-4dd5-ad84-9fe4bf144672">
The custom error message will only be displayed in the context of the
hosts view and asset details, which is where this new prop is being
used. Everywhere else will display the default error handling provided
by Lens
<img width="500px" alt="image"
src="https://github.com/user-attachments/assets/38ecaee9-5f25-4d34-85e4-f176095982c5">
### How to test
- Start a local kibana and es instances
- Run `node scripts/synthtrace infra_hosts_with_apm_hosts --live`
- Change the the index patter in Settings to `metrics-apm`
- Navigate to Infrastructure > Hosts View and open the asset details
flyout
- The missing field message should be displayed as the screenshot above
- Open a chart in lens
- The default Lens message will be shown
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Fixes#189056, #189057, #164623
Bring back the window handler for other tests correctly.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes: https://github.com/elastic/observability-dev/issues/3371
## Description
The Obs Alert Rule Detail view has a card that is clickable with a
focusable element inside it. This is a confusing paradigm and prevents
keyboard users from filtering by all alerts because it's not focusable.
It would be better to make the two alert number widgets the focusable
elements. Screenshot attached below.
PR is based on the following comment posted by @1Copenut in
https://github.com/elastic/observability-dev/issues/3371#issuecomment-2129446431_
> @alexwizp Agreed, panels should not be focusable. The highlighted
panel is clickable, and that was unexpected. I could click the entire
panel, and click the "1 Active now" text to filter by all alerts or
active alerts in the table below.
>
> It would be better to have the "All alerts" text be clickable and
focusable, and keep the "1 Active now" clickable and focusable. That way
the two text blocks have the interactive behavior, while the panel
(card) is just a container.
### Steps to recreate
1. Open the [Obs
Alerts](https://keepserverless-qa-oblt-b4ba07.kb.eu-west-1.aws.qa.elastic.cloud/app/observability/alerts)
table
2. Click the "Manage Rules" link
3. Create a new rule and verify it appears in the Rules table
4. Click on the rule name to load the Rule Detail view
6. Verify the `1 Active Now`
### What was done?:
1. The click event was **REMOVED** from the panel and has been moved to
`All alerts.`
2. `aria-describedby` attributes were added for `AllAlertCounts` and
`ActiveAlertCounts`
3. `h3` attributes were replaced to `EuiTitle` in `AllAlertCounts` and
`ActiveAlertCounts`
Closes https://github.com/elastic/kibana/issues/174959
## Summary
This PR converts the Saved Search embeddable to the new React embeddable
framework. There should not be **any** changes in user-facing behaviour
(except for the intentional change described
[here](https://github.com/elastic/kibana/pull/180536#discussion_r1647924825))
- therefore, testing of this PR should be focused on ensuring that no
behaviour is changed and/or broken with this refactor.
> [!WARNING]
> The saved search embeddable takes **many forms** and so, while I tried
my best to test everything thoroughly, it is very, very likely that I
missed some test cases due to not being the expert in this area. It is
important that @elastic/kibana-data-discovery in particular approaches
this PR review with a fine-tooth comb 🙇 Thanks so much.
### Notes about the embeddable state:
As part of this refactor, I made three significant changes to how the
state is managed:
1. Once the embeddable is being built in `buildEmbeddable`, the **only
difference** between the runtime state of a by reference and a by value
panel is that the by reference one will have three saved object-specific
keys: `savedObjectId`, `savedObjectDescription`, and `savedObjectTitle`.
2. Number 1 made it possible for me to "flatten out" the runtime state
of the embeddable by removing the `attributes` key, which makes it
easier to access the pieces of state that you need.
3. Previously, the `savedSearch` element of the Saved Search embeddable
object was never modified; instead, changes made to the columns, sort,
sample size, etc. from the dashboard were stored in `explicitInput`.
This essentially created two sources of truth.
With the new embeddable system, we only ever want **one** source of
truth - so, the saved search is now modified **directly** when making
changes from the dashboard. However, in order to keep behaviour
consistent with the old embeddable, changes made from the dashboard to a
by reference saved search **should not** modify the underlying saved
object (this behaviour will have to change if we ever want inline
editing for saved searches, but that is another discussion) - therefore,
when **serializing** the runtime state (which happens when the dashboard
is saved), we [only serialize state that has **changed** from the
initial
state](https://github.com/elastic/kibana/pull/180536/files#diff-7346937694685b85c017fb608c6582afb3aded0912bfb42fffa4b32a6d27fdbbR93-R117);
then, on deserialization, we take this "changed" state and
[**overwrite** the state of the saved search with
it](https://github.com/elastic/kibana/pull/180536/files#diff-7346937694685b85c017fb608c6582afb3aded0912bfb42fffa4b32a6d27fdbbR44-R54).
Note that this **only** applies to by reference saved searches - with by
value saved searches, we don't have to worry about this and can freely
modify the state.
I also had to make changes to how the **search source** is stored in
runtime state. Previously, when initializing the embeddable, fetching
the saved search saved object also created and returned an
**unserializable** search source object. However, in the new system,
runtime state **most always be serializable** (see
https://github.com/elastic/kibana/pull/186052) - therefore, I've had to
instead use the **serialized** search source in my runtime state saved
search - therefore, I've had to make changes to `toSavedSearch` method
to [allow for a **serializable** saved search to be
returned](https://github.com/elastic/kibana/pull/180536/files#diff-3baaeaeef5893a5a4db6379a1ed888406a8584cb9d0c7440f273040e4aa28166R160-R169).
| | Runtime state (`input`) before | Runtime state after |
|--------|--------|--------|
| **By value** |
|
|
| **By reference** |
|
|
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
- https://github.com/elastic/kibana/issues/187630
- https://github.com/elastic/kibana/issues/187768
These changes fix the error on saving the alert
> An error occurred during rule execution: message: "[1:6952] failed to
parse field [event.original] of type [keyword] in document with id
'330b17dc2ac382dbdd2f2577c28e83b42c5dc66eaf95e857ec0f222abfc486fa'..."
The issue happens when source index has non-ECS compliant text field
which is expected to be a keyword. If the text value is longer than
32766 bytes and keyword field does not have ignore_above parameter set,
then on trying to store the text value in keyword field we will hit the
Lucene's term byte-length limit (for more details see [this
page](https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-above.html)).
See the main ticket for steps to reproduce the issue.
---------
Co-authored-by: Vitalii Dmyterko <92328789+vitaliidm@users.noreply.github.com>
The PR attempts to fix the flakiness in the e2e tests by avoiding clicks
on an already opened popover. The click statement within
`retry.tryForTime` can be called in succession, which could
inadvertently close the popover, which we want to avoid in this case.
The screenshot from failed tests suggests that the assertion is made on
a closed down popover:
## Summary
Closes https://github.com/elastic/kibana/issues/184871
This PR adds a check for if the license is disabled for reporting and
does not show the Export tab in the share modal. It might be good to
have a message in the export tab to show the users that they need to
update their license but that might need some feedback from
@elastic/kibana-design. This can be accomplished in another PR but this
PR is just to avoid the nasty error to the users who might be in this
situation.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
part of [3628](https://github.com/elastic/observability-dev/issues/3628)
- private
## Summary
After adding 20 items, users can no longer add more fields and will see
the message below
<img width="1725" alt="image"
src="fd504212-0e0f-485d-a8fe-b991c829950e">
### Extra
- There was an unused and duplicate `metrics_explorer` route in infra. I
removed it. It should've been removed when the `metrics_data_access`
plugin was created.
- Cleaned up `constants` field in `metrics_data_access` and `infra`
plugins
### How to test
- Start a local Kibana instance pointing to an oblt cluster
- Navigate to Infrastructure > Metrics Explorer
- Try to select more than 20 fields in the metrics field
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
When user selected fields in query mode, goes to chat mode and then back
to query mode. Some fields may return to default value
---------
Co-authored-by: Joseph McElroy <joseph.mcelroy@elastic.co>
## Summary
* delete underlying trained model during `after all` clean up
* handle request time out error when creating inference endpoint
Tested against QA deployment and locally.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes#187007
Hides ML embeddables from the "Add panel" flyout when
1. ML feature isn't available for the user role
2. ML is hidden in a current space
### How to test
1. Create a custom role with disabled ML privilege and assign it to a
user
2. Remove ML feature visibility in a current space
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Updates to unified field list on typing are debounced - this way we
don't get so many updates when typing in the search input.
Flaky test runner:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6424
## Performance comparison
Test: typing the string: activem for metricbeat data (~6000 fields)
before (costly update on every keystroke):
<img width="669" alt="Screenshot 2024-06-28 at 17 28 38"
src="7075f7bc-2d90-4177-acac-69ac101b2ef1">
after (only one costly update when user stops typing):
<img width="269" alt="Screenshot 2024-06-28 at 17 24 43"
src="8c0ce4a3-7c1a-428b-a482-f6b4d87911e0">
## Summary
Ticket https://github.com/elastic/kibana/issues/187384
These changes fix the error on saving the alert
> An error occurred during rule execution: message: "[1:6778] failed to
parse field [kibana.alert.original_event.action] of type [keyword] in
document with id '027b925ae2799635a0dee97a6aa9d58dc87d9771'."
which happens due to not stripping non-ECS compliant sub-fields of the
`event.action` field.
See the main ticket for steps to reproduce the issue.
## Summary
This PR is a prerequisite to the Locator Implementation for Logs
Explorer - https://github.com/elastic/kibana/pull/186287
## Problem Statement
- Integrations were fetched when the main DQ page loads and stored in
the State Machine. This means when the Flyout Opens, it was referencing
already fetched data from the main page, updating the URL and then that
was used to render certain sections on the Flyout. This causes issues as
when a Locator is used to directly open the Flyout from some other page.
In that case everything happen asynchronously causing the data to be not
present when the flyout open thus those integration sections were not
present.
## Solution
- Now when the flyout is opened or is already open, it reads the basic
params from the URL like `DataStream`. With this information, it make
API call to fetch Integration information and thus making it
independent.
- Does this means you duplicated the Logic to fetch Integrations ? Yes
and No. Logic has to be duplicated as Flyout is moving to its own page
very soon. This means it would anyhow not be able to re-use that
Integration Information available. Secondly the duplication is not one
to one, its more catered towards Flyout logic
- Split the state machine to make Integration Calls only when the opened
Dataset is actually an integration. This is done by chaining the
respective states after the `DataStreamSettings` state confirms presence
of Integration.
## What else has been done
- Type cleaning: A lot of types has to be refactored to make this
change. Also simplified some duplicate types. We were using
- Runtime types
- Types Derived from Runtime Types
- Inferred Types from API Responses
We don't need the 3rd one. 1 and 2 and sufficient.
## Summary
After improving the synthtrace data creation for containers we were able
to add more specific tests for container view, the aim of this spacetime
is to add some improvements to hosts so we can in the future use
synthtrace for testing
### What was done
First I thought that adding `event.dataset` was needed to get the
metadata, or make the request work, as I did for containers, but in
containers was needed not because of the metadata query itself but the
integration check to know if we need to display k8s or docker metrics.
I simplified the scenarios and data generation in the tests, adding the
metadata fields we need in the synthtrace clients for host and docker
and k8s containers, the values of the metadata fields doesn't need to
change for different scenarios, so it's ok to have them set in the
client.
## Summary
This PR introduces Alert Suppression for ML Detection Rules. This
feature is behaviorally similar to alerting suppression for other
Detection Engine Rule types, and nearly identical to the analogous
features for EQL rules.
There are some additional UI behaviors introduced here as well, mainly
intended to cover the shortcomings discovered in
https://github.com/elastic/kibana/issues/183100. Those behaviors are:
1. Populating the suppression field list with fields from the anomaly
index(es).
1. Disabling the suppression UI if no selected ML jobs are running
(because we cannot populate the list of fields on which they'll be
suppressing).
1. Warning the user if _some_ selected ML jobs are not running (because
the list of suppression fields may be incomplete).
See screenshots below for more info.
### Intermediate Serverless Deployment
As per the "intermediate deployment" requirements for serverless, while
the schema (and declared alert SO mappings) will be extended to allow
this functionality, the user-facing features are currently hidden behind
a feature flag. Once this is merged and released, we can issue a "final"
deployment in which the feature flag is enabled, and the feature
effectively released.
## Screenshots
* Overview of new UI fields
<img width="1044" alt="Screenshot 2024-05-16 at 3 22 02 PM"
src="8c07700d-5860-4d1e-a701-eac84fc35558">
* Example of Anomaly fields in suppression combobox
<img width="881" alt="Screenshot 2024-06-06 at 5 14 17 PM"
src="9aa6ed99-1e02-44a0-ad1b-785136510d68">
* Suppression disabled due to no jobs running
<img width="668" alt="Screenshot 2024-06-17 at 11 23 39 PM"
src="a8636a52-31bd-4579-9bcd-d59d93c26984">
* Warning due to not all jobs running
<img width="776" alt="Screenshot 2024-06-17 at 11 26 16 PM"
src="f44c2400-570e-4fde-adce-e5841a2de08d">
## Steps to Review
1. Review the Test Plan for an overview of behavior
2. Review Integration tests for an overview of implementation and edge
cases
3. Review Cypress tests for an overview of UX changes
4. Testing on [Demo
Instance](https://rylnd-pr-181926-ml-rule-alert-suppression.kbndev.co/)
(elastic/changeme)
1. This instance has the relevant feature flag enabled, has some sample
auditbeat data, as well as the [anomalies archive
data](https://github.com/elastic/kibana/tree/main/x-pack/test/functional/es_archives/security_solution/anomalies)
for the purposes of exercising an ML rule against "real" anomalies
1. There are a few example rules in the default space:
1. A simple [query
rule](f6f5960d-7e4b-40c1-ae15-501112822130)
against auditbeat data
1. An [ML
rule](9122669e-b2e1-41ce-af25-eeae15aa9ece)
with per-execution suppression on both `by_field_name` and
`by_field_value` (which ends up not actually suppressing anything)
1. An [ML
rule](0aabc280-00bd-42d4-82e6-65997c751797)
with per-execution suppression on `by_field_name` (which suppresses all
anomalies into a single alert)
## Related Issues
- This feature was temporarily blocked by
https://github.com/elastic/kibana/issues/183100, but those changes are
now in this PR.
## Checklist
- [x] Functional changes are hidden behind a feature flag. If not
hidden, the PR explains why these changes are being implemented in a
long-living feature branch.
- [x] Functional changes are covered with a test plan and automated
tests.
* [Test Plan](https://github.com/elastic/security-team/pull/9279)
- [x] Stability of new and changed tests is verified using the [Flaky
Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner) in
both ESS and Serverless. By default, use 200 runs for ESS and 200 runs
for Serverless.
* [ESS - Cypress x
200](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6449)
* [Serverless - Cypress x
200](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6450)
* [ESS - API x
200](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6447)
* [Serverless - API x
200](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6448)
- [ ] Comprehensive manual testing is done by two engineers: the PR
author and one of the PR reviewers. Changes are tested in both ESS and
Serverless.
- [ ] Mapping changes are accompanied by a technical design document. It
can be a GitHub issue or an RFC explaining the changes. The design
document is shared with and approved by the appropriate teams and
individual stakeholders.
- [ ] (OPTIONAL) OpenAPI specs changes include detailed descriptions and
examples of usage and are ready to be released on
https://docs.elastic.co/api-reference. NOTE: This is optional because at
the moment we don't have yet any OpenAPI specs that would be fully
"documented" and "GA-ready" for publishing on
https://docs.elastic.co/api-reference.
- [ ] Functional changes are communicated to the Docs team. A ticket is
opened in https://github.com/elastic/security-docs using the [Internal
documentation request (Elastic
employees)](https://github.com/elastic/security-docs/issues/new?assignees=&labels=&projects=&template=docs-request-internal.yaml&title=%5BRequest%5D+)
template. The following information is included: feature flags used,
target ESS version, planned timing for ESS and Serverless releases.
---------
Co-authored-by: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes https://github.com/elastic/kibana/issues/181309
This PR
- allows users to create, edit or delete templates via cases > settings
page
- allows users to create case using templates
39226aa4-9d9a-41a8-a900-ca765ed98e1b
## Testing
1. Go to all solutions and create cases with all fields (including all
fields of all supported connectors) without using templates. Verify that
everything is working as expected.
2. Go to all solutions and create and edit templates with various
fields. Verify that everything is working as expected.
3. Go to all solutions, create different templates on each solution, and
verify that when creating a case you can use templates and everything is
working as expected.
4. Go to the alerts table of o11y and security and attach alerts to a
new case. Verify that in the flyout the templates are working as
expected.
5. Go to ML and try to attach an ML visualization to a new case. Verify
that the solution picker is working as expected and it resets the form
when changing solutions.
6. Create a template with custom fields. Delete one of the custom fields
from the settings page. Verify that it is also deleted from the
template.
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
**Flaky test runner**:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6425
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Release notes
Allow users to create case using templates.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>
Co-authored-by: adcoelho <antonio.coelho@elastic.co>
Implement the Event Based Telemetry for Datasets table (main
page) and the Dataset details (the flyout at the time of
writing).
The following EBT events are reported:
- `"Dataset Quality Navigated"`
- `"Dataset Quality Dataset Details Opened"`
- `"Dataset Quality Dataset Details Navigated"`
The above allow to track the following:
1. Used query, available and chosen filters for "Integrations",
"Namespaces" and "Qualities" when user clicks on the degraded documents
percentage link (with `_ignored` filter) from main table's column or
"Open" link from row actions.
2. Dataset health, percentage of degraded documents, duration to load
and breakdown field when user opens the flyout.
3. All included in 2 plus whether `_ignored` filter is present in
navigation, source and target of the navigation when user navigates away
from the flyout.
4. All events also track selected date range and user's privileges state
for the respective data stream.
### Main page - Datasets table
Event Name: `"Dataset Quality Navigated"`
This event is reported whenever a degraded percentage link or "Open"
link is navigated to on the main datasets table.
The following properties are tracked:
### <a name="dqn" >Properties</a>
| Property | Type | Schema Type | Description |
| --- | --- | --- | --- |
| `index_name` | string | keyword | The name of the index e.g.
`logs-apache.access-default` |
| `data_stream` | object | object | Object containing [ECS Data Stream
Fields](https://www.elastic.co/guide/en/ecs/current/ecs-data_stream.html)
i.e. `dataset`, `namespace`, and `type` |
| `data_stream_health` | object | keyword | Any of `"poor"`,
`"degraded"` and `"good"` representing the health/quality of data stream
|
| `data_stream_aggregatable` | boolean | boolean | A boolean indicating
whether the data stream is aggregatable for the `_ignored` field |
| `from` | string | date | ISO start date string selected on datepicker
|
| `to` | string | date | ISO end date string selected on datepicker |
| `degraded_percentage` | number | float | A number representing the
percentage of degraded documents in the data stream |
| `integration` | string (optional) | keyword | An optional string
representing the integration name associated with the dataset |
| `privileges` | object | object | An object representing the
privileges. It includes `can_monitor_data_stream`,
`can_view_integrations`, and an optional `can_view_dashboards`. All are
boolean. |
| `filters` | object | object | An object containing filter details. It
includes `is_degraded`, `query_length`, `integrations`, `namespaces`,
and `qualities`. See below for more details |
The `filters` property is an object with the following sub-properties:
| <div style="min-width:80px">Sub-Property</div> | <div
style="min-width:60px">Type</div> | <div style="min-width:60px">Schema
Type</div> | Description |
| --- | --- | --- | --- |
| `is_degraded` | boolean | boolean | A boolean indicating whether
navigation included `ignored` filter |
| `query_length` | number | short | The length of the query used |
| `integrations` | object | object | An object including `total`,
`included`, and `excluded` properties representing applied filters. |
| `namespaces` | object | object | An object including `total`,
`included`, and `excluded` properties representing applied filters |
| `qualities` | object | object | An object including `total`,
`included`, and `excluded` properties representing applied filters |
### Details page - Flyout
Event `"Dataset Quality Dataset Details Opened"` is reported when flyout
is opened whereas `"Dataset Quality Dataset Details Navigated"` is
reported when a link is clicked on the flyout which navigates the user
away from Dataset Quality page. Important properties are tracked which
help analyse the state user had before the navigation e.g. breakdown
field, selected date range and whether user clicked the degraded docs or
all docs link.
_Note that, the flyout is expected to be converted into a routed page,
hence "Dataset Details" is used for event names instead of the flyout._
#### Properties
`"Dataset Quality Dataset Details Opened"` only differs from [`"Dataset
Quality Navigated"`](#dqn) by the following properties:
| <div style="min-width:80px">Property</div> | <div
style="min-width:60px">Type</div> | <div style="min-width:60px">Schema
Type</div> | Description |
| --- | --- | --- | --- |
| `tracking_id` | string | keyword | Id to group flyout opening and
navigation for funnel analysis |
| `duration` | number | long | Time it took in milliseconds from opening
the flyout until the data stream details are available |
| `breakdown_field` | string (optional) | keyword | Fields used to break
the chart down by |
`"Dataset Quality Dataset Details Navigated"` only differs from
[`"Dataset Quality Navigated"`](#dqn) by the following properties:
| <div style="min-width:80px">Property</div> | <div
style="min-width:60px">Type</div> | <div style="min-width:60px">Schema
Type</div> | Description |
| --- | --- | --- | --- |
| `tracking_id` | string | keyword | Id to group flyout opening and
navigation for funnel analysis |
| `filters` | object | object | `{ "is_degraded": <boolean> }` which
represent whether the user is navigating with `_ignored` filter applied|
| `breakdown_field` | string (optional) | keyword | Fields used to break
the chart down by |
| `target` | enum value | keyword | Action that user took to navigate
away from the dataset details page. Possible values are `Exit`,
`LogsExplorer`, `Discover`, `Lens`, `Integration`, `IndexTemplate`,
`Dashboard`, `Hosts` and `Services` |
| `source` | enum value | keyword | Section of dataset details page the
action is originated from. Possible values are `"Header"`, `"Footer"`,
`"Summary"`, `"Chart"`, `"Table"` and `"ActionMenu"` |
Add links to Dataset Quality in the following places:
1. "Data sets" link on Logs Explorer nav header (on both Serverless and
Stateful)
2. "Data Set Quality" side nav menu item under Stack Management -> Data
(Stateful)
3. "Data Set Quality" card under Management -> Data (Serverless)
On Logs Explorer - Stateful
On Logs Explorer - Serverless
Stack Management - Stateful
Stack Management - Serverless
---------
Co-authored-by: Yngrid Coello <yngrid.coello@elastic.co>
