Closes https://github.com/elastic/logs-dev/issues/183,
https://github.com/elastic/logs-dev/issues/184 and
https://github.com/elastic/logs-dev/issues/185.
## Summary
This PR aims to support failure store in dataset quality page. The
following acceptance criteria items were resolved
### Dataset quality page
- [x] A column for Failed docs is included in the table
- [x] A tooltip is placed in the title of the column
- [x] A % of documents inside Failure store is calculated for every
dataStream
- [x] If % is lesser than 0.0001 but greater than 0 we should show ⚠
symbol next to the ~0 value (as we do with degraded docs)
- [x] Failed docs percentages greater than 0 should link to discover
🎥 Demo
https://github.com/user-attachments/assets/6d9e3f4c-02d9-43ab-88cb-ae70716b05d9
### Dataset details page
- [x] A metric, Failed docs, is included in the Overview panel under
Data set quality. This metric includes the number of documents inside
the failure store for the specific dataStream.
- [x] A tooltip is placed in the title of the Failed docs metric with
message: `The percentage of docs sent to failure store due to an issue
during ingestion.`
- [x] Degraded docs graph section is transformed to Document trends
allowing the users to switch between Degraded docs and Failed docs
trends over time.
- [x] A new chart for failed documents is created with links to
discover/Logs explorer using the right dataView
🎥 Demo
https://github.com/user-attachments/assets/6a3a1f09-2668-4e83-938e-ecdda798c199
### Failed docs ingestion issue flyout
- [x] Whenever documents are found in failure store we should list
Document indexing failed in Quality issues table
- [x] User should be able to expand Document indexing failed and see
more information in the flyout
- [x] The flyout will show Docs count, an aggregation of the number of
documents inside failure store for the selected timeframe
- [x] The flyout will show Last ocurrence, the datetime registered for
the most recent document in the failure store.
- [x] The flyout will contain a section called Error messages where a
list of unique error messages should be shown, exposing Content (error
message) and Type (Error Type).
- [x] Type should contain a tooltip where message (`Error message
category`) explain users how we are categorising the errors.
- [x] Other issues inside Quality issues table will be appended by field
ignored and the field will be shown in bold.
https://github.com/user-attachments/assets/94dc81f0-9720-4596-b256-c9d289cefd94
Note: This PR was reconstructed from
https://github.com/elastic/kibana/pull/199806 which it supersedes.
## How to test
1. Execute `failed_logs` synthtrace scenario
2. Open dataset quality page
## Follow ups
- Enable in serverless
- Deployment agnostic tests cannot be added until we enable this in
serverless
- FTR tests will be added as part of
https://github.com/elastic/logs-dev/issues/182
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Part of: https://github.com/elastic/kibana/issues/201813
- [x] Memory Usage. Check ML entities are filtered according to the
project type.
- [x] Notifications page. Check ML entities are filtered according to
the project type.
Since the timerange in Discover of the main request is stable we don't need to trigger a main fetch for all data when the histogram/chart is being hidden/displayed, unless it's necessary to get the data (e.g. when the histogram/chart was hiden when a discover session was being loaded)
Part of https://github.com/elastic/kibana/issues/203716
## Summary
This PR introduces a new test case for LogsDB in both Stateful and
Serverless:
> Verify that users can override LogsDB index settings including:
ignore_above, ignore_malformed, ignore_dynamic_beyond_limit, subobjects
and timestamp format.
For modify `subobjects` and `timestamp` format it must be done from the
mappings tab. For `ignore_above`, `ignore_malformed`,
`ignore_dynamic_beyond_limit` the configuration is done in the Settings
tab.
It also introduces a test case only for Stateful
(enableMappingsSourceFieldSection [is
disabled](9c6de6aabc/config/serverless.yml (L112))
for serverless)
> Verify that users cannot disable synthetic source for a LogsDB index.
## Summary
Recently enabled ES flag in QA (see
https://github.com/elastic/serverless-gitops/pull/6185), caused Kibana
to begin rendering predefined roles on the roles management page. This
caused QA test failures. Skipping these tests will eliminate the noise
in the QA test alert channel while we resolve the issue.
Solution will be to make all predefined roles reserved so we can ensure
they are not rendered in Kibana.
cc @wayneseymour
Resolves https://github.com/elastic/kibana/issues/182229
## Release Note
Remove Logs Explorer
## Summary
- Removes the Discover/Logs Explorer tabs in the top nav
- Changes all links to Logs Explorer in other applications to Discover
- Removes all APIs exposed by Logs Explorer
Note: This does not remove Logs Explorer from the codebase. There will
be a follow up for that work.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Part of https://github.com/elastic/kibana/issues/203716
## Summary
* Functional test for verifying that users can create indices using the
“logsdb” index mode via the Kibana interface (Stateful and serverless).
* Functional test for verifying that users can upgrade a logs data
stream that is using a “standard” index mode to a data stream using
“logsdb” index mode. (Stateful and serverless).
* Functional test for verifying that users can downgrade a data stream
that is using a “logsdb” index mode to a data stream using “standard”
index mode. (Stateful and serverless).
* Functional test for verifying that users can upgrade a data stream
that is using a “time series” index mode to a data stream using “logsdb”
index mode. (Stateful and serverless).
* Functional test for verifying that users can downgrade a data stream
that is using a “logsdb” index mode to a data stream using “time series”
index mode. (Stateful and serverless).
The rollover operation has not been tested since is not expected to
happen automatically. See [this
comment](https://github.com/elastic/kibana/issues/203716#issuecomment-2554491054).
Newer builds of ES ingest `event.ingested` and `tags` fields into APM
indices making the Dataset Quality tests fail which rely on ingested
field counts against set number of field limits. The PR addresses this
by explicitly ingesting the fields beforehand.
After the merge to main, the PR will be backported to `8.x` to unblock
es-promotion.
Adds a `model_id` property to the model config and renames the service
to `elasticsearch`, as `elser` has been deprecated.
This should avoid triggering API deprecation warning.
- Closes https://github.com/elastic/kibana/issues/200257
## Summary
This PR adds `discover_profile_resolved` EBT events when a new
contextual profile is resolved at any level (root, data source, or
document level).
Events will be triggered from Discover pages and dashboards.
Please note that duplicate events will not be sent in the current
implementation. Only when profile ID is changing.
<img width="1710" alt="Screenshot 2024-12-06 at 17 39 23"
src="https://github.com/user-attachments/assets/54eef630-8bcc-4185-917f-d251c003df65">
<img width="1699" alt="Screenshot 2024-12-06 at 17 38 36"
src="https://github.com/user-attachments/assets/541522eb-9cc5-47b6-a51a-81186232b7e5">
## Testing
Add logging to `trackContextualProfileResolvedEvent` or do the
following:
Enable "Usage collection" global setting.
Navigate to Discover and observe `kibana-browser` requests in Network
tab.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Davis McPhee <davismcphee@hotmail.com>
## Summary
Fixing flaky test when choosing a connector with the new EuiComboBox
component.
https://github.com/elastic/kibana/issues/203462
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## 📓 Summary
Closes#202258
This change updates the colors scale for Discover's `log.level`
indicators to differentiate errors from other levels better.
**N.B. As this relies on some hard-coded values defined
[here](https://github.com/elastic/kibana/issues/186273#issuecomment-2505817075),
it is not a definitive version, but a middle step to enhance the scale
in v8.x versions.**
With the introduction of the Borealis theme in v9, a new scale
token-based will replace this.
<img width="934" alt="Screenshot 2024-12-04 at 17 40 32"
src="https://github.com/user-attachments/assets/b3da1300-b39a-4ad0-92c9-fde5dabe91ec">
---------
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
## Summary
Functional tests for data usage UI.
- `data_streams` route is intercepted, due to filtering out zero size
data streams which will happen because metering api needs time to
aggregate data
- `autoops_api` is using the mock server as there will be no data for it
to return
- tests will only run in local serverless and not MKI due to using the
autoops mock server that won't return data for created data streams
- adds `interceptRequest` functionality to FTR `browser` service
## Tests
- data stream filter dropdown renders with created data streams of
`data_streams` response and are checked
- data stream filter dropdown renders badge with correct number of
selected data streams
- charts render from `data_streams` route response
- chart legends render with correct items
- popover renders for legend items
- links in popovers correctly navigate and update navigation between
different data stream items
The geoip database is not available in CI, so a `tags` field is added to
all ingested APM app log documents. This covers up the difference in
field counts between serverless CI and MKI by always having a `tags`
field in the documents. It also unskips the test that failed in MKI
because of this.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Currently, the FTR tests are written to expect only two preconfigured
endpoints. However, there might be more incoming, and this PR
generalizes these tests so they do not depend on the number of
preconfigured endpoints in the future.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Closes https://github.com/elastic/kibana/issues/190795
Adds the logic to display a message to the user in case there is already
an existing Firehose data in their cluster and to show the identified
AWS services in the "Visualize Data" step right away without waiting for
the window to loose focus first.
## How to test
1. Open the Firehose flow
2. Make sure there is no callout and the third step is not active
3. Go to Kibana dev console and ingest some dummy data (see examples
bellow)
4. Refresh the page with the Firehose flow
5. make sure there is a callout and the third steps shows the identified
AWS service
```
POST logs-aws.apigateway_logs-default/_doc
{
"@timestamp": "2024-11-25T13:32:01.000Z",
"some": 111,
"aws.kinesis.name": "Elastic-CloudwatchLogs"
}
POST metrics-aws.apigateway_metrics-default/_doc
{
"@timestamp": "2024-11-25T13:31:01.000Z",
"agent": {
"type": "firehose"
},
"aws": {
"cloudwatch": {
"namespace": "AWS/ApiGateway"
},
"exporter": {
"arn": "arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J"
}
},
"cloud": {
"account": {
"id": "975050175126"
},
"provider": "aws",
"region": "us-west-2"
}
}
```
## Summary
This PR swaps the `EuiSelectable` by the `EuiComboBox` component as we
have in ESS.
### Context:
This PR belongs to this initiative
https://github.com/elastic/search-team/issues/8000 where we agreed on
bring low hanging fruit artefacts from ESS to ES3 before being replace
completly with the full new experience which will be later. Therefor we
are not investing effort on make the code scalable and reusable at this
point.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Navarone Feekery <13634519+navarone-feekery@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Rodney Norris <rodney@tattdcodemonkey.com>
## Summary
Closes https://github.com/elastic/kibana/issues/183245Closes#197630
[Observability AI Assistant] Partially migrate to inference client
replacing `inferenceClient.chatComplete` to
`observabilityAIAssistantClient.chat` -
`observabilityAIAssistantClient.complete` does a bunch of stuff on top
of `chat`. keepping `observabilityAIAssistantClient.chat` as a wrapper
for now because it also adds instrumentation and logging.
- handling of 2 error cases to error handler
- `security_exception` due to lack of privileges. Metering api will
respond when one of the following isn't available as a user privilege
`monitor,view_index_metadata,manage,all`.
- `index_not_found_exception`. Metering api will respond with this when
no indices exist for the privileges it has access to or when no indices
are found.
- api integration tests for data_streams route for the following cases
- returns no data streams when there are none it has access to and
responds with appropriate message
- returns no data streams without necessary privileges and responds with
appropriate message
- returns data streams when user only has access to a subset of indices
with necessary privileges
- functional tests for same as above. these remain skipped due to not
being able to create data streams picked up by metering api since we
implemented filtering out zero storage size data streams, but useful for
local testing with some code changes.
### `security_exception` view
<img width="1555" alt="Screenshot 2024-12-04 at 1 14 10 PM"
src="https://github.com/user-attachments/assets/241a2eb8-1c77-4592-ba18-b971512e712e">
### `index_not_found_exception` view
<img width="1589" alt="Screenshot 2024-12-04 at 1 13 13 PM"
src="https://github.com/user-attachments/assets/12b68d66-9178-4957-b014-5765be348694">
---------
Co-authored-by: Ashokaditya <ashokaditya@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
