Commit graph

259 commits

Author SHA1 Message Date
Matthew Kime
2403bc8c88
[data views] Stored fields are always requested, we don't need logic around it (#171815)
## Summary

Stored field handling is basically a dead code path. `['*']` is always
sent so lets push this closer to the query.

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-11-30 14:29:12 -06:00
Mykola Harmash
ad2ca2443c
[ObsUX] Add Top Functions to Infra Profilling tab (#171974)
Closes https://github.com/elastic/kibana/issues/171962

## Summary

This integrated Profiling Top Functions embeddable into the Infra's
Profiling tab in asset details.

![CleanShot 2023-11-28 at 14 20
38@2x](408ca866-1bc9-4b66-9ba1-d090cce0f7da)

## How to Test

* Connect local kibana to oblt cluster that has Profiling configured
(e.g. edge)
* Add this to your dev `kibana.yml`
```
xpack.profiling.enabled: true
xpack.infra.profilingEnabled: true

# Direct ES URL on the oblt cluster that you're using, in case of edge it's https://edge-oblt.es.us-west2.gcp.elastic-cloud.com:443
xpack.profiling.elasticsearch.hosts: REMOTE_CLUSTER_ES_URL

# If needed create a new user on the remote oblt cluster
xpack.profiling.elasticsearch.username: REMOTE_CLUSTER_USER
xpack.profiling.elasticsearch.password: REMOTE_CLUSTER_PASWORD
```

* Open kibana, go to Hosts
* Open a flyout for one of the hosts and make sure you see the Profiling
tab with both Flamegraph and Top Functions
* Open Host details as a full page and also make sure you see the same
* Make sure Profiling data updates when you change dates in the date
picker
2023-11-29 16:59:37 +01:00
Mykola Harmash
3f9b2ce448
[ObsUX] Add Profiling tab to Asset Details (#171764)
## Summary

This adds a new "Universal Profiling" tab to asset details with a
flamegrapth for a selected host. The tab is behind a feature flag and is
disabled by default. It will be enabled by default for clound/onprem
once we implement Profiling empty state, serverless is tbd.

* Added two new endpoints for fetching profiling status (not used by the
FE yet) and flamegraph data.
* Added a `profilingEnabled` feature flag
* Added a new tab in the UI and integrated the Flamegraph embeddable

![CleanShot 2023-11-22 at 16 38
00@2x](acc789df-00c0-45af-878a-39fa5e2064cc)


## How to test

* Connect local kibana to oblt cluster that has Profiling configured
(e.g. edge)
* Add this to your dev `kibana.yml`
```
xpack.profiling.enabled: true
xpack.infra.profilingEnabled: true

# Direct ES URL on the oblt cluster that you're using, in case of edge it's https://edge-oblt.es.us-west2.gcp.elastic-cloud.com:443
xpack.profiling.elasticsearch.hosts: REMOTE_CLUSTER_ES_URL

# If needed create a new user on the remote oblt cluster
xpack.profiling.elasticsearch.username: REMOTE_CLUSTER_USER
xpack.profiling.elasticsearch.password: REMOTE_CLUSTER_PASWORD
```

* Open kibana, go to Hosts
* Open a flyout for one of the hosts and make sure you see the Profiling
tab with a flamegraph
* Open Host details as a full page and also make sure you see the new
tab

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-11-27 10:28:32 +01:00
Milton Hultgren
4963e6b11d
[infra] Shorten IDs for ML jobs (#168234)
Closes #47477

### Summary

ML job IDs have a limit of 64 characters. For the log ML jobs we add the
string `kibana-logs-ui` plus the space and log view IDs as a prefix to
the job names (`log-entry-rate` and `log-entry-categories-count`) which
can quickly eat up the 64 character limit (even our own Stack Monitoring
log view hits the limit). This prevents users from being able to create
ML jobs and it's hard to rename a space or log view, and the limit is
not hinted at during space creation (because they are unrelated in some
sense).

In order to achieve a more stable length to the ID, this PR introduces a
new format for the prefix which creates a UUID v5 which uses the space
and log view ID as seed information (it then removes the dashes to still
be within the size limit for the categorization job).

Since there is no technical difference between the new and old format,
this PR makes an effort to continue to support the old format and allow
migration of old jobs as needed. The old jobs work and may contain
important data so the user should not feel forced to migrate.

The main addition is a new small API that checks if any ML jobs are
available and which format they use for the ID so that the app can
request data accordingly and the APIs have been modified to take the ID
format into account (except during creation which should always use the
new format).

The solution applied is not ideal. It simply passes the ID format along
with the space and log view ID to each point where the ID is re-created
(which is multiple). The ideal solution would be to store the job data
in the store and pass that around instead but that seemed like a
considerably larger effort. This PR does introduce some functional tests
around the ML job creation process, so such a future refactor should be
a bit safer than previously.

### How to test

* Start from `main`
* Start Elasticsearch
* Start Kibana
* Load the Sample web logs (Kibana home -> Try sample data -> Other
sample data sets)
* Visit the Anomalies page in the Logs UI
* Set up any of the two ML jobs or both, wait for some results to show
up
* Checkout the PR branch
* Visit the anomalies page and verify that it still works (requests go
to resolve the ID format, should return 'legacy' which should then load
the data for the legacy job)
* Recreate the ML job and verify that the new job works and results
still show up (new requests should go out with the new format being
used, which may be a mixed mode if you have two jobs and only migrate
one of them)

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-11-19 15:00:11 +01:00
mohamedhamed-ahmed
ddc07c53a5
[Log Stream] Support date nanos (#170308)
closes #88290 

## 📝  Summary

As described in #88290 we need to add `date_nanos` support to the Stream
UI page. In this PR the necessary changes have been made all over the
Stream UI and the usages of it.

##   Testing

⚠️ Testing the Stream UI with old timestamp indices is important to make
sure that the behavior is still as before and not affected at all. This
can be done by running local env from the PR and simulating all
interactions on edge-lit cluster for example, to make sure that the
behavior is not changed.

For testing the new changes with support of `date_nano`:

1. You can use [the steps
here](https://github.com/elastic/kibana/issues/88290#issuecomment-1713400873)
to create and ingest documents with nano precision.
2. Navigate to the stream UI and the documents should be displayed
properly.
3. Sync with the URL state should be changed from timestamp to ISO
string date.
4. Changing time ranges should behave as before, as well as Text
Highlights.
5. Open the logs flyout and you should see the timestamp in nano
seconds.
6. Play around with the minimap, it should behave exactly as before.

### Stream UI:

<img width="2556" alt="Screenshot 2023-11-02 at 14 15 49"
src="596966cd-0ee0-44ee-ba15-f387f3725f66">

- The stream UI has been affected in many areas:
- The logPosition key in the URL should now be in ISO string, but still
backward compatible incase the user has bookmarks with timestamps.
- The minimap should still behave as before in terms of navigation
onClick and also highlighting displayed areas

### Stream UI Flyout:

<img width="2556" alt="Screenshot 2023-11-02 at 14 15 23"
src="6081533c-3bed-43e1-872d-c83fe78ab436">

- The logs flyout should now display the date in nanos format if the
document is ingested using a nano format.

### Anomalies:

<img width="1717" alt="Screenshot 2023-11-01 at 10 37 22"
src="b6170d76-40a4-44db-85de-d8ae852bc17e">

-Anomalies tab under logs as a navigation to stream UI which should
still behave as before passing the filtration and time.

### Categories:

<img width="1705" alt="Screenshot 2023-11-01 at 10 38 19"
src="c4c19202-d27f-410f-b94d-80507542c775">

-Categories tab under logs as a navigation to stream UI which should
still behave as before passing the filtration and time.

### External Links To Stream:
- All links to the Stream UI should still work as before:
   - APM Links for traces, containers, hosts
   - Infra links in Inventory and Hosts views

## 🎥 Demo


9a39bc5a-ba37-49e0-b7f2-e73260fb01f0
2023-11-13 14:51:33 +01:00
Sandra G
ea7ae45028
[Infra UI] Removed unused saved object attribute and unused source configuration code (#169430)
Resolves https://github.com/elastic/kibana/issues/168240

### Changes

- Removes `fields.message` from the `infrastructure-ui-source` saved
object who's value was being populated by
`xpack.infra.sources.default.fields.message` from config. See
https://www.elastic.co/guide/en/kibana/master/logs-ui-settings-kb.html
- Removes `getInternalSourceConfiguration` and
`defineInternalSourceConfiguration` functions introduced in
https://github.com/elastic/kibana/pull/36066 as I cannot see them being
used anywhere. Stops exposing `defineInternalSourceConfiguration` as
part of server plugin interface.
- Removes `getStaticDefaultSourceConfiguration` from InfraSources class
as we aren't using `sources` from kibana config in source configuration
anymore.
- Removes deprecations warning of removal in 8.0 for other fields
belonging to config xpack.infra.sources.* introduced in
https://github.com/elastic/kibana/pull/115103
- Removes `getAllSourceConfigurations` used only in removed deprecations
file
f427278322 (diff-081721894fc437938eb652beae0a0640ddeee532ec5e48af1f3093c8074f1eecL195)
- Removed `getAllSavedSourceConfigurations` only used in
`getAllSourceConfigurations`

### How to test 
- `infrastructure-ui-source` saved object no longer has `fields`
attributes
- in kibana.yml set `xpack.infra.sources.default.fields.message:
['testmessage', '@testmessage']`
  - go to Infra -> Settings
  - Change the name and save the form
- view the `infrastructure-ui-source` saved object and no `fields`
attribute should exist (unless it previously existed in an already
existing `infrastructure-ui-source` saved object
- changes do not affect`fields.message` is used in logs_shared plugin
- i'm not sure how to easily test this and relying on someone with more
Logs knowledge to help out, but from what I can see logs accesses the
config directly (`xpack.infra.sources.default.fields.message`) and never
uses the saved object field being removed.
2023-11-07 11:54:32 +00:00
Kevin Lacabane
f1fa4b0b98
[apm][infra] remove infra dependency from apm (#167531)
## Summary

Closes https://github.com/elastic/kibana/issues/166834

Moves the following components out of infra plugin so we can remove the
apm->infra dependency
- `/infra/metrics_explorer` route
- (Host|Pod|Container)MetricsTable components
- InfraAppId
- InfraLocators

### Testing
We should focus the testing on the metrics_explorer route which is used
by 1. Tables in the Infrastructure section of apm and 2. Infrastructure
> Metrics explorer app. No functionality was added so these apps should
have the same existing behavior. Easiest way to get these views loaded
is by connecting kibana to an edge-oblt cluster, alternatively load
service and metrics with a data loader.

### Follow up
- (Host|Pod|Container)MetricsTable components are part of the
metricsDataAccess plugin contract but should be moved to stateless
package

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Jason Rhodes <jason.rhodes@elastic.co>
2023-11-02 16:28:32 +02:00
Mykola Harmash
78d2eda0ff
[ObsUX] Hide Alerts and rules dropdown behind feature flag in serverless (#169611)
Closes https://github.com/elastic/kibana/issues/169339

## Summary

* Hides the "Alerts and rule" dropdown in the Infra header behind a
feature flag in serverless
* Moves serverless infra tests to the `index.feature_flags.ts` to
prevent failures on MKI

![CleanShot 2023-10-24 at 09 38
40@2x](f9d47dcb-4b6c-4a39-a8cd-7fcd1a69ba80)


## How to test

* Run in serverless
* Make sure the Alerts and rules dropdown is not there anymore
* Run in stateful
* Make sure dropdown works as before
2023-10-26 09:39:15 +02:00
jennypavlova
fbc1a49869
[infra UI] Fix: Processes tab is showing error when cpu is null (#169272)
Closes #168196 

## Summary

This PR handles the case when `process_list` API returns `null` value
for `cpu`.

## Investigation
I checked when that happened and my first assumption was that the API
would return null in case the process data is not fully available for
all processes (in case of restarting metricbeat for example) but I saw
some hosts also on `edge lite` that have null values (only for some
processes - not all of them). I tried the same query on the edge lite
and then I saw documents with the cpu value `null` and some processes
that have the cpu value available for the same timeframe:

![image](175e2b98-c95c-46f4-a619-a4ef680de58a)
I don't think it's related to certain processes or indices as it is
reproducible using different ones. This could be considered an edge case
but as mentioned in [the
comment](https://github.com/elastic/kibana/issues/168196#issuecomment-1752967009)
having the null values is likely to happen when using the auto-refresh
option.

## The fix
To fix the issue we agreed on handling the case when `null` cpu value is
present by showing a `N/A` with an explanation and a CPU chart
placeholder with the same explanation in the process list table instead
of throwing an error

| Before | After |
| ------ | ------ |
|
![image](fb70a69b-62fe-466a-93ae-3313c3b7ba5b)
|
![image](4fb6cf39-8840-4a78-886b-9293aba7b521)
![image](6a804fe8-a564-48e8-a00b-60d83457600d)
|

## Testing


- Go to infra -> Hosts -> open the hosts flyout and select the processes
tab:
- It can be tricky to reproduce the issue locally some options are:
- Remote cluster: In case egde **lite** cluster is used the host who I
find to have this issue is
[gke-edge-lite-oblt-edge-lite-oblt-poo-c1d12345-sbnt](https://edge-lite-oblt.kb.us-west2.gcp.elastic-cloud.com/app/metrics/hosts?waffleTime=(currentTime:1697469375748,isAutoReloading:!f)&_a=(dateRange:(from:now-2m,to:now-1m),filters:!(),limit:50,panelFilters:!(),query:(language:kuery,query:%27%27))&controlPanels=(cloud.provider:(explicitInput:(fieldName:cloud.provider,id:cloud.provider,title:%27Cloud%20Provider%27),grow:!f,order:1,type:optionsListControl,width:medium),host.os.name:(explicitInput:(fieldName:host.os.name,id:host.os.name,title:%27Operating%20System%27),grow:!f,order:0,type:optionsListControl,width:medium))&tableProperties=(detailsItemId:gke-edge-lite-oblt-edge-lite-oblt-poo-c1d12345-sbnt-Ubuntu,pagination:(pageIndex:0,pageSize:20),sorting:(direction:asc,field:name))&assetDetails=(dateRange:(from:%272023-10-17T09:18:11.097Z%27,to:%272023-10-17T09:19:11.097Z%27),name:gke-edge-lite-oblt-edge-lite-oblt-poo-c1d12345-sbnt,tabId:processes)&waffleFilter=(expression:%27%27,kind:kuery)&waffleOptions=(accountId:%27%27,autoBounds:!t,boundsOverride:(max:1,min:0),customMetrics:!(),customOptions:!(),groupBy:!(),legend:(palette:cool,reverseColors:!f,steps:10),metric:(type:cpu),nodeType:host,region:%27%27,sort:(by:name,direction:desc),source:url,timelineOpen:!f,view:map))
- Metricbeat: It can be reproduced by stoping metricbeat for around a
minute then stating it again and refreshing the interval to `now` until
data with `null` CPU values are displayed.
- Look at the processes and find a process with `N/A` value in the CPU
column (A tooltip should appear after clicking on the question mark icon
next to `N/A`) and extend the process:

![image](4fb6cf39-8840-4a78-886b-9293aba7b521)
- The same tooltip should appear after clicking on the question mark
icon next to the "No results found" placeholder:

![image](6a804fe8-a564-48e8-a00b-60d83457600d)

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-10-25 10:28:11 -07:00
Mykola Harmash
3e8058bdf2
[Infra UI] Disable Infrastructure and Metrics alerts in Serverless (#167978)
Closes https://github.com/elastic/kibana/issues/164683

## Summary

This PR disables the infrastructure, metrics and logs alerts rule in
Serverless:
- Deletes the code responsible for the "Metric Anomaly" rule as it was
[previously disabled](https://github.com/elastic/kibana/pull/93813) with
plans to re-enable it as the previous PR describes but that never
happened.
- Adds feature flags for all three types of alert rules
- Prevents rules registration in serverless based on the feature flags
- Adds logic for showing/hiding items in the "Alerts and rules" dropdown
- Disables custom threshold rule in the Infra UI by default in
serverless as the rule needs to first be enabled by default by
@elastic/actionable-observability team
([context](https://elastic.slack.com/archives/C023GDA0WMP/p1696853751040269))

**Dropdown**
![CleanShot 2023-10-05 at 15 22
48@2x](fb7344c6-b5ee-4020-bd69-473dcd6be446)

**Host details**
![CleanShot 2023-10-05 at 15 23
02@2x](8164f82b-323c-4a2a-8cdc-c65a6c0f0c63)

### How to test

- Checkout locally Run in Serveless mode
- Enable, Infra plugin, custom threshold in Infra, and custom threshold
rule in general:
```
xpack.infra.enabled: true
xpack.infra.featureFlags.customThresholdAlertsEnabled: true
xpack.observability.unsafe.thresholdRule.enabled: true
```
- Go to `/app/metrics/hosts` and make sure there are no "Infrastructure"
and "Metrics" items in the "Alerts and rules" dropdown
- Click on "Manage rules" in the "Alerts and rules" dropdown, then
"Create rule" to open the rule selection flyout
- Make sure there are no rules for "Inventory", "Metrics" or "Logs"
threshold
- Run Kibana in traditional mode
- Make sure the "Alerts and rules" dropdown looks as usual and you can
create "Infrastructure" and "Metrics" alerts

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-10-12 15:24:30 +02:00
Carlos Crespo
2f8ce938be
[Infra UI] Disable infra logs UI in serverless (#168011)
closes https://github.com/elastic/kibana/issues/166977
## Summary

This PR creates a feature flag to disable logs UI features in
serverless.

_Serverless_

**Frontend routes**

<img width="1466" alt="image"
src="a856055b-f78b-4d47-b27e-678e19bdfae4">


`/logs/stream`
<img width="1714" alt="image"
src="83d50be2-ed3a-4441-b31c-1afe6f8fd0b9">

`/logs/log-categories`
<img width="1714" alt="image"
src="85cd369d-55b2-46a2-a61a-135c5f18e299">

`/logs/anomalies`
<img width="1714" alt="image"
src="d68e6f91-fe29-483f-931d-d05f4303effa">

**Server routes**

These will return 404

```
POST kbn://api/infra/log_analysis/results/log_entry_anomalies_datasets

POST kbn://api/infra/log_analysis/results/log_entry_anomalies

POST kbn://api/infra/log_analysis/results/log_entry_categories

POST kbn://api/infra/log_analysis/results/latest_log_entry_category_datasets_stats

POST kbn://api/infra/log_analysis/results/log_entry_category_datasets

POST kbn://api/infra/log_analysis/results/log_entry_category_examples

POST kbn://api/infra/log_analysis/results/log_entry_examples

POST kbn://api/infra/log_analysis/results/log_entry_datasets

POST kbn://api/infra/log_analysis/results/log_entry_rate_indices

POST kbn://api/infra/log_alerts/chart_preview_data                            
```


_Stateful_

<img width="1716" alt="image"
src="4406434f-78e4-437f-b3b5-1b03dc23f3c0">

**Server routes**

These will return 400 - currently there is no way to pass `version` via
dev tools

```
POST kbn://api/infra/log_analysis/results/log_entry_anomalies_datasets

POST kbn://api/infra/log_analysis/results/log_entry_anomalies

POST kbn://api/infra/log_analysis/results/log_entry_categories

POST kbn://api/infra/log_analysis/results/latest_log_entry_category_datasets_stats

POST kbn://api/infra/log_analysis/results/log_entry_category_datasets

POST kbn://api/infra/log_analysis/results/log_entry_category_examples

POST kbn://api/infra/log_analysis/results/log_entry_examples

POST kbn://api/infra/log_analysis/results/log_entry_datasets

POST kbn://api/infra/log_analysis/results/log_entry_rate_indices

POST kbn://api/infra/log_alerts/chart_preview_data                            
```

### How to test

### How to test

- Start a local es instance: `yarn es serverless --kill --clean
--license trial --ssl`
- Enable `infra` in the `serverless.oblt.dev.yml` file:
  - `xpack.infra.enabled: true`
- Start a local kibana instance: `yarn serverless-oblt --ssl` 
- Verify the items listed above both on serverless and stateful
2023-10-06 10:15:43 +02:00
Mykola Harmash
48be97bea2
[Infra UI] Add custom threshold alerts for serverless (#167639)
Closes https://github.com/elastic/kibana/issues/166612

## Summary

- Adds a `customThresholdAlertsEnabled` feature flag
- Based on the feature flag shows an additional menu item in the alerts
dropdown
- Adds flyout component for rendering the custom alerts flyout

**Serverless**
<video
src="4fc79427-1e0c-4692-8525-b3b35e8a04e9"
controls="controls"></video>

**Traditional**

![Traditional](b2c16478-a40d-4312-a41e-5e34c98daa47)

## How to test

- Checkout locally and run in serverless mode (infra plugin on)
- Make sure you see the "Create custom threshold alert" menu item in the
"Alerts and rules" dropdown
- Click on the item and make sure you can create an alert
- Run Kibana in traditional mode and make sure the dropdown item is not
there
2023-10-04 15:06:55 +02:00
Carlos Crespo
ede81cfdb0
[Infra UI] Disable Osquery in serverless (#167233)
closes https://github.com/elastic/kibana/issues/163280

## Summary

Disables OSquery in serverless. 


<img width="1670" alt="image"
src="0ece0503-a5f3-46a1-a99b-29285088526c">

<img width="1670" alt="image"
src="5d20aaa4-7bdc-47e5-b9c2-469ecf62ab1f">


<img width="1670" alt="image"
src="97874aee-be98-4f0a-984a-562abbd4187d">

Osquery is now only used in the Asset Details flyout/full-page view. The
header component in the Asset Details now checks the feature flag object
to show/hide the OSquery tab.

### How to test

- Start a local es instance: `yarn es serverless --kill --clean
--license trial --ssl`
- Enable `infra` in the `serverless.oblt.dev.yml` file:
  - `xpack.infra.enabled: true`
- Start a local kibana instance: `yarn serverless-oblt --ssl` and see if
the side nav contains the Infrastructure item
- Navigate to `https://0.0.0.0:5601/ftw/app/metrics` and open the Asset
Details flyout from Inventory and Hosts View. Navigate to the details
view and check that Osquery tab isn't present.

- Confirm if OSquery tab is present in stateful.

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-10-04 12:48:30 +02:00
Mykola Harmash
d0a0a1f9e6
[Infra IU] Disable Metrics Explorer for serverless (#167022)
Closes #163282 

## Summary

This PR:
* Adds a `featureFlags.metricsExplorerEnabled` property to the Infra
plugin config to enable and disable Metrics Explorer depending on the
offering type
* Prevents `MetricsExplorerViewsService` initialization for serveless
based on the feature flag
* Prevents creating Metrics Explorer frontend routes when in serverless
* Prevents registration of the MetricsExplorerViews saved object when in
serverless
* Prevents initialization of the `metrics_explorer_views` API routes
when in serverless

**Trying to access Metrics Explorer in serverless**
<img width="1829" alt="CleanShot 2023-09-22 at 12 59 35@2x"
src="2b039925-0f0b-4c07-be29-bbe910de7a34">

**Trying to access views API**
<img width="1829" alt="CleanShot 2023-09-22 at 13 00 00@2x"
src="15269ec2-becd-4ee3-9b5e-d916df28a7b8">

**`infra/metrics_explorer` API still works as per ticket requirements**
<img width="1829" alt="CleanShot 2023-09-22 at 13 00 06@2x"
src="fb23f912-c6fd-46c8-9084-c17c51e5b064">


## How to test
* Checkout locally
* Enable Infra in `serverless.oblt.yml`: `xpack.infra.enabled: true`
* Run Kibana in serverless mode
* Try accessing `/app/metrics/explorer` route and make sure it's not
available
* Make sure other Infra routes (`/app/metrics/inventory` and
`/app/metrics/hosts`) still load as expected
* In Kibana dev console make sure you get 404 for `GET
kbn:/api/infra/metrics_explorer_views`
* Also check that you don't see `metrics-explorer-view` saved object in
the response for `GET
kbn:/api/kibana/management/saved_objects/_allowed_types`
* Run Kibana in non-serverless mode and make sure Metrics Explorer is
accessible and works as usual

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-09-29 11:27:19 +02:00
Carlos Crespo
d79a38363a
[Infra UI] Cleanup host-specific Node Detail Page code (#166828)
closes: https://github.com/elastic/kibana/issues/166428

## Summary

This PR removes code that is no longer needed after replacing the Node
Details View for Host with the Asset Details.

### TSVB

The TSVB files were apparently only used to display charts in the Node
Details view. Due to the Asset Details using Lens to power the charts,
corresponding `host` TSVB formulas and configs are no longer needed.

Therefore, `host*`, `hostK8s*`, and `hostDocker*` (the latter appears to
have never been used) have been removed. Additionally, `aws*` from
`required_metrics` was also removed because it was host-specific.

### FE Components

The main change is in the `useMetadata` hook. I have changed the hook
signature, making `requiredMetrics` optional. This parameter is used to
process additional filtering and is only used in asset types that the
old Node Details page supports. Not passing it is not expected to break
other places that depend on this hook.

### Server

Removing TSVB files has a direct impact on the
`api/metrics/node_details` endpoint. This endpoint is only used to
provide data to the Node Details page. It returns a 400 error if an
invalid metric is passed - which will be the case for hosts

**Example Request:**
```json
POST kbn:api/metrics/node_details
{
  "metrics": [
    "hostK8sCpuCap",
    "hostSystemOverview"
  ],
  "nodeId": "gke-release-oblt-release-oblt-pool-c4163099-bcaj",
  "nodeType": "host",
  "timerange": {
    "from": 1695210522045,
    "to": 1695214122045,
    "interval": ">=1m"
  },
  "cloudId": "6106013995483209805",
  "sourceId": "default"
}
```

**Response:**
```json
{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "Failed to validate: \n  in metrics/0: \"hostK8sCpuCap\" does not match expected type \"podOverview\" | \"podCpuUsage\" | \"podMemoryUsage\" | \"podLogUsage\" | \"podNetworkTraffic\" | \"containerOverview\" | \"containerCpuKernel\" | \"containerCpuUsage\" | \"containerDiskIOOps\" | \"containerDiskIOBytes\" | \"containerMemory\" | \"containerNetworkTraffic\" | \"containerK8sOverview\" | \"containerK8sCpuUsage\" | \"containerK8sMemoryUsage\" | \"nginxHits\" | \"nginxRequestRate\" | \"nginxActiveConnections\" | \"nginxRequestsPerConnection\" | \"awsEC2CpuUtilization\" | \"awsEC2NetworkTraffic\" | \"awsEC2DiskIOBytes\" | \"awsS3TotalRequests\" | \"awsS3NumberOfObjects\" | \"awsS3BucketSize\" | \"awsS3DownloadBytes\" | \"awsS3UploadBytes\" | \"awsRDSCpuTotal\" | \"awsRDSConnections\" | \"awsRDSQueriesExecuted\" | \"awsRDSActiveTransactions\" | \"awsRDSLatency\" | \"awsSQSMessagesVisible\" | \"awsSQSMessagesDelayed\" | \"awsSQSMessagesSent\" | \"awsSQSMessagesEmpty\" | \"awsSQSOldestMessage\" | \"custom\"\n  in metrics/1: \"hostSystemOverview\" does not match expected type \"podOverview\" | \"podCpuUsage\" | \"podMemoryUsage\" | \"podLogUsage\" | \"podNetworkTraffic\" | \"containerOverview\" | \"containerCpuKernel\" | \"containerCpuUsage\" | \"containerDiskIOOps\" | \"containerDiskIOBytes\" | \"containerMemory\" | \"containerNetworkTraffic\" | \"containerK8sOverview\" | \"containerK8sCpuUsage\" | \"containerK8sMemoryUsage\" | \"nginxHits\" | \"nginxRequestRate\" | \"nginxActiveConnections\" | \"nginxRequestsPerConnection\" | \"awsEC2CpuUtilization\" | \"awsEC2NetworkTraffic\" | \"awsEC2DiskIOBytes\" | \"awsS3TotalRequests\" | \"awsS3NumberOfObjects\" | \"awsS3BucketSize\" | \"awsS3DownloadBytes\" | \"awsS3UploadBytes\" | \"awsRDSCpuTotal\" | \"awsRDSConnections\" | \"awsRDSQueriesExecuted\" | \"awsRDSActiveTransactions\" | \"awsRDSLatency\" | \"awsSQSMessagesVisible\" | \"awsSQSMessagesDelayed\" | \"awsSQSMessagesSent\" | \"awsSQSMessagesEmpty\" | \"awsSQSOldestMessage\" | \"custom\""
}
```

### How to Test
- Start a local Kibana instance pointing to an oblt cluster.
- Navigate to `Infrastructure`.
- Try different asset types and navigate to the Node Details view.

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-09-28 12:39:04 +02:00
Kerry Gallagher
b2271a9fd7
[Logs+] Remove configurable redirects to Discover (#167151)
## Summary

Closes https://github.com/elastic/kibana/issues/165227.

Removes configurable app targets. Only the standard logs app is
registered (Note: [Infra is disabled in
serverless](https://github.com/elastic/kibana/pull/165289) observability
/ security projects currently).

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-09-27 18:02:37 +01:00
jennypavlova
a511426e83
[Infra UI] Add tooltips to Asset Details (#164858)
Closes https://github.com/elastic/kibana/issues/164594
## Summary

This PR adds tooltips to explain the time range used to collect process
data and metadata. The tooltips will be shown inside the overview
(metadata summary section) tab, metadata tab, and processes tab (both
flyout and full page views).

## Fixes

I saw that the icons showing different explanation/ docs links inside
the overveiew tab are not consistent ( sometimes we have
`questionInCircle` and sometimes `iInCircles`) - the designs are using
`iInCircles` so I changed it everywhere:
<img width="671" alt="icons"
src="edda271b-5030-4d83-9722-448fbae8cf8b">


## Testing
1. Go to host view and open the flyout for any host. The new explanation
and tooltips should be shown in:
   - Overview tab
   
<img width="938" alt="overview"
src="ac4ae369-d825-4fba-8865-c9410de29c28">

   - Metadata tab
   
<img width="945" alt="metadata"
src="4d174bf3-3411-40a5-a208-eb5df2266c61">

   - Processed tab
   
<img width="937" alt="processes"
src="11d32c66-4a25-4fce-a95e-42698f2e1682">
2. Click Open as page and check the same on the asset details page 


<img width="1653" alt="Screenshot 2023-08-28 at 11 28 47"
src="342d1565-bb51-4961-b8ac-8b8270c851ff">
<img width="1637" alt="Screenshot 2023-08-28 at 11 28 32"
src="63b66a12-d634-4ecc-83de-ad1e1b79334c">
<img width="1649" alt="Screenshot 2023-08-28 at 11 28 16"
src="59720bf3-88ad-44b1-8584-769b38d956ed">

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-08-30 11:26:42 -07:00
Maryam Saeidi
9584a3ae0d
[AO] Use data view timestamp in the new threshold rule (#162542)
Closes #159777

## 📝 Summary 

In this PR, I changed the timestamp used in rule execution and the
preview chart of the new threshold rule.

I created a separate ticket
(https://github.com/elastic/kibana/issues/162560) to implement
`infra/metrics_explorer` in the observability plugin, and in the
meantime, I adjusted the API also to accept timeFieldName.

Also, I have a separate ticket for improving data view validation
(https://github.com/elastic/kibana/issues/162554)

## 🧪 How to test
### New threshold rule
- Create a data view of a timestamp other than `@timestamp`
- Create a rule with this data view and make sure the preview and
generated alerts are as expected
- Create a rule with grouping and filtering and check the preview and
generated alerts

### Metric threshold rule
- Create a metric threshold rule to make sure preview and rule execution
works as before
2023-07-28 16:43:00 +02:00
Sander Philipse
0a6b5e92b8
[Serverless Search] Remove unwanted alterting rule and connector types (#162105)
## Summary

This moves alerting in Serverless Search to the Stack Management
alerting page, and removes

To remove these rule types, I had to disable the observability,
securitySolution, uptime, monitoring and infra plugins and make sure
that their server plugins (not just the frontend plugins) respected the
`enabled: false` flag.
2023-07-19 16:21:02 +08:00
Carlos Crespo
a0a83c1d3c
[Infrastructure UI] Add strict payload validation to metrics_explorer_views endpoint (#160982)
closes [#157520](https://github.com/elastic/kibana/issues/157520)
## Summary

This PR adds strict payload validation to `metrics_explorer_views`
endpoint. This PR depends on this to be merged
https://github.com/elastic/kibana/pull/160852


### How to test

- Call the endpoint below trying to use invalid values. see
[here](https://github.com/elastic/kibana/pull/160982/files#diff-4573683b3b62cdf5f6426ec345b7ad6c7d6e6328237b213ca7519f686d8fa951R125-R131).

```bash
[POST|PUT] kbn:/api/infra/metrics_explorer_views
{
  "attributes": {
    "name": "Ad-hoc",
    "options": {
      "aggregation": "avg",
      "metrics": [
        {
          "aggregation": "avg",
          "field": "system.cpu.total.norm.pct",
          "color": "color0"
        },
      ],
      "source": "default",
      "groupBy": [
        "host.name"
      ]
    },
    "chartOptions": {
      "type": "line",
      "yAxisMode": "fromZero",
      "stack": false
    },
    "currentTimerange": {
      "from": "now-1h",
      "to": "now",
      "interval": ">=10s"
    }
  }
}
```

- Set up a local Kibana instance
- Navigate to `Infrastructure > Metrics Explorer`
- In the UI, use the Saved View feature and try different field
combinations

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-07-14 14:20:03 +02:00
Marco Antonio Ghiani
abe58cb011
[Logs Shared] Move LogStream and LogView into new shared plugin (#161151)
## 📓 Summary

Closes #159128 

Due to a dependencies issue when disabling a plugin in serverless mode,
the LogStream feature and related logic were disabled for every
consumer.

We decided to split this shared component and endpoint into their own
plugin of shared logs utilities, reducing to the minimum the required
dependency that could disable the plugin.

What we moved can be summarized with:
- `infrastructure-monitoring-log-view` saved object definition and
registration
- LogViews server/client services (exposed with start contract) +
related endpoints
- LogEntries server service + related endpoints
- LogEntriesDomain logic (exposed with start contract)
- `<LogStream />` component
- `<ScrollableLogTextStreamView />` component and related logic
- LogView state machine
- Containers/Hooks to consume the moved APIs.
- Common types/utils definition, now exported and consumed as a
dependency from the `infra` plugin.

## 🤓 Review hints

Most of the changes are just renaming and moving stuff into the new
plugin, but for some operations was required to implement new logic,
which may deserve a more critical review:
- server/public `plugin.ts` files for the `infra` and `logs_shared`
plugins. The new plugin now registers the fallback actions to retrieve a
source configuration if there's no stored log view. It also set the
configuration for the message field and registers the log view saved
object.
- the `logEntriesDomain` has also been moved inside the new plugin, but
is also used by the logs-analysis endpoints, so it is exposed by the
logs_shared plugin and consumed by `infra`.

## 👣 Following steps

We currently are still using the `observability` plugin for consuming
the CoPilot feature on our LogsStream flyout.
The plugin dependency is marked as optional, so disabling the
`observability` plugin in a serverless environment won't disable also
the exposed features in this new plugin, but it'll affect only the
CoPilot feature, which won't be loaded.

In future, would be nice to extract the CoPilot feature into its own
package/plugin, so that also serverless projects can consume it without
depending on `observability.

---------

Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-07-05 10:30:28 +02:00
Carlos Crespo
d79e69a1ed
[Infrastructure UI] Add strict payload validation to inventory_views endpoint (#160852)
closes [#157505](https://github.com/elastic/kibana/issues/157505)
fixes [#157740](https://github.com/elastic/kibana/issues/157740)
## Summary

This PR adds strict payload validation to `inventory_views` endpoint. I
tried to make the types consistent between sever and frontend code and
because of that more changes had to be made. I also refactored the
`toolbar_control` component, decoupling them from Inventory View and
Metrics Explorer View types


### How to test

- Call the endpoint below trying to use invalid values. see
[here](https://github.com/elastic/kibana/pull/160852/files#diff-058b21e249ebbe2795d450d07025d8904a58cfb07a97979e85975f87e931ffceR133-R143).

```bash
[POST|PUT] kbn:/api/infra/inventory_views
{
  "attributes": {
    "metric": {
        "type": [TYPE]
    },
    "sort": {
        "by": "name"
        "direction": "desc"
    },
    "groupBy": [],
    "nodeType": "host",
    "view": "map",
    "customOptions": [],
    "customMetrics": [],
    "boundsOverride": {
        "max": 1,
        "min": 0
    },
    "autoBounds": true,
    "accountId": "",
    "region": "",
    "autoReload": false,
    "filterQuery": {
        "expression": "",
        "kind": "kuery"
    },
    "legend": {
        "palette": "cool",
        "steps": 10,
        "reverseColors": false
    },
    "timelineOpen": false,
    "name": "test-uptime"
  }
}
```

- Set up a local Kibana instance
- Navigate to `Infrastructure`
- In the UI, use the Saved View feature and try different field
combinations

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-07-05 08:52:24 +02:00
Bena Kansara
395ff8e7f4
Include group-by information in query for Logs spike analysis in Logs alert detail page (#160647)
Resolves https://github.com/elastic/kibana/issues/160257

The group by field/value are added to the filter of the query that is
passed to the Log Spikes analysis component.

Example of the query when group by on `host.name` in the below Log
threshold rule.

<img width="497" alt="Screenshot 2023-06-27 at 17 15 18"
src="8e2b6622-a6d3-4273-b312-4a17a8d72824">


### Before
```
{
    "bool": {
        "filter": [
            {
                "term": {
                    "log.level": {
                        "value": "info"
                    }
                }
            }
        ],
        "must_not": [
            {
                "match": {
                    "error.log.message": "test"
                }
            }
        ]
    }
}
```

### After
```
{
    "bool": {
        "filter": [
            {
                "term": {
                    "log.level": {
                        "value": "info"
                    }
                }
            },
            {
                "term": {
                    "host.name": {
                        "value": "gke-edge-oblt-pool-2-ce49b4ca-l0nr"
                    }
                }
            }
        ],
        "must_not": [
            {
                "match": {
                    "error.log.message": "test"
                }
            }
        ]
    }
}
```
2023-06-30 16:32:15 +01:00
Carlos Crespo
8aab6fa522
[Infrastructure UI] Reorganize metric charts in the hosts view (#159457)
## Summary

This PR is the first step towards making metrics across infra
consistent. For the Hosts view, these are the new metrics:

<img width="667" alt="image"
src="6bcbc9a2-b9d5-42fc-bee0-d74edffa9960">

_note: KPI's titles are now capitalized._


<img width="667" alt="image"
src="16c51b3f-ccfc-400f-ad86-cbca53dd2f6a">



### How to test
- Start a local Kibana instance
- Navigate to `Infastructure > Hosts`

### For reviewer

- Some translation keys had to be changed and others removed because the
metrics no longer exist or the copy has completely changed. No new
translations have been added.
- All tooltips are temporary, they will be addressed in another PR

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-06-15 14:33:31 +02:00
Bena Kansara
a2bd9a7897
Add log rate spike analysis to Logs alert details page (#158960)
Resolves https://github.com/elastic/actionable-observability/issues/8

Adds log rate spike analysis to Logs threshold alert details page for
the "Count" alerts.

The logs spike analysis is shown for the following configurations:
- Count alerts when condition is to check "more than" or "more than or
equals" threshold

The logs spike analysis is hidden for the following configurations:
- Ratio alerts
- If rule params contain group by
- If the condition is to check "less than" or "less than or equals"
threshold

### Note about Co-Pilot
The PR also integrates Co-Pilot prompt. The prompt message needs to be
updated with the correct field values from the analysis results, which
will be done in a separate PR.

### Manual testing
1. Create a Log threshold rule
2. Ensure "Count" is selected
3. Ensure "more than" or "more than or equals" is selected for the
threshold
4. Enter threshold
5. Save rule
6. Wait for alerts to be generated
7. Go to alert details page
8. Ensure "Explain Log Rate Spikes" is shown

[Guide to generate test
data](https://docs.google.com/document/d/1PfPU5mjFx_jWFUkgtaHVy7qPvgBC7g1l4LKe7Jn-qUc)

### Feature flag
The Logs threshold alert details page is behind a feature flag.

To test, add the following in the `kibana.dev.yml`:
 `xpack.observability.unsafe.alertDetails.logs.enabled: true`


<img width="1511" alt="Screenshot 2023-06-12 at 20 18 34"
src="dd9391ad-53cd-4484-a11c-090f6a9acdd1">

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-06-14 12:30:27 +02:00
Zacqary Adam Xeper
0152724d47
[RAM] Add AAD Fields API (#158516)
## Summary

Closes #158324

Adds API to retrieve the `fieldsForAAD` from a given rule type. This is
a list of fields used for Alerts As Data; at the moment we need these
fields to provide an accurate autocomplete list to the Conditional
Actions UI.

### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Patryk Kopyciński <contact@patrykkopycinski.com>
2023-06-13 08:31:33 -04:00
mohamedhamed-ahmed
3785c93c8f
[Logs+] Replace usages of link-to routes in infra server (#158994)
closes [#157985](https://github.com/elastic/kibana/issues/157985)

## 📝  Summary

This PR moves the previously implemented [infra
locators](https://github.com/elastic/kibana/pull/155156) to infra common
and replaces the last usage of `link-to` routes in the infra public with
the appropriate locator.

##   Testing

1. Navigate to Observability Alerts
2. Create a new Log Threshold Rule that should fire
3. Open the alert details flyout
4. Click on the `View in app` button
5. Should correctly navigate to Stream view or Discover in case of
serverless.
* All other links to Logs should still work properly as before as well 



9af85ea6-5fdd-4505-9707-6489b760f311
2023-06-08 15:45:20 +01:00
Marco Antonio Ghiani
e3fddf38f7
[Logs UI] Versioning for Logs related APIs (#158710)
## 📓  Summary

Closes #157324 
Closes #159275 
Closes #159303 

This work converts the APIs related to the Logs UI feature into
versioned APIs using the new [Kibana versioned
router](https://docs.elastic.dev/kibana-dev-docs/versioning-http-apis#4-adhere-to-the-http-versioning-specification).

The converted APIs are the following, where each endpoint now is set to
version `1`:
-
[log_views](https://github.com/elastic/kibana/tree/main/x-pack/plugins/infra/server/routes/log_views)
-
[log_entries](https://github.com/elastic/kibana/tree/main/x-pack/plugins/infra/server/routes/log_entries)
-
[log_analysis](https://github.com/elastic/kibana/tree/main/x-pack/plugins/infra/server/routes/log_analysis)
-
[log_alerts](https://github.com/elastic/kibana/tree/main/x-pack/plugins/infra/server/routes/log_alerts)

The PR also includes moving the interfaces and runtime types relatives
to each endpoint's group into the recommended practices for [versioning
interfaces](https://docs.elastic.dev/kibana-dev-docs/versioning-interfaces).

## 🧪 Testing

- Navigate to the Logs UI settings page and verify the log view is
correctly retrieved and can be successfully updated.
- Navigate to the Logs stream page and verify the stream entries are
retrieved and rendered.
- Navigate to the Anomalies and Categories pages page and verify the
anomalies entries are retrieved and rendered correctly.
- Create a Log threshold alert and verify the chart preview data are
correctly retrieved and shown.

---------

Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-06-08 15:43:46 +02:00
Sandra G
13806c8c8e
[Infrastructure UI] Decouple source config HTTP API schema from saved object schema (#158562)
Resolves https://github.com/elastic/kibana/issues/152279

Separates out the type for saved object `infrastructure-ui-source` so
the HTTP API no longer references the same type.

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-05-30 08:45:07 -04:00
Pierre Gayvallet
8453fe820a
Cleanup spread operators in reduce calls (#157471)
## Summary

The spread operator is costly and put pressure on GC. It should be
avoided when possible, especially in loops.

This PR adapts a lot of `reduce` calls in the codebase to remove the
usages of the diabolic spread operator, when possible.

Note: the PR is not fully exhaustive. I focused on the server-side, as
we're more directly impacted than on browser-side code regarding
performances.

## Removing `...` usages in `kittens.reduce()`

For `reduce` loops, the spread operator can usually easily be replaced:

#### - setting a value on the accum object and returning it

#### BAD
```ts
  return this.toArray().reduce(
      (acc, renderer) => ({
        ...acc,
        [renderer.name]: renderer,
      }),
      {} as Record<string, ExpressionRenderer>
    );
```

#### GOOD
```ts
  return this.toArray().reduce((acc, renderer) => {
      acc[renderer.name] = renderer;
      return acc;
    }, {} as Record<string, ExpressionRenderer>);
```


#### - assigning values to the accum object and returning it 

#### BAD
```ts
  const allAggs: Record<string, any> = fieldAggRequests.reduce(
      (aggs: Record<string, any>, fieldAggRequest: unknown | null) => {
        return fieldAggRequest ? { ...aggs, ...(fieldAggRequest as Record<string, any>) } : aggs;
      },
      {}
    );
```

#### GOOD
```ts
  const allAggs = fieldAggRequests.reduce<Record<string, any>>(
      (aggs: Record<string, any>, fieldAggRequest: unknown | null) => {
        if (fieldAggRequest) {
          Object.assign(aggs, fieldAggRequest);
        }
        return aggs;
      },
      {}
    );
```

#### - pushing items to the accum list and returning it 

#### BAD
```ts
  const charsFound = charToArray.reduce(
    (acc, char) => (value.includes(char) ? [...acc, char] : acc),
    [] as string[]
  );
```

#### GOOD
```ts
  const charsFound = charToArray.reduce((acc, char) => {
    if (value.includes(char)) {
      acc.push(char);
    }
    return acc;
  }, [] as string[]);
```

## Questions

#### Are you sure all the changes in this are strictly better for
runtime performances?

Yes, yes I am.

#### How much better?

Likely not much.

#### Are you planning on analyzing the perf gain?

Nope.

#### So why did you do it?

I got tired of seeing badly used spread operators in my team's owned
code, and I had some extra time during on-week, so I spent a few hours
adapting the usages in all our runtime/production codebase.

#### Was it fun?

Take your best guess.

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-05-22 04:50:24 -07:00
mohamedhamed-ahmed
5d96ef99d7
[Logs UI] Register Logs UI Locators (#155156)
## Summary

Closes https://github.com/elastic/kibana/issues/104855

This PR creates 2 registered locators:

1. Logs Locator
2. Node Logs Locator

With these 2 locators, we now have a typed navigation to the logs UI
which also redirects to Discover in serverless mode.

## Testing

### Normal behaviour

When Kibana is used as always then on any navigation to the logs UI, the
user will be redirected to the stream UI.
All links to `link-to` routes should still behave as before.
- Launch the Kibana dev environment with `yarn start`
- Navigate to Hosts UI
- Click the logs tab
- Add a filter text in the search bar
- Click on the Open in Logs link
- Verify that navigation to the Stream view and the state is maintained



https://user-images.githubusercontent.com/11225826/234514430-ddc1ffaa-0cb2-4f2a-84e9-6c6230937d9f.mov



### Serverless behaviour

When Kibana is used in serverless mode, we want to redirect any user
landing to Logs UI to the Discover page
- Launch the Kibana dev environment with `yarn serverless-oblt`
- Navigate to Hosts UI
- Click the logs tab
- Add a filter text in the search bar
- Click on the Open in Logs link
- Verify to be redirected to Discover and that the state is maintained



https://user-images.githubusercontent.com/11225826/234514454-dfb2774e-d6f1-4f4c-ba10-77815dc1ae9d.mov


### Next Steps
A separate PR will be created to fulfill the below AC

- All usages of link-to routes in other apps are replaced with usage of
the appropriate locator.
2023-05-17 13:19:26 +01:00
Marco Antonio Ghiani
26a559d3d1
[Infrastructure UI] Correct time generation Default views (#157739)
## 📓 Summary

This PR fixes generating a static `time` property for the default saved
view for inventory and metrics-explorer, switching to a new date
timestamp on each request as the previous behaviour.

## 🧪 Testing

Navigate to `Infrastructure -> Inventory`, remove all the URL query
params and check that the time selected is the current time for the
Default view.
The same behaviour is expected for the metrics explorer Default view.

---------

Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
2023-05-16 07:01:42 -07:00
Maryam Saeidi
fc9f19e3a2
[AO] Add metric threshold integration test (#157489)
## Summary

Closes #157189

This PR adds a metric threshold integration test. This is the first step
in adding more test coverage for observability rules.

**Steps during the test**
1. Generating fake host data by using a similar implementation as
https://github.com/elastic/high-cardinality-cluster
    - Data is generated for the last 15 mins
- Implementation was simplified only to cover fake hosts and was
converted to typescript
2. Creating an action using an index connector
3. Creating a metric threshold rule containing step number 2 action
4. Checking the status of the rule to be active
5. Checking the triggered action to have the correct parameters
6. Checking the generated alert to have the correct information
7. Clean up

**How to run locally**
- Run server
```
node scripts/functional_tests_server --config x-pack/test/api_integration/apis/metrics_ui/config.ts
```
- Then run the test
```
node scripts/functional_tests__runner --include-pack/test/api_integration/apis/metrics_ui/cometric_threshold_rule.ts --config x-pack/test/api_integration/apis/metrics_ui/config.ts
```

**Reference**
I created https://github.com/elastic/integrations/issues/6168 to find a
better way to generate data and make sure that data matches what
metricbeats generates

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-05-16 10:11:53 +02:00
jennypavlova
89f310c537
[Infrastructure UI] Processes tab: add N/A if the process summary is missing (#157374)
Closes [#153942](https://github.com/elastic/kibana/issues/153942)

## Summary

This PR shows 'N/A' when the values inside the process summary are not
available (and the actual value when available)


## Testing
- Open Hosts view and find a host ( where the process summary is not
available )
- Open the flyout and select the "Processes" tab
- The values inside the process summary should be 'N/A'
(Same should be shown in inventory)

<img width="2259" alt="Screenshot 2023-05-11 at 13 34 58"
src="7f8a6e3d-cd87-452a-9f73-13dbc567501d">


<img width="2258" alt="Screenshot 2023-05-11 at 13 36 05"
src="2708ef2f-b119-46a4-80e8-048e25a22c21">

- Open Hosts view and find a host ( where the process summary is
available )
- Open the flyout and select the "Processes" tab
- The values inside the process summary should be available (and total
processes > 0)
(Same should be shown in inventory)

<img width="2262" alt="Screenshot 2023-05-11 at 13 35 33"
src="89aeb53f-c7c2-483f-b21f-9a0b67ca22c4">
 
<img width="2265" alt="Screenshot 2023-05-11 at 13 36 29"
src="06b44138-21bb-4c47-a31b-b43749df58a0">
2023-05-11 09:50:26 -07:00
Marco Antonio Ghiani
56fa5b90a1
[Infrastructure UI] Clean dead code after saved objects migration (#156158)
## 📓 Summary

Closes https://github.com/elastic/kibana/issues/155117
Closes https://github.com/elastic/kibana/issues/152617

This PR removes all the legacy code unused for handling the saved views
client logic.

---------

Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: jennypavlova <jennypavlova94@gmail.com>
2023-05-04 05:01:36 -07:00
Walter Rafelsberger
385d0dc84a
[ML] Package @kbn/ml-anomaly-utils (#155697)
Creates Package `@kbn/ml-anomaly-utils`.

- This moves some of the utility functions, constants and types related
to Anomaly Detection we previously exposed from the `ml` plugin itself
to a package `@kbn/ml-anomaly-utils`, resulting in ~5.9KB reduction of
the plugin's page load bundle size.
- To reduce increases in async bundle size for consuming plugins the
utils have been refactored into individual files to allow deep imports
and get some optimization through that. Some previously duplicated code
in consuming plugins has been deleted and replaced with deep imports of
the corresponding original code in the package.
- Types have been prefixed with `Ml`.
- Constants have been prefixed with `ML_`.

Created package via `node scripts/generate package @kbn/ml-anomaly-utils
--web --dir ./x-pack/packages/ml/anomaly_utils`.
2023-05-03 08:46:27 +02:00
Marco Antonio Ghiani
5422d08239
[Infrastructure UI] Create MetricsExplorerViewsService and MetricsExplorerViewsClient (#155878)
## 📓 Summary

Depends on #154900
Closes #155112  

This PR implements the `InventoryViewsService` and
`InventoryViewsClient`, injecting an instance of the client in the
KibanaContextForPlugin and exposing so a set of utilities to
retrieve/update inventory views:
- `findMetricsExplorerViews`
- `getMetricsExplorerView`
- `createMetricsExplorerView`
- `updateMetricsExplorerView`
- `deleteMetricsExplorerView`

## 👣 Next steps
- Implement #154725 to consume the service

---------

Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: Carlos Crespo <crespocarlos@users.noreply.github.com>
2023-04-27 08:21:22 +02:00
Marco Antonio Ghiani
61bb52c65b
[Infrastructure UI] Implement Metrics explorer views CRUD endpoints (#155621)
## 📓  Summary

Part of #152617 
Closes #155111  

This PR implements the CRUD endpoints for the metrics explorer views.
Following the approach used for the InventoryView service, it exposes a
client that abstracts all the logic concerned to the
`metrics-explorer-view` saved objects.

It also follows the guideline provided for [Versioning
interfaces](https://docs.elastic.dev/kibana-dev-docs/versioning-interfaces)
and [Versioning HTTP
APIs](https://docs.elastic.dev/kibana-dev-docs/versioning-http-apis),
preparing for the serverless.

## 🤓 Tips for the reviewer
You can open the Kibana dev tools and play with the following snippet to
test the create APIs, or you can perform the same requests with your
preferred client:
```
// Get all
GET kbn:/api/infra/metrics_explorer_views

// Create one
POST kbn:/api/infra/metrics_explorer_views
{
  "attributes": {
    "name": "My view"
  }
}

// Get one
GET kbn:/api/infra/metrics_explorer_views/<switch-with-id>

// Update one
PUT kbn:/api/infra/metrics_explorer_views/<switch-with-id>
{
  "attributes": {
    "name": "My view 2"
  }
}

// Delete one
DELETE kbn:/api/infra/metrics_explorer_views/<switch-with-id>
```

---------

Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-04-26 14:23:48 +02:00
Marco Antonio Ghiani
13517601d5
[Infrastructure UI] Create InventoryViewsService and InventoryViewsClient (#155126)
## 📓 Summary

Depends on #154900
Closes #155110 

This PR implements the `InventoryViewsService` and
`InventoryViewsClient`, injecting an instance of the client in the
KibanaContextForPlugin and exposing so a set of utilities to
retrieve/update inventory views:
- `findInventoryViews`
- `getInventoryView`
- `createInventoryView`
- `updateInventoryView`
- `deleteInventoryView`

## 👣 Next steps
- Implement #154725 to consume the service

---------

Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-04-26 10:50:26 +02:00
Maryam Saeidi
dfea483ef2
[AO] Add threshold information to the metric threshold alert details page (#155493)
Closes #153740, closes #153833, closes #155593

This PR adds threshold information and rule name to the metric threshold
alert details page.


![image](https://user-images.githubusercontent.com/12370520/233968325-8a66166b-2534-4b9b-9054-9085270db5f6.png)


## 🧪 How to test
- Add xpack.observability.unsafe.alertDetails.metrics.enabled: true to
the Kibana config
- Create a metric threshold rule with multiple conditions that generates
an alert
- Go to the alert details page and check threshold information
- Click on the rule link; it should send you to the rule page
2023-04-25 02:14:47 -07:00
Carlos Crespo
06545277d7
[Infrastructure UI] Replace Snapshot API with InfraMetrics API in Hosts View (#155531)
closes [#154443](https://github.com/elastic/kibana/issues/154443)
## Summary

This PR replaces the usage of the Snapshot API in favor of the new
`metrics/infra` endpoint and also includes a new control in the Search
Bar to allow users to select how many hosts they want the API to return.


https://user-images.githubusercontent.com/2767137/233728658-bccc7258-6955-47fb-8f7b-85ef6ec5d0f9.mov

Because the KPIs now needs to show an "Average (of X hosts)", they will
only start fetching the data once the table has been loaded.

The hosts count KPI tile was not converted to Lens, because the page
needs to know the total number of hosts.

### Possible follow-up

Since now everything depends on the table to be loaded, I have been
experimenting with batched requests to the new API. The idea is to fetch
at least the host names as soon as possible.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-04-24 15:02:44 -03:00
Marco Antonio Ghiani
6ac4e1919c
[Infrastructure UI] Implement inventory views CRUD endpoints (#154900)
## 📓  Summary

Part of #152617 
Closes #155158 

This PR implements the CRUD endpoints for the inventory views.
Following the approach used for the LogViews service, it exposes a
client that abstracts all the logic concerned to the `inventory-view`
saved objects.

It also follows the guideline provided for [Versioning
interfaces](https://docs.elastic.dev/kibana-dev-docs/versioning-interfaces)
and [Versioning HTTP
APIs](https://docs.elastic.dev/kibana-dev-docs/versioning-http-apis),
preparing for the serverless.

## 🤓 Tips for the reviewer
You can open the Kibana dev tools and play with the following snippet to
test the create APIs, or you can perform the same requests with your
preferred client:
```
// Get all
GET kbn:/api/infra/inventory_views

// Create one
POST kbn:/api/infra/inventory_views
{
  "attributes": {
    "name": "My inventory view"
  }
}

// Get one
GET kbn:/api/infra/inventory_views/<switch-with-id>

// Update one
PUT kbn:/api/infra/inventory_views/<switch-with-id>
{
  "attributes": {
    "name": "My inventory view 2"
  }
}

// Delete one
DELETE kbn:/api/infra/inventory_views/<switch-with-id>
```

## 👣 Next steps
- Replicate the same logic for the metrics explorer saved object
- Create a client-side abstraction to consume the service
- Update the existing react custom hooks to consume the endpoint

---------

Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-04-24 11:54:43 +02:00
Carlos Crespo
ef218cd704
[Infrastructure UI] New Infra api (#155084)
closes: [#152103](https://github.com/elastic/kibana/issues/152103)
fixes: https://github.com/elastic/kibana/issues/151768

## Summary

This PR adds a new API to return host metrics for the Hosts View page. 

The difference between this API and Snapshot API is that this one runs a
terms aggregation and the API can return a range of 1 to 500 hosts,
preventing it from returning all data.

It uses Inventory Model aggregations, so the performance should be
similar to that of the Snapshot API. The `limit` parameter is what will
allow the client to try to get a better response time from the API.

#### Snapshot API

Returns all 500 hosts

15 minutes
<img width="317" alt="image"
src="https://user-images.githubusercontent.com/2767137/230073865-d0d98ce8-76ff-4295-be9b-94f5b342c9f7.png">

1 day
<img width="317" alt="image"
src="https://user-images.githubusercontent.com/2767137/230076350-affbad3d-a498-481e-bcba-bcec019ab739.png">

1 week
<img width="317" alt="image"
src="https://user-images.githubusercontent.com/2767137/230077049-320d90c8-4f3e-47ba-bd16-1c5878a83254.png">


#### Hosts API

100 hosts limit

15 minutes
<img width="317" alt="image"
src="https://user-images.githubusercontent.com/2767137/233349080-50acdf55-d269-49f7-be97-3fc259021020.png">

1 day
<img width="317" alt="image"
src="https://user-images.githubusercontent.com/2767137/233349614-826a791a-aef5-4e7a-b103-2900c4f64f49.png">

1 week
<img width="317" alt="image"
src="https://user-images.githubusercontent.com/2767137/233350318-388ae9b3-8b7b-4365-9f32-7dcbc2342480.png">

### How to test
```bash
curl --location -u elastic:changeme 'http://0.0.0.0:5601/ftw/api/metrics/infra' \
--header 'kbn-xsrf: xxxx' \
--header 'Content-Type: application/json' \
--data '{
    "type": "host",
    "limit": 100,
    "metrics": [
        {
            "type": "rx"
        },
        {
            "type": "tx"
        },
        {
            "type": "memory"
        },
        {
            "type": "cpu"
        },
        {
            "type": "diskLatency"
        },
        {
            "type": "memoryTotal"
        }
    ],
    "query": {
        "bool": {
            "must": [],
            "filter": [],
            "should": [],
            "must_not": []
        }
    },
    "range": {
        "from": "2023-04-18T11:15:31.407Z",
        "to":   "2023-04-18T11:30:31.407Z"
    },
    "sourceId": "default"
}'
```

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-04-20 16:31:27 -03:00
Marco Antonio Ghiani
b40b89e711
[Logs UI] Redirect Logs UI to Discover when in serverless mode (#154145)
## 📓 Summary

Closes #153890

The implementation creates a new LogsApp service where we should keep
any logic concerned with what `target_app` parameter is configured and
the actions related to a specific configuration. I thought it could be a
good approach to avoid drilling down the global config till we need it
and keep it cleaner by injecting only the service with predefined
actions.

In this first case, we create a redirect to discover using its locator,
and the exposed method can be used anywhere across the app for
triggering the redirect.

## 🧪 Testing

### Normal behaviour

When Kibana is used as always, we want to keep the current behaviour and
the user will stay on the Logs UI pages.
- Launch the Kibana dev environment with `yarn start`
- Navigate to Logs UI
- Verify the navigation works normally and that no redirect to Discover
occurs

### Serverless behaviour

When Kibana is used in serverless mode, we want to redirect any user
landing to Logs UI to the Discover page, configuring the same data view
or creating an ad-hoc one starting from the index pattern
- Launch the Kibana dev environment with `yarn serverless-oblt`
- Navigate to Logs UI
- Verify to be redirected to Discover and a temporary data view is
created from the current index pattern

---------

Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-04-19 08:30:45 -07:00
Faisal Kanout
01c0d0152c
[AO] - Add logs threshold ratio chart to the alert details page (#155024)
## Summary

It fixes #155022 by adding the new ratio chart

<img width="1197" alt="Screenshot 2023-04-17 at 16 17 45"
src="https://user-images.githubusercontent.com/6838659/232513836-3ac2e18b-a1d6-404b-aaa7-9670d3ea0fff.png">

### Checklist

Delete any items that are not applicable to this PR.

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)

---------
2023-04-17 09:21:39 -07:00
Faisal Kanout
e57c853329
[AO] - Add chart series filter capability for the Logs Alert Details page (#154335)
## Summary

It closes #154262 

### Before:

![](https://user-images.githubusercontent.com/6838659/229560256-e8965afd-fd45-4451-9b45-4496a477283b.png)


### After, we can filter the shown series on the chart alert-based


![](https://user-images.githubusercontent.com/6838659/229560443-279d26a7-a3dd-4ebb-8371-c6e35d50c555.png)
2023-04-04 16:16:30 +02:00
Kerry Gallagher
4c586a71d4
[Logs UI] Support inline Log Views in the UI (#152933)
## Summary

This closes #142840. It is the UI portion of support for inline Log
Views.

## Visible changes to the UI

### ML warning

![Screenshot 2023-03-10 at 14 55
34](https://user-images.githubusercontent.com/471693/224348959-8db70d91-dc8b-4f4e-926b-ec05e7481b78.png)

### Alert dropdown warning

![Screenshot 2023-03-10 at 14 55
43](https://user-images.githubusercontent.com/471693/224349026-cdf17767-225a-4ecd-af8a-b90e2a21816f.png)


### Settings page warning

![Screenshot 2023-03-10 at 14 56
02](https://user-images.githubusercontent.com/471693/224349066-bcb63ba8-fee8-4b7a-b41b-7d89e09f002a.png)


## Reviewer hints / notes

- The ACs on the issue are quite extensive and should provide a good
number of things to test.
  - Make use of the "playground" page (see below) to make this easier
 
- The `AlertDropdown` has been made lazy as the page load bundle
increased by 100kb otherwise.

- Our `link-to` functionality is scoped to persisted Log Views only at
the moment as historically they've only accepted a path segment, not
full query parameters. We can look to extend this in the future once we
have concrete linking needs.

## Questions

- I have allowed the Log View client to handle both the inline and
persisted Log Views. I wonder if the function names become confusing?
(e.g. are the RESTful prefixes misleading now?).

- The ML warning splash page links to settings to revert to a persisted
Log View. It could also be done in place on the page. I went back and
forth over whether we should keep the reverting in one place?


## Testing

There is now a "state machine playground" available at the following
URL: `/app/logs/state-machine-playground`, it is enabled in developer
mode only. It's not fancy or pretty it just serves to make testing
things easier. There are various buttons, e.g. `Switch to inline Log
View`, to facilitate easier testing whilst a Log View switcher is not in
the UI itself. You can utilise these buttons, and then head to other
pages to ensure things are working correctly, e.g. warning callouts and
disabled buttons etc. If you'd like to play with the options used, e.g.
for `update`, amend the code within `state_machine_playground.tsx`. It's
also useful just to sit on this page, try various things, and verify
what happens in the developer tools (does the context update correctly
etc).

## Known issues

- When saving on the settings page we actually revert to a "Loading data
sources" state. This is also the case on `main`. The reason for this is
the check within settings looks like so:

```ts
if ((isLoading || isUninitialized) && !resolvedLogView) {
    return <SourceLoadingPage />;
}
```

but the `resolvedLogView` state matching looks like so:

```ts
const resolvedLogView = useSelector(logViewStateService, (state) =>
    state.matches('checkingStatus') ||
    state.matches('resolvedPersistedLogView') ||
    state.matches('resolvedInlineLogView')
      ? state.context.resolvedLogView
      : undefined
  );
```

so even though we have resolved a Log View previously the state matching
overrides this. I'd prefer to follow this up in a separate issue as I'd
like to think through the ramifications a bit more. It's not a bug, but
it is jarring UX.

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-03-29 14:15:39 +01:00
Faisal Kanout
c1a8b90f51
[AO] - Add scaffolding and the main chart to the Logs threshold Alert Details page (#153081)
## Summary
### This is a kickoff PR; more PRs will follow up. 

It closes #152738
<img width="1191" alt="Screenshot 2023-03-23 at 13 10 09"
src="https://user-images.githubusercontent.com/6838659/227239913-4516cf44-92f6-454c-a7af-72dafb12cfb6.png">


### Checklist

Delete any items that are not applicable to this PR.

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
2023-03-28 18:15:47 +02:00
Marco Antonio Ghiani
b64dc95618
[Infrastructure UI] Kubernetes Pods correctly show Network chart (#153631)
## 📓 Summary

Closes #129239 

This issue was caused by a Validation error when creating the tsvb model
for the `podNetworkTraffic` metric series.
It was basically referencing an aggregation property that didn't exist,
caused by different property names matching.

## 🧪 Testing

- Navigate to Inventory
- Select `Show` -> `Kubernetes Pods`
- Select any available pod and click on Kubernetes pod metrics to open
the pod details view
- Verify the Network chart is available and matches the **Inbound (RX)**
and **Outbound (TX)** values.


https://user-images.githubusercontent.com/34506779/227875972-07aa35e1-eb0c-480a-8500-342de6ffad80.mov

---------

Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
2023-03-28 00:52:38 -07:00
Marco Antonio Ghiani
ca8848e00d
[Infrastructure UI] Improve error handling metric index (#152840)
## 📓 Summary

Closes #144882 

This PR updates the error handling in case the user configure a remote
index that was returning an error, giving more meaningful feedback to
the user.

It also includes a refactor for the `useSource` custom hook and removes
the usages for the legacy `useSourceViaHttp` custom hook.

## 🧪 Testing

- Navigate to the metrics setting page
- Set a wrongly formatted or non-existing remote cluster
- Verify to get a toast notification about the issue and to get notified
about the same issue when accessing inventory and hosts pages.

**Before**
<img width="1741" alt="before"
src="https://user-images.githubusercontent.com/34506779/223672635-abb3a7f3-e8a6-43b6-972c-813514d1b63a.png">

**After**


https://user-images.githubusercontent.com/34506779/223672676-017422fb-ccc7-456e-854c-ec79837da281.mp4

## 👣 Next step
As happens in the Data View stack management section, would be nice to
inform the user while he types the index pattern if no associated data
view matches the typed wildcard.
An additional value could be, in case the user wants to anyway set the
index pattern, to inform him with a callout message that it doesn't
match any data view, so that the issue cause is immediately clear when
no data are viewed on the metrics pages.

- [ ] https://github.com/elastic/kibana/issues/152885

---------

Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: florent-leborgne <florent.leborgne@elastic.co>
2023-03-23 18:34:09 +01:00