Integrate Asset Inventory with backend (#208417)

## Summary

Fetch and render backend data upon opening the Asset Inventory page.

### Depends on

- https://github.com/elastic/security-team/issues/11270
- https://github.com/elastic/kibana/issues/201709
- https://github.com/elastic/kibana/issues/201710
- https://github.com/elastic/security-team/issues/11687

### Screenshots

<details><summary>No applied filters</summary>
<img width="1452" alt="Screenshot 2025-02-18 at 08 40 51"
src="https://github.com/user-attachments/assets/e8970f92-701f-4bcf-9c43-8c1ce3155ba2"
/>
</details>

<details><summary>Filtering through search bar with KQL</summary>
<img width="1448" alt="Screenshot 2025-02-18 at 08 40 38"
src="https://github.com/user-attachments/assets/fdffe535-bb76-44da-be43-096e3007e680"
/>
</details>

<details><summary>Filtering through filter dropdowns</summary>
<img width="1451" alt="Screenshot 2025-02-18 at 08 41 03"
src="https://github.com/user-attachments/assets/ec68d9e8-5b4f-4c70-ba90-9fb7e4ddf18b"
/>
</details>

<details><summary>Filtering through both search bar and filter dropdowns
- no results found in this case</summary>
<img width="1447" alt="Screenshot 2025-02-18 at 08 40 28"
src="https://github.com/user-attachments/assets/2b2347e1-86fe-4d67-b859-0f84108c58bc"
/>
</details>

<details><summary>Default empty state (no rows fetched)</summary>
<img width="1452" alt="Screenshot 2025-02-18 at 09 39 49"
src="https://github.com/user-attachments/assets/79876021-c09b-42a0-a776-5e5fde688994"
/>
</details>

### Definition of done

- [x] Asset Inventory page fetches data prepared by the data-view that
comes pre-installed with the "Cloud Asset Inventory" integration
  - [x] Search bar
  - [x] Filters
  - [x] Data Grid
  - [x] Empty state when number of fetched rows is zero

### How to test

1. Prepare cloud user
- Go to [users
page](https://keep-long-live-env-ess.kb.us-west2.gcp.elastic-cloud.com/app/management/security/users)
on Elastic Cloud
    - Create a new user with a custom username and password
    - Copy the same roles from the user called `paulo_remote_dev`
2. Start local env running these commands
- Run ES with `node scripts/es snapshot --license trial -E
path.data=../default -E
reindex.remote.whitelist=cb8e85476870428d8c796950e38a2eda.us-west2.gcp.elastic-cloud.com:443
-E xpack.security.authc.api_key.enabled=true`
    - Run Kibana with `yarn start --no-base-path`
3. Go to Integrations page, switch on the "*Display beta integrations*"
control, then add the **Cloud Asset Inventory** integration on your
local environment. Postpone Elastic Agent addition.
4. Go to Dev Tools page, click on the "config" tab and add the following
environment variables:
Use the dev tools config tab to save your  as follows:
- `${ES_REMOTE_HOST}`:
[https://cb8e85476870428d8c796950e38a2eda.us-west2.gcp.elastic-cloud.com:443](https://cb8e85476870428d8c796950e38a2eda.us-west2.gcp.elastic-cloud.com/)
- `${ES_REMOTE_USER}`: (the username you set for your user on step 0)
    - `${ES_REMOTE_PASS}`: (the pass you set for your user on step 0)
5. Run the following script:

<details><summary>Script</summary>

```
POST _reindex?wait_for_completion=false
{
  "conflicts": "proceed", 
  "source": {
    "remote": {
      "host": "${ES_REMOTE_HOST}",
      "username": "${ES_REMOTE_USER}",
      "password": "${ES_REMOTE_PASS}"
    },
    "index": "logs-cloud_asset_inventory*",
    "query": {
      "bool": {
        "must": [
          {
            "range": {
              "@timestamp": {
                "gte": "now-1d"
              }
            }
          }
        ]
      }
    }
  },
  "dest": {
    "op_type": "create",
    "index": "logs-cloud_asset_inventory.asset_inventory-default"
  },
  "script": {
    "source": """
      ctx._source['entity.category'] = ctx._source.asset.category;
      ctx._source['entity.name'] = ctx._source.asset.name;
      ctx._source['entity.type'] = ctx._source.asset.type;
      ctx._source['entity.sub_type'] = ctx._source.asset.sub_type;
      ctx._source['entity.sub_category'] = ctx._source.asset.sub_category;
    """
  }
}
```

</details> 

Finally, open Discover page and set the DataView filter on the top-right
corner to `logs-cloud_asset_inventory.asset_inventory-*`, as in the
screenshot below. If the grid is populated, you've got data and the
whole setup worked!

<details><summary>Discover page</summary>

![discover-page](https://github.com/user-attachments/assets/5e719e64-0b99-4f0a-9687-1821d70fb84e)
</details> 

### Checklist

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

### Identify risks

No risks at all.
This commit is contained in:
Alberto Blázquez 2025-02-20 15:55:07 +01:00 committed by GitHub
parent 14605f462a
commit 596ced5b7e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
7 changed files with 206 additions and 93 deletions

View file

@ -4,27 +4,20 @@
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import React, { useMemo, useCallback } from 'react';
import React, { useMemo } from 'react';
import { EuiSpacer, EuiFlexItem } from '@elastic/eui';
import { i18n } from '@kbn/i18n';
import { FilterGroup } from '@kbn/alerts-ui-shared/src/alert_filter_controls/filter_group';
import type { FilterControlConfig } from '@kbn/alerts-ui-shared';
import type { Filter } from '@kbn/es-query';
import { createKbnUrlStateStorage, Storage } from '@kbn/kibana-utils-plugin/public';
import { Storage } from '@kbn/kibana-utils-plugin/public';
import { ControlGroupRenderer } from '@kbn/controls-plugin/public';
import { useHistory } from 'react-router-dom';
import { useDataViewContext } from '../../hooks/data_view_context';
import { useSpaceId } from '../../../common/hooks/use_space_id';
import { URL_PARAM_KEY } from '../../../common/hooks/use_url_state';
import { useGlobalTime } from '../../../common/containers/use_global_time';
import { ASSET_INVENTORY_INDEX_PATTERN } from '../../constants';
import { FilterGroupLoading } from './filters_loading';
import { ASSET_INVENTORY_RULE_TYPE_IDS } from './rule_type_ids';
const SECURITY_ASSET_INVENTORY_DATA_VIEW = {
id: 'asset-inventory-logs-default',
name: 'asset-inventory-logs',
};
const DEFAULT_ASSET_INVENTORY_FILTERS: FilterControlConfig[] = [
{
title: i18n.translate('xpack.securitySolution.assetInventory.filters.type', {
@ -57,64 +50,22 @@ export interface FiltersProps {
}
export const Filters = ({ onFiltersChange }: FiltersProps) => {
const { dataView: indexPattern, dataViewIsLoading, dataViewIsRefetching } = useDataViewContext();
const { from, to } = useGlobalTime();
const { dataView, dataViewIsLoading, dataViewIsRefetching } = useDataViewContext();
const spaceId = useSpaceId();
const history = useHistory();
const urlStorage = useMemo(
() =>
createKbnUrlStateStorage({
history,
useHash: false,
useHashQuery: false,
}),
[history]
);
const filterControlsUrlState = useMemo(
() =>
urlStorage.get<FilterControlConfig[] | undefined>(URL_PARAM_KEY.assetInventory) ?? undefined,
[urlStorage]
);
const setFilterControlsUrlState = useCallback(
(newFilterControls: FilterControlConfig[]) => {
urlStorage.set(URL_PARAM_KEY.assetInventory, newFilterControls);
},
[urlStorage]
);
const dataViewSpec = useMemo(
() =>
indexPattern
dataView
? {
id: SECURITY_ASSET_INVENTORY_DATA_VIEW.id,
name: SECURITY_ASSET_INVENTORY_DATA_VIEW.name,
// TODO We need this hard-coded id because `ASSET_INVENTORY_INDEX_PATTERN` does not populate the filter dropdowns
id: 'cloud_asset_inventory-2773feaf-50bb-43f8-9fa9-8f9a5f85e566',
name: ASSET_INVENTORY_INDEX_PATTERN,
allowNoIndex: true,
title: indexPattern.title,
title: dataView.getIndexPattern(),
timeFieldName: '@timestamp',
}
: null,
[indexPattern]
);
const handleFilterChanges = useCallback(
(newFilters: Filter[]) => {
if (!onFiltersChange) {
return;
}
const updatedFilters = newFilters.map((filter) => {
return {
...filter,
meta: {
...filter.meta,
disabled: false,
},
};
});
onFiltersChange(updatedFilters);
},
[onFiltersChange]
[dataView]
);
if (!spaceId || !dataViewSpec) {
@ -134,21 +85,14 @@ export const Filters = ({ onFiltersChange }: FiltersProps) => {
<EuiSpacer size="l" />
<FilterGroup
dataViewId={dataViewSpec?.id || null}
onFiltersChange={handleFilterChanges}
onFiltersChange={onFiltersChange}
ruleTypeIds={ASSET_INVENTORY_RULE_TYPE_IDS}
Storage={Storage}
defaultControls={DEFAULT_ASSET_INVENTORY_FILTERS}
chainingSystem="HIERARCHICAL"
spaceId={spaceId}
controlsUrlState={filterControlsUrlState}
setControlsUrlState={setFilterControlsUrlState}
ControlGroupRenderer={ControlGroupRenderer}
maxControls={4}
timeRange={{
from,
to,
mode: 'absolute',
}}
/>
<EuiSpacer size="l" />
</>

View file

@ -48,7 +48,7 @@ export const AssetInventorySearchBar = ({
<div css={getContainerStyle(euiTheme)}>
<SearchBar
appName="Asset Inventory"
showFilterBar={true}
showFilterBar={false}
showQueryInput={true}
showDatePicker={false}
isLoading={loading}

View file

@ -0,0 +1,10 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export const MAX_ASSETS_TO_LOAD = 500; // equivalent to MAX_FINDINGS_TO_LOAD in @kbn/cloud-security-posture-common
export const DEFAULT_VISIBLE_ROWS_PER_PAGE = 25;
export const ASSET_INVENTORY_INDEX_PATTERN = 'logs-cloud_asset_inventory.asset_inventory-*';

View file

@ -0,0 +1,141 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { useInfiniteQuery } from '@tanstack/react-query';
import { lastValueFrom } from 'rxjs';
import { number } from 'io-ts';
import type * as estypes from '@elastic/elasticsearch/lib/api/types';
import { buildDataTableRecord } from '@kbn/discover-utils';
import type { EsHitRecord } from '@kbn/discover-utils/types';
import type { RuntimePrimitiveTypes } from '@kbn/data-views-plugin/common';
import { showErrorToast } from '@kbn/cloud-security-posture';
import type { IKibanaSearchResponse, IKibanaSearchRequest } from '@kbn/search-types';
import type { FindingsBaseEsQuery } from '@kbn/cloud-security-posture';
import { useKibana } from '../../common/lib/kibana';
import { MAX_ASSETS_TO_LOAD, ASSET_INVENTORY_INDEX_PATTERN } from '../constants';
interface UseAssetsOptions extends FindingsBaseEsQuery {
sort: string[][];
enabled: boolean;
pageSize: number;
}
const ASSET_INVENTORY_TABLE_RUNTIME_MAPPING_FIELDS: string[] = ['entity.id', 'entity.name'];
const getRuntimeMappingsFromSort = (sort: string[][]) => {
return sort
.filter(([field]) => ASSET_INVENTORY_TABLE_RUNTIME_MAPPING_FIELDS.includes(field))
.reduce((acc, [field]) => {
const type: RuntimePrimitiveTypes = 'keyword';
return {
...acc,
[field]: {
type,
},
};
}, {});
};
const getMultiFieldsSort = (sort: string[][]) => {
return sort.map(([id, direction]) => {
return {
...getSortField({ field: id, direction }),
};
});
};
/**
* By default, ES will sort keyword fields in case-sensitive format, the
* following fields are required to have a case-insensitive sorting.
*/
const fieldsRequiredSortingByPainlessScript = ['entity.name']; // TODO TBD
/**
* Generates Painless sorting if the given field is matched or returns default sorting
* This painless script will sort the field in case-insensitive manner
*/
const getSortField = ({ field, direction }: { field: string; direction: string }) => {
if (fieldsRequiredSortingByPainlessScript.includes(field)) {
return {
_script: {
type: 'string',
order: direction,
script: {
source: `doc["${field}"].value.toLowerCase()`,
lang: 'painless',
},
},
};
}
return { [field]: direction };
};
const getAssetsQuery = ({ query, sort }: UseAssetsOptions, pageParam: unknown) => {
return {
index: ASSET_INVENTORY_INDEX_PATTERN,
sort: getMultiFieldsSort(sort),
runtime_mappings: getRuntimeMappingsFromSort(sort),
size: MAX_ASSETS_TO_LOAD,
ignore_unavailable: true,
query: {
...query,
bool: {
...query?.bool,
filter: [...(query?.bool?.filter ?? [])],
must_not: [...(query?.bool?.must_not ?? [])],
},
},
...(pageParam ? { from: pageParam } : {}),
};
};
interface Asset {
'@timestamp': string;
name: string;
risk: number;
criticality: string;
category: string;
}
type LatestAssetsRequest = IKibanaSearchRequest<estypes.SearchRequest>;
type LatestAssetsResponse = IKibanaSearchResponse<estypes.SearchResponse<Asset, never>>;
export function useFetchData(options: UseAssetsOptions) {
const {
data,
notifications: { toasts },
} = useKibana().services;
return useInfiniteQuery(
['asset_inventory', { params: options }],
async ({ pageParam }) => {
const {
rawResponse: { hits },
} = await lastValueFrom(
data.search.search<LatestAssetsRequest, LatestAssetsResponse>({
params: getAssetsQuery(options, pageParam) as LatestAssetsRequest['params'],
})
);
return {
page: hits.hits.map((hit) => buildDataTableRecord(hit as EsHitRecord)),
total: number.is(hits.total) ? hits.total : 0,
};
},
{
enabled: options.enabled,
keepPreviousData: true,
onError: (err: Error) => showErrorToast(toasts, err),
getNextPageParam: (lastPage, allPages) => {
if (lastPage.page.length < options.pageSize) {
return undefined;
}
return allPages.length * options.pageSize;
},
}
);
}

View file

@ -50,6 +50,7 @@ import { AdditionalControls } from '../components/additional_controls';
import { AssetInventorySearchBar } from '../components/search_bar';
import { RiskBadge } from '../components/risk_badge';
import { Filters } from '../components/filters/filters';
import { EmptyState } from '../components/empty_state';
import { useDataViewContext } from '../hooks/data_view_context';
import { useStyles } from '../hooks/use_styles';
@ -58,6 +59,8 @@ import {
type AssetsBaseURLQuery,
type URLQuery,
} from '../hooks/use_asset_inventory_data_table';
import { useFetchData } from '../hooks/use_fetch_data';
import { DEFAULT_VISIBLE_ROWS_PER_PAGE, MAX_ASSETS_TO_LOAD } from '../constants';
const gridStyle: EuiDataGridStyle = {
border: 'horizontal',
@ -66,8 +69,6 @@ const gridStyle: EuiDataGridStyle = {
header: 'underline',
};
const MAX_ASSETS_TO_LOAD = 500; // equivalent to MAX_FINDINGS_TO_LOAD in @kbn/cloud-security-posture-common
const title = i18n.translate('xpack.securitySolution.assetInventory.allAssets.tableRowTypeLabel', {
defaultMessage: 'assets',
});
@ -129,10 +130,7 @@ const getDefaultQuery = ({ query, filters }: AssetsBaseURLQuery): URLQuery => ({
});
export interface AllAssetsProps {
rows: DataTableRecord[];
isLoading: boolean;
height?: number | string;
loadMore: () => void;
nonPersistedFilters?: Filter[];
hasDistributionBar?: boolean;
/**
@ -155,9 +153,6 @@ const getEntity = (row: DataTableRecord): EntityEcs => {
const ASSET_INVENTORY_TABLE_ID = 'asset-inventory-table';
const AllAssets = ({
rows,
isLoading,
loadMore,
nonPersistedFilters,
height,
hasDistributionBar = true,
@ -196,16 +191,35 @@ const AllAssets = ({
};
// -----------------------------------------------------------------------------------------
const {
filters,
pageSize,
sort,
query,
queryError,
urlQuery,
getRowsFromPages,
onChangeItemsPerPage,
onResetFilters,
onSort,
setUrlQuery,
} = assetInventoryDataTable;
const {
// columnsLocalStorageKey,
pageSize,
onChangeItemsPerPage,
setUrlQuery,
onSort,
filters,
data: rowsData,
// error: fetchError,
isFetching,
fetchNextPage: loadMore,
isLoading,
} = useFetchData({
query,
sort,
} = assetInventoryDataTable;
enabled: !queryError,
pageSize: DEFAULT_VISIBLE_ROWS_PER_PAGE,
});
const rows = getRowsFromPages(rowsData?.pages);
const totalHits = rowsData?.pages[0].total || 0;
const [columns, setColumns] = useLocalStorage(
columnsLocalStorageKey,
@ -351,7 +365,7 @@ const AllAssets = ({
const externalAdditionalControls = (
<AdditionalControls
total={rows.length}
total={totalHits}
dataView={dataView}
title={title}
columns={currentColumns}
@ -389,7 +403,7 @@ const AllAssets = ({
};
const loadingState =
isLoading || dataViewIsLoading || dataViewIsRefetching || !dataView
isLoading || isFetching || dataViewIsLoading || dataViewIsRefetching || !dataView
? DataLoadingState.loading
: DataLoadingState.loaded;
@ -397,7 +411,7 @@ const AllAssets = ({
<I18nProvider>
{!dataView ? null : (
<AssetInventorySearchBar
query={getDefaultQuery({ query: { query: '', language: '' }, filters: [] })}
query={urlQuery}
setQuery={setUrlQuery}
loading={loadingState === DataLoadingState.loading}
/>
@ -428,7 +442,11 @@ const AllAssets = ({
/>
</h1>
</EuiTitle>
<Filters onFiltersChange={() => {}} />
<Filters
onFiltersChange={(newFilters: Filter[]) => {
setUrlQuery({ filters: newFilters });
}}
/>
<CellActionsProvider getTriggerCompatibleActions={uiActions.getTriggerCompatibleActions}>
<div
data-test-subj={rest['data-test-subj']}
@ -438,7 +456,9 @@ const AllAssets = ({
}}
>
<EuiProgress size="xs" color="accent" style={loadingStyle} />
{!dataView ? null : (
{!dataView ? null : loadingState === DataLoadingState.loaded && totalHits === 0 ? (
<EmptyState onResetFilters={onResetFilters} />
) : (
<UnifiedDataTable
key={computeDataTableRendering.mode}
className={styles.gridStyle}
@ -457,7 +477,7 @@ const AllAssets = ({
renderDocumentView={EmptyComponent}
sort={sort}
rowsPerPageState={pageSize}
totalHits={rows.length}
totalHits={totalHits}
services={services}
onUpdateRowsPerPage={onChangeItemsPerPage}
rowHeightState={0}

View file

@ -16,6 +16,7 @@ import { PluginTemplateWrapper } from '../common/components/plugin_template_wrap
import { SecurityRoutePageWrapper } from '../common/components/security_route_page_wrapper';
import { DataViewContext } from './hooks/data_view_context';
import { useDataView } from './hooks/use_asset_inventory_data_table/use_data_view';
import { ASSET_INVENTORY_INDEX_PATTERN } from './constants';
const AllAssetsLazy = lazy(() => import('./pages/all_assets'));
@ -30,8 +31,6 @@ const queryClient = new QueryClient({
},
});
const ASSET_INVENTORY_INDEX_PATTERN = 'logs-cloud_asset_inventory.asset_inventory-*';
export const AssetInventoryRoutes = () => {
const dataViewQuery = useDataView(ASSET_INVENTORY_INDEX_PATTERN);
@ -49,7 +48,7 @@ export const AssetInventoryRoutes = () => {
<DataViewContext.Provider value={dataViewContextValue}>
<SecuritySolutionPageWrapper noPadding>
<Suspense fallback={<EuiLoadingSpinner />}>
<AllAssetsLazy rows={[]} isLoading={false} loadMore={() => {}} />
<AllAssetsLazy />
</Suspense>
</SecuritySolutionPageWrapper>
</DataViewContext.Provider>

View file

@ -36,5 +36,4 @@ export const URL_PARAM_KEY = {
timerange: 'timerange',
pageFilter: 'pageFilters',
rulesTable: 'rulesTable',
assetInventory: 'assetInventory',
} as const;