[UnifiedFieldList] Migrate field list components from Lens to UnifiedFieldList (#142758)

* [UnifiedFieldList] Extract FieldsAccordion component from Lens

* [UnifiedFieldList] Extract FieldList component from Lens

* [CI] Auto-commit changed files from 'node scripts/build_plugin_list_docs'

* [UnifiedFieldList] Rename component

* [UnifiedFieldList] Start extracting logic for fetching fields existence info

* [UnifiedFieldList] Start extracting logic for fetching fields existence info

* [UnifiedFieldList] Fix special and runtime fields

* [UnifiedFieldList] Start extracting logic for fetching fields existence info

* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'

* [UnifiedFieldList] Make API stricter

* [UnifiedFieldList] Make sure that key is specified for items

* [UnifiedFieldList] Fetch data for all active data views

* [UnifiedFieldList] Refactor some other occurances

* [UnifiedFieldList] Update more tests

* [UnifiedFieldList] Fix some checks

* [UnifiedFieldList] Some progress on updating tests

* [UnifiedFieldList] Update more tests

* [UnifiedFieldList] Skip redundant request's results

* [UnifiedFieldList] Update more tests

* [UnifiedFieldList] Improve tests

* [UnifiedFieldList] Improve tests

* [UnifiedFieldList] Improve tests

* [UnifiedFieldList] Move grouping into a customizable hook

* [UnifiedFieldList] Fix after the merge

* [UnifiedFieldList] Fix checks

* Revert "[UnifiedFieldList] Fix after the merge"

This reverts commit 500db7ed89.

* [UnifiedFieldList] Handle merge better

* [UnifiedFieldList] Update the naming

* [UnifiedFieldList] Support Selected fields

* [UnifiedFieldList] Update tests

* [UnifiedFieldList] Fix grouping

* [UnifiedFieldList] Update more tests

* [UnifiedFieldList] Fix refetch after adding a field

* [UnifiedFieldList] Load es query builder in async way

* [UnifiedFieldList] Fix a bug in case of renaming a field

* [UnifiedFieldList] Small refactoring

* [UnifiedFieldList] Refactor text based view

* [UnifiedFieldList] Better types support

* [UnifiedFieldList] Simplify props

* [UnifiedFieldList] Fix types

* [UnifiedFieldList] Async loading for FieldListGrouped code

* [UnifiedFieldList] Add more tests

* [UnifiedFieldList] Add more tests

* [UnifiedFieldList] Add more tests

* [UnifiedFieldList] Add more tests

* [UnifiedFieldList] Add more tests

* [UnifiedFieldList] Add more tests

* [UnifiedFieldList] Add more tests

* [UnifiedFieldList] Add docs

* [UnifiedFieldList] Clean up

* [UnifiedFieldList] Fix onNoData callback

* [UnifiedFieldList] Address PR comments

* [UnifiedFieldList] Address PR comments

* [UnifiedFieldList] Support a custom data-test-subj

* [UnifiedFieldList] Fix concurrency handling logic

* [UnifiedFieldList] Remove a generic tooltip message. Lens and Discover will have their own tooltips.

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>

src/plugins/unified_field_list/README.md

@@ -6,6 +6,8 @@ This Kibana plugin contains components and services for field list UI (as in fie
 
 ## Components
 
+* `<FieldListGrouped .../>` - renders a fields list which is split in sections (Selected, Special, Available, Empty, Meta fields). It accepts already grouped fields, please use `useGroupedFields` hook for it. 
+ 
 * `<FieldStats .../>` - loads and renders stats (Top values, Distribution) for a data view field.
 
 * `<FieldVisualizeButton .../>` - renders a button to open this field in Lens.
@@ -13,7 +15,7 @@ This Kibana plugin contains components and services for field list UI (as in fie
 * `<FieldPopover .../>` - a popover container component for a field.
 
 * `<FieldPopoverHeader .../>` - this header component included a field name and common actions.
-* 
+
 * `<FieldPopoverVisualize .../>` - renders Visualize action in the popover footer.
 
 These components can be combined and customized as the following:
@@ -59,6 +61,47 @@ These components can be combined and customized as the following:
 
 * `loadFieldExisting(...)` - returns the loaded existing fields (can also work with Ad-hoc data views)
 
+## Hooks
+
+* `useExistingFieldsFetcher(...)` - this hook is responsible for fetching fields existence info for specified data views. It can be used higher in components tree than `useExistingFieldsReader` hook.
+
+* `useExistingFieldsReader(...)` - you can call this hook to read fields existence info which was fetched by `useExistingFieldsFetcher` hook. Using multiple "reader" hooks from different children components is supported. So you would need only one "fetcher" and as many "reader" hooks as necessary.
+
+* `useGroupedFields(...)` - this hook groups fields list into sections of Selected, Special, Available, Empty, Meta fields.
+
+An example of using hooks together with `<FieldListGrouped .../>`:
+
+```
+const { refetchFieldsExistenceInfo, isProcessing } = useExistingFieldsFetcher({
+  dataViews,
+  query,
+  filters,
+  fromDate,
+  toDate,
+  ...
+});
+const fieldsExistenceReader = useExistingFieldsReader()
+const { fieldGroups } = useGroupedFields({
+  dataViewId: currentDataViewId,
+  allFields,
+  fieldsExistenceReader,
+  ...
+});
+
+// and now we can render a field list
+<FieldListGrouped
+  fieldGroups={fieldGroups}
+  fieldsExistenceStatus={fieldsExistenceReader.getFieldsExistenceStatus(currentDataViewId)}
+  fieldsExistInIndex={!!allFields.length}
+  renderFieldItem={renderFieldItem}
+  screenReaderDescriptionForSearchInputId={fieldSearchDescriptionId}
+/>
+
+// or check whether a field contains data
+const { hasFieldData } = useExistingFieldsReader();
+const hasData = hasFieldData(currentDataViewId, fieldName) // return a boolean
+```
+
 ## Server APIs
 
 * `/api/unified_field_list/field_stats` - returns the loaded field stats (except for Ad-hoc data views)

src/plugins/unified_field_list/public/components/field_list/field_list_grouped.scss

@@ -2,7 +2,7 @@
  * 1. Don't cut off the shadow of the field items
  */
 
-.lnsIndexPatternFieldList {
+.unifiedFieldList__fieldListGrouped {
   @include euiOverflowShadow;
   @include euiScrollBar;
   margin-left: -$euiSize; /* 1 */
@@ -11,7 +11,7 @@
   overflow: auto;
 }
 
-.lnsIndexPatternFieldList__accordionContainer {
+.unifiedFieldList__fieldListGrouped__container {
   padding-top: $euiSizeS;
   position: absolute;
   top: 0;

src/plugins/unified_field_list/public/components/field_list/field_list_grouped.test.tsx

@@ -0,0 +1,413 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import React from 'react';
+import { act } from 'react-dom/test-utils';
+import { stubLogstashDataView as dataView } from '@kbn/data-views-plugin/common/data_view.stub';
+import { EuiText, EuiLoadingSpinner } from '@elastic/eui';
+import { mountWithIntl } from '@kbn/test-jest-helpers';
+import { DataViewField } from '@kbn/data-views-plugin/common';
+import { ReactWrapper } from 'enzyme';
+import { dataViewPluginMocks } from '@kbn/data-views-plugin/public/mocks';
+import FieldListGrouped, { type FieldListGroupedProps } from './field_list_grouped';
+import { ExistenceFetchStatus } from '../../types';
+import { FieldsAccordion } from './fields_accordion';
+import { NoFieldsCallout } from './no_fields_callout';
+import { useGroupedFields, type GroupedFieldsParams } from '../../hooks/use_grouped_fields';
+
+describe('UnifiedFieldList <FieldListGrouped /> + useGroupedFields()', () => {
+  let defaultProps: FieldListGroupedProps<DataViewField>;
+  let mockedServices: GroupedFieldsParams<DataViewField>['services'];
+  const allFields = dataView.fields;
+  // 5 times more fields. Added fields will be treated as empty as they are not a part of the data view.
+  const manyFields = [...new Array(5)].flatMap((_, index) =>
+    allFields.map((field) => {
+      return new DataViewField({ ...field.toSpec(), name: `${field.name}${index || ''}` });
+    })
+  );
+
+  beforeEach(() => {
+    const dataViews = dataViewPluginMocks.createStartContract();
+    mockedServices = {
+      dataViews,
+    };
+
+    dataViews.get.mockImplementation(async (id: string) => {
+      return dataView;
+    });
+
+    defaultProps = {
+      fieldGroups: {},
+      fieldsExistenceStatus: ExistenceFetchStatus.succeeded,
+      fieldsExistInIndex: true,
+      screenReaderDescriptionForSearchInputId: 'testId',
+      renderFieldItem: jest.fn(({ field, itemIndex, groupIndex }) => (
+        <EuiText
+          data-test-subj="testFieldItem"
+          data-name={`${field.name}-${groupIndex}-${itemIndex}`}
+        >
+          {field.name}
+        </EuiText>
+      )),
+    };
+  });
+
+  interface WrapperProps {
+    listProps: Omit<FieldListGroupedProps<DataViewField>, 'fieldGroups'>;
+    hookParams: Omit<GroupedFieldsParams<DataViewField>, 'services'>;
+  }
+
+  async function mountGroupedList({ listProps, hookParams }: WrapperProps): Promise<ReactWrapper> {
+    const Wrapper: React.FC<WrapperProps> = (props) => {
+      const { fieldGroups } = useGroupedFields({
+        ...props.hookParams,
+        services: mockedServices,
+      });
+
+      return <FieldListGrouped {...props.listProps} fieldGroups={fieldGroups} />;
+    };
+
+    let wrapper: ReactWrapper;
+    await act(async () => {
+      wrapper = await mountWithIntl(<Wrapper hookParams={hookParams} listProps={listProps} />);
+      // wait for lazy modules if any
+      await new Promise((resolve) => setTimeout(resolve, 0));
+      await wrapper.update();
+    });
+
+    return wrapper!;
+  }
+
+  it('renders correctly in empty state', () => {
+    const wrapper = mountWithIntl(
+      <FieldListGrouped
+        {...defaultProps}
+        fieldGroups={{}}
+        fieldsExistenceStatus={ExistenceFetchStatus.unknown}
+      />
+    );
+
+    expect(
+      wrapper.find(`#${defaultProps.screenReaderDescriptionForSearchInputId}`).first().text()
+    ).toBe('');
+  });
+
+  it('renders correctly in loading state', async () => {
+    const wrapper = await mountGroupedList({
+      listProps: {
+        ...defaultProps,
+        fieldsExistenceStatus: ExistenceFetchStatus.unknown,
+      },
+      hookParams: {
+        dataViewId: dataView.id!,
+        allFields,
+      },
+    });
+
+    expect(wrapper.find(FieldListGrouped).prop('fieldsExistenceStatus')).toBe(
+      ExistenceFetchStatus.unknown
+    );
+    expect(
+      wrapper.find(`#${defaultProps.screenReaderDescriptionForSearchInputId}`).first().text()
+    ).toBe('');
+    expect(wrapper.find(FieldsAccordion)).toHaveLength(3);
+    expect(wrapper.find(EuiLoadingSpinner)).toHaveLength(3);
+    expect(
+      wrapper.find(FieldsAccordion).map((accordion) => accordion.prop('hasLoaded'))
+    ).toStrictEqual([false, false, false]);
+    expect(wrapper.find(NoFieldsCallout)).toHaveLength(0);
+
+    await act(async () => {
+      await wrapper.setProps({
+        listProps: {
+          ...defaultProps,
+          fieldsExistenceStatus: ExistenceFetchStatus.succeeded,
+        },
+      });
+      await wrapper.update();
+    });
+
+    expect(wrapper.find(FieldListGrouped).prop('fieldsExistenceStatus')).toBe(
+      ExistenceFetchStatus.succeeded
+    );
+    expect(
+      wrapper.find(`#${defaultProps.screenReaderDescriptionForSearchInputId}`).first().text()
+    ).toBe('25 available fields. 0 empty fields. 3 meta fields.');
+    expect(wrapper.find(FieldsAccordion)).toHaveLength(3);
+    expect(wrapper.find(EuiLoadingSpinner)).toHaveLength(0);
+    expect(
+      wrapper.find(FieldsAccordion).map((accordion) => accordion.prop('hasLoaded'))
+    ).toStrictEqual([true, true, true]);
+    expect(
+      wrapper.find(FieldsAccordion).map((accordion) => accordion.prop('paginatedFields').length)
+    ).toStrictEqual([25, 0, 0]);
+    expect(wrapper.find(NoFieldsCallout)).toHaveLength(1);
+  });
+
+  it('renders correctly in failed state', async () => {
+    const wrapper = await mountGroupedList({
+      listProps: {
+        ...defaultProps,
+        fieldsExistenceStatus: ExistenceFetchStatus.failed,
+      },
+      hookParams: {
+        dataViewId: dataView.id!,
+        allFields,
+      },
+    });
+
+    expect(wrapper.find(FieldListGrouped).prop('fieldsExistenceStatus')).toBe(
+      ExistenceFetchStatus.failed
+    );
+    expect(
+      wrapper.find(`#${defaultProps.screenReaderDescriptionForSearchInputId}`).first().text()
+    ).toBe('25 available fields. 0 empty fields. 3 meta fields.');
+    expect(wrapper.find(FieldsAccordion)).toHaveLength(3);
+    expect(wrapper.find(EuiLoadingSpinner)).toHaveLength(0);
+    expect(
+      wrapper.find(FieldsAccordion).map((accordion) => accordion.prop('hasLoaded'))
+    ).toStrictEqual([true, true, true]);
+    expect(
+      wrapper.find(FieldsAccordion).map((accordion) => accordion.prop('showExistenceFetchError'))
+    ).toStrictEqual([true, true, true]);
+  });
+
+  it('renders correctly in no fields state', async () => {
+    const wrapper = await mountGroupedList({
+      listProps: {
+        ...defaultProps,
+        fieldsExistInIndex: false,
+        fieldsExistenceStatus: ExistenceFetchStatus.failed,
+      },
+      hookParams: {
+        dataViewId: dataView.id!,
+        allFields: [],
+      },
+    });
+
+    expect(
+      wrapper.find(`#${defaultProps.screenReaderDescriptionForSearchInputId}`).first().text()
+    ).toBe('0 available fields. 0 empty fields. 0 meta fields.');
+    expect(wrapper.find(FieldsAccordion)).toHaveLength(3);
+    expect(wrapper.find(EuiLoadingSpinner)).toHaveLength(0);
+    expect(
+      wrapper.find(NoFieldsCallout).map((callout) => callout.prop('fieldsExistInIndex'))
+    ).toStrictEqual([false, false, false]);
+  });
+
+  it('renders correctly for text-based queries (no data view)', async () => {
+    const wrapper = await mountGroupedList({
+      listProps: {
+        ...defaultProps,
+        fieldsExistenceStatus: ExistenceFetchStatus.succeeded,
+      },
+      hookParams: {
+        dataViewId: null,
+        allFields,
+        onSelectedFieldFilter: (field) => field.name === 'bytes',
+      },
+    });
+
+    expect(
+      wrapper.find(`#${defaultProps.screenReaderDescriptionForSearchInputId}`).first().text()
+    ).toBe('1 selected field. 28 available fields.');
+    expect(
+      wrapper.find(FieldsAccordion).map((accordion) => accordion.prop('paginatedFields').length)
+    ).toStrictEqual([1, 28]);
+  });
+
+  it('renders correctly when Meta gets open', async () => {
+    const wrapper = await mountGroupedList({
+      listProps: {
+        ...defaultProps,
+        fieldsExistenceStatus: ExistenceFetchStatus.succeeded,
+      },
+      hookParams: {
+        dataViewId: dataView.id!,
+        allFields,
+      },
+    });
+
+    expect(
+      wrapper.find(`#${defaultProps.screenReaderDescriptionForSearchInputId}`).first().text()
+    ).toBe('25 available fields. 0 empty fields. 3 meta fields.');
+    expect(
+      wrapper.find(FieldsAccordion).map((accordion) => accordion.prop('paginatedFields').length)
+    ).toStrictEqual([25, 0, 0]);
+
+    await act(async () => {
+      await wrapper
+        .find('[data-test-subj="fieldListGroupedMetaFields"]')
+        .find('button')
+        .first()
+        .simulate('click');
+      await wrapper.update();
+    });
+
+    expect(
+      wrapper.find(FieldsAccordion).map((accordion) => accordion.prop('paginatedFields').length)
+    ).toStrictEqual([25, 0, 3]);
+  });
+
+  it('renders correctly when paginated', async () => {
+    const wrapper = await mountGroupedList({
+      listProps: {
+        ...defaultProps,
+        fieldsExistenceStatus: ExistenceFetchStatus.succeeded,
+      },
+      hookParams: {
+        dataViewId: dataView.id!,
+        allFields: manyFields,
+      },
+    });
+
+    expect(
+      wrapper.find(`#${defaultProps.screenReaderDescriptionForSearchInputId}`).first().text()
+    ).toBe('25 available fields. 112 empty fields. 3 meta fields.');
+    expect(
+      wrapper.find(FieldsAccordion).map((accordion) => accordion.prop('paginatedFields').length)
+    ).toStrictEqual([25, 0, 0]);
+
+    await act(async () => {
+      await wrapper
+        .find('[data-test-subj="fieldListGroupedEmptyFields"]')
+        .find('button')
+        .first()
+        .simulate('click');
+      await wrapper.update();
+    });
+
+    expect(
+      wrapper.find(FieldsAccordion).map((accordion) => accordion.prop('paginatedFields').length)
+    ).toStrictEqual([25, 50, 0]);
+
+    await act(async () => {
+      await wrapper
+        .find('[data-test-subj="fieldListGroupedMetaFields"]')
+        .find('button')
+        .first()
+        .simulate('click');
+      await wrapper.update();
+    });
+
+    expect(
+      wrapper.find(FieldsAccordion).map((accordion) => accordion.prop('paginatedFields').length)
+    ).toStrictEqual([25, 88, 0]);
+  });
+
+  it('renders correctly when filtered', async () => {
+    const hookParams = {
+      dataViewId: dataView.id!,
+      allFields: manyFields,
+    };
+    const wrapper = await mountGroupedList({
+      listProps: {
+        ...defaultProps,
+        fieldsExistenceStatus: ExistenceFetchStatus.succeeded,
+      },
+      hookParams,
+    });
+
+    expect(
+      wrapper.find(`#${defaultProps.screenReaderDescriptionForSearchInputId}`).first().text()
+    ).toBe('25 available fields. 112 empty fields. 3 meta fields.');
+
+    await act(async () => {
+      await wrapper.setProps({
+        hookParams: {
+          ...hookParams,
+          onFilterField: (field: DataViewField) => field.name.startsWith('@'),
+        },
+      });
+      await wrapper.update();
+    });
+
+    expect(
+      wrapper.find(`#${defaultProps.screenReaderDescriptionForSearchInputId}`).first().text()
+    ).toBe('2 available fields. 8 empty fields. 0 meta fields.');
+
+    await act(async () => {
+      await wrapper.setProps({
+        hookParams: {
+          ...hookParams,
+          onFilterField: (field: DataViewField) => field.name.startsWith('_'),
+        },
+      });
+      await wrapper.update();
+    });
+
+    expect(
+      wrapper.find(`#${defaultProps.screenReaderDescriptionForSearchInputId}`).first().text()
+    ).toBe('0 available fields. 12 empty fields. 3 meta fields.');
+  });
+
+  it('renders correctly when non-supported fields are filtered out', async () => {
+    const hookParams = {
+      dataViewId: dataView.id!,
+      allFields: manyFields,
+    };
+    const wrapper = await mountGroupedList({
+      listProps: {
+        ...defaultProps,
+        fieldsExistenceStatus: ExistenceFetchStatus.succeeded,
+      },
+      hookParams,
+    });
+
+    expect(
+      wrapper.find(`#${defaultProps.screenReaderDescriptionForSearchInputId}`).first().text()
+    ).toBe('25 available fields. 112 empty fields. 3 meta fields.');
+
+    await act(async () => {
+      await wrapper.setProps({
+        hookParams: {
+          ...hookParams,
+          onSupportedFieldFilter: (field: DataViewField) => field.aggregatable,
+        },
+      });
+      await wrapper.update();
+    });
+
+    expect(
+      wrapper.find(`#${defaultProps.screenReaderDescriptionForSearchInputId}`).first().text()
+    ).toBe('23 available fields. 104 empty fields. 3 meta fields.');
+  });
+
+  it('renders correctly when selected fields are present', async () => {
+    const hookParams = {
+      dataViewId: dataView.id!,
+      allFields: manyFields,
+    };
+    const wrapper = await mountGroupedList({
+      listProps: {
+        ...defaultProps,
+        fieldsExistenceStatus: ExistenceFetchStatus.succeeded,
+      },
+      hookParams,
+    });
+
+    expect(
+      wrapper.find(`#${defaultProps.screenReaderDescriptionForSearchInputId}`).first().text()
+    ).toBe('25 available fields. 112 empty fields. 3 meta fields.');
+
+    await act(async () => {
+      await wrapper.setProps({
+        hookParams: {
+          ...hookParams,
+          onSelectedFieldFilter: (field: DataViewField) =>
+            ['@timestamp', 'bytes'].includes(field.name),
+        },
+      });
+      await wrapper.update();
+    });
+
+    expect(
+      wrapper.find(`#${defaultProps.screenReaderDescriptionForSearchInputId}`).first().text()
+    ).toBe('2 selected fields. 25 available fields. 112 empty fields. 3 meta fields.');
+  });
+});

src/plugins/unified_field_list/public/components/field_list/field_list_grouped.tsx

@@ -0,0 +1,245 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { partition, throttle } from 'lodash';
+import React, { useState, Fragment, useCallback, useMemo } from 'react';
+import { i18n } from '@kbn/i18n';
+import { EuiScreenReaderOnly, EuiSpacer } from '@elastic/eui';
+import { type DataViewField } from '@kbn/data-views-plugin/common';
+import { NoFieldsCallout } from './no_fields_callout';
+import { FieldsAccordion, type FieldsAccordionProps } from './fields_accordion';
+import type { FieldListGroups, FieldListItem } from '../../types';
+import { ExistenceFetchStatus } from '../../types';
+import './field_list_grouped.scss';
+
+const PAGINATION_SIZE = 50;
+
+function getDisplayedFieldsLength<T extends FieldListItem>(
+  fieldGroups: FieldListGroups<T>,
+  accordionState: Partial<Record<string, boolean>>
+) {
+  return Object.entries(fieldGroups)
+    .filter(([key]) => accordionState[key])
+    .reduce((allFieldCount, [, { fields }]) => allFieldCount + fields.length, 0);
+}
+
+export interface FieldListGroupedProps<T extends FieldListItem> {
+  fieldGroups: FieldListGroups<T>;
+  fieldsExistenceStatus: ExistenceFetchStatus;
+  fieldsExistInIndex: boolean;
+  renderFieldItem: FieldsAccordionProps<T>['renderFieldItem'];
+  screenReaderDescriptionForSearchInputId?: string;
+  'data-test-subj'?: string;
+}
+
+function InnerFieldListGrouped<T extends FieldListItem = DataViewField>({
+  fieldGroups,
+  fieldsExistenceStatus,
+  fieldsExistInIndex,
+  renderFieldItem,
+  screenReaderDescriptionForSearchInputId,
+  'data-test-subj': dataTestSubject = 'fieldListGrouped',
+}: FieldListGroupedProps<T>) {
+  const hasSyncedExistingFields =
+    fieldsExistenceStatus && fieldsExistenceStatus !== ExistenceFetchStatus.unknown;
+
+  const [fieldGroupsToShow, fieldGroupsToCollapse] = partition(
+    Object.entries(fieldGroups),
+    ([, { showInAccordion }]) => showInAccordion
+  );
+  const [pageSize, setPageSize] = useState(PAGINATION_SIZE);
+  const [scrollContainer, setScrollContainer] = useState<Element | undefined>(undefined);
+  const [accordionState, setAccordionState] = useState<Partial<Record<string, boolean>>>(() =>
+    Object.fromEntries(
+      fieldGroupsToShow.map(([key, { isInitiallyOpen }]) => [key, isInitiallyOpen])
+    )
+  );
+
+  const lazyScroll = useCallback(() => {
+    if (scrollContainer) {
+      const nearBottom =
+        scrollContainer.scrollTop + scrollContainer.clientHeight >
+        scrollContainer.scrollHeight * 0.9;
+      if (nearBottom) {
+        setPageSize(
+          Math.max(
+            PAGINATION_SIZE,
+            Math.min(
+              pageSize + PAGINATION_SIZE * 0.5,
+              getDisplayedFieldsLength<T>(fieldGroups, accordionState)
+            )
+          )
+        );
+      }
+    }
+  }, [scrollContainer, pageSize, setPageSize, fieldGroups, accordionState]);
+
+  const paginatedFields = useMemo(() => {
+    let remainingItems = pageSize;
+    return Object.fromEntries(
+      fieldGroupsToShow.map(([key, fieldGroup]) => {
+        if (!accordionState[key] || remainingItems <= 0) {
+          return [key, []];
+        }
+        const slicedFieldList = fieldGroup.fields.slice(0, remainingItems);
+        remainingItems = remainingItems - slicedFieldList.length;
+        return [key, slicedFieldList];
+      })
+    );
+  }, [pageSize, fieldGroupsToShow, accordionState]);
+
+  return (
+    <div
+      className="unifiedFieldList__fieldListGrouped"
+      ref={(el) => {
+        if (el && !el.dataset.dynamicScroll) {
+          el.dataset.dynamicScroll = 'true';
+          setScrollContainer(el);
+        }
+      }}
+      onScroll={throttle(lazyScroll, 100)}
+    >
+      <div className="unifiedFieldList__fieldListGrouped__container">
+        {Boolean(screenReaderDescriptionForSearchInputId) && (
+          <EuiScreenReaderOnly>
+            <div
+              aria-live="polite"
+              id={screenReaderDescriptionForSearchInputId}
+              data-test-subj={`${dataTestSubject}__ariaDescription`}
+            >
+              {hasSyncedExistingFields
+                ? [
+                    fieldGroups.SelectedFields &&
+                      (!fieldGroups.SelectedFields?.hideIfEmpty ||
+                        fieldGroups.SelectedFields?.fields?.length > 0) &&
+                      i18n.translate(
+                        'unifiedFieldList.fieldListGrouped.fieldSearchForSelectedFieldsLiveRegion',
+                        {
+                          defaultMessage:
+                            '{selectedFields} selected {selectedFields, plural, one {field} other {fields}}.',
+                          values: {
+                            selectedFields: fieldGroups.SelectedFields?.fields?.length || 0,
+                          },
+                        }
+                      ),
+                    fieldGroups.AvailableFields?.fields &&
+                      i18n.translate(
+                        'unifiedFieldList.fieldListGrouped.fieldSearchForAvailableFieldsLiveRegion',
+                        {
+                          defaultMessage:
+                            '{availableFields} available {availableFields, plural, one {field} other {fields}}.',
+                          values: {
+                            availableFields: fieldGroups.AvailableFields.fields.length,
+                          },
+                        }
+                      ),
+                    fieldGroups.EmptyFields &&
+                      (!fieldGroups.EmptyFields?.hideIfEmpty ||
+                        fieldGroups.EmptyFields?.fields?.length > 0) &&
+                      i18n.translate(
+                        'unifiedFieldList.fieldListGrouped.fieldSearchForEmptyFieldsLiveRegion',
+                        {
+                          defaultMessage:
+                            '{emptyFields} empty {emptyFields, plural, one {field} other {fields}}.',
+                          values: {
+                            emptyFields: fieldGroups.EmptyFields?.fields?.length || 0,
+                          },
+                        }
+                      ),
+                    fieldGroups.MetaFields &&
+                      (!fieldGroups.MetaFields?.hideIfEmpty ||
+                        fieldGroups.MetaFields?.fields?.length > 0) &&
+                      i18n.translate(
+                        'unifiedFieldList.fieldListGrouped.fieldSearchForMetaFieldsLiveRegion',
+                        {
+                          defaultMessage:
+                            '{metaFields} meta {metaFields, plural, one {field} other {fields}}.',
+                          values: {
+                            metaFields: fieldGroups.MetaFields?.fields?.length || 0,
+                          },
+                        }
+                      ),
+                  ]
+                    .filter(Boolean)
+                    .join(' ')
+                : ''}
+            </div>
+          </EuiScreenReaderOnly>
+        )}
+        <ul>
+          {fieldGroupsToCollapse.flatMap(([, { fields }]) =>
+            fields.map((field, index) => (
+              <Fragment key={field.name}>
+                {renderFieldItem({ field, itemIndex: index, groupIndex: 0, hideDetails: true })}
+              </Fragment>
+            ))
+          )}
+        </ul>
+        <EuiSpacer size="s" />
+        {fieldGroupsToShow.map(([key, fieldGroup], index) => {
+          const hidden = Boolean(fieldGroup.hideIfEmpty) && !fieldGroup.fields.length;
+          if (hidden) {
+            return null;
+          }
+          return (
+            <Fragment key={key}>
+              <FieldsAccordion<T>
+                id={`${dataTestSubject}${key}`}
+                initialIsOpen={Boolean(accordionState[key])}
+                label={fieldGroup.title}
+                helpTooltip={fieldGroup.helpText}
+                hideDetails={fieldGroup.hideDetails}
+                hasLoaded={hasSyncedExistingFields}
+                fieldsCount={fieldGroup.fields.length}
+                isFiltered={fieldGroup.fieldCount !== fieldGroup.fields.length}
+                paginatedFields={paginatedFields[key]}
+                groupIndex={index + 1}
+                onToggle={(open) => {
+                  setAccordionState((s) => ({
+                    ...s,
+                    [key]: open,
+                  }));
+                  const displayedFieldLength = getDisplayedFieldsLength(fieldGroups, {
+                    ...accordionState,
+                    [key]: open,
+                  });
+                  setPageSize(
+                    Math.max(
+                      PAGINATION_SIZE,
+                      Math.min(Math.ceil(pageSize * 1.5), displayedFieldLength)
+                    )
+                  );
+                }}
+                showExistenceFetchError={fieldsExistenceStatus === ExistenceFetchStatus.failed}
+                showExistenceFetchTimeout={fieldsExistenceStatus === ExistenceFetchStatus.failed} // TODO: deprecate timeout logic?
+                renderCallout={() => (
+                  <NoFieldsCallout
+                    isAffectedByGlobalFilter={fieldGroup.isAffectedByGlobalFilter}
+                    isAffectedByTimerange={fieldGroup.isAffectedByTimeFilter}
+                    isAffectedByFieldFilter={fieldGroup.fieldCount !== fieldGroup.fields.length}
+                    fieldsExistInIndex={!!fieldsExistInIndex}
+                    defaultNoFieldsMessage={fieldGroup.defaultNoFieldsMessage}
+                  />
+                )}
+                renderFieldItem={renderFieldItem}
+              />
+              <EuiSpacer size="m" />
+            </Fragment>
+          );
+        })}
+      </div>
+    </div>
+  );
+}
+
+export type GenericFieldListGrouped = typeof InnerFieldListGrouped;
+const FieldListGrouped = React.memo(InnerFieldListGrouped) as GenericFieldListGrouped;
+
+// Necessary for React.lazy
+// eslint-disable-next-line import/no-default-export
+export default FieldListGrouped;

src/plugins/unified_field_list/public/components/field_list/fields_accordion.scss

@@ -0,0 +1,8 @@
+.unifiedFieldList__fieldsAccordion__titleTooltip {
+  margin-right: $euiSizeXS;
+}
+
+.unifiedFieldList__fieldsAccordion__fieldItems {
+  // Quick fix for making sure the shadow and focus rings are visible outside the accordion bounds
+  padding: $euiSizeXS;
+}

src/plugins/unified_field_list/public/components/field_list/fields_accordion.test.tsx

@@ -0,0 +1,62 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import React from 'react';
+import { stubLogstashDataView as dataView } from '@kbn/data-views-plugin/common/data_view.stub';
+import { EuiLoadingSpinner, EuiNotificationBadge, EuiText } from '@elastic/eui';
+import { mountWithIntl } from '@kbn/test-jest-helpers';
+import { FieldsAccordion, FieldsAccordionProps } from './fields_accordion';
+import { FieldListItem } from '../../types';
+
+describe('UnifiedFieldList <FieldsAccordion />', () => {
+  let defaultProps: FieldsAccordionProps<FieldListItem>;
+  const paginatedFields = dataView.fields;
+
+  beforeEach(() => {
+    defaultProps = {
+      initialIsOpen: true,
+      onToggle: jest.fn(),
+      groupIndex: 0,
+      id: 'id',
+      label: 'label-test',
+      hasLoaded: true,
+      fieldsCount: paginatedFields.length,
+      isFiltered: false,
+      paginatedFields,
+      renderCallout: () => <div id="lens-test-callout">Callout</div>,
+      renderFieldItem: ({ field }) => <EuiText key={field.name}>{field.name}</EuiText>,
+    };
+  });
+
+  it('renders fields correctly', () => {
+    const wrapper = mountWithIntl(<FieldsAccordion {...defaultProps} />);
+    expect(wrapper.find(EuiText)).toHaveLength(paginatedFields.length + 1); // + title
+    expect(wrapper.find(EuiText).first().text()).toBe(defaultProps.label);
+    expect(wrapper.find(EuiText).at(1).text()).toBe(paginatedFields[0].name);
+    expect(wrapper.find(EuiText).last().text()).toBe(
+      paginatedFields[paginatedFields.length - 1].name
+    );
+  });
+
+  it('renders callout if no fields', () => {
+    const wrapper = mountWithIntl(
+      <FieldsAccordion {...defaultProps} fieldsCount={0} paginatedFields={[]} />
+    );
+    expect(wrapper.find('#lens-test-callout').length).toEqual(1);
+  });
+
+  it('renders accented notificationBadge state if isFiltered', () => {
+    const wrapper = mountWithIntl(<FieldsAccordion {...defaultProps} isFiltered={true} />);
+    expect(wrapper.find(EuiNotificationBadge).prop('color')).toEqual('accent');
+  });
+
+  it('renders spinner if has not loaded', () => {
+    const wrapper = mountWithIntl(<FieldsAccordion {...defaultProps} hasLoaded={false} />);
+    expect(wrapper.find(EuiLoadingSpinner).length).toEqual(1);
+  });
+});

src/plugins/unified_field_list/public/components/field_list/fields_accordion.tsx

@@ -1,12 +1,12 @@
 /*
  * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
  * or more contributor license agreements. Licensed under the Elastic License
- * 2.0; you may not use this file except in compliance with the Elastic License
- * 2.0.
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
  */
 
-import './datapanel.scss';
-import React, { memo, useCallback, useMemo } from 'react';
+import React, { useMemo, Fragment } from 'react';
 import { i18n } from '@kbn/i18n';
 import {
   EuiText,
@@ -17,26 +17,11 @@ import {
   EuiIconTip,
 } from '@elastic/eui';
 import classNames from 'classnames';
-import { FieldFormatsStart } from '@kbn/field-formats-plugin/public';
-import { Filter } from '@kbn/es-query';
-import type { Query } from '@kbn/es-query';
-import { ChartsPluginSetup } from '@kbn/charts-plugin/public';
-import { UiActionsStart } from '@kbn/ui-actions-plugin/public';
-import { FieldItem } from './field_item';
-import type { DatasourceDataPanelProps, IndexPattern, IndexPatternField } from '../../types';
+import { type DataViewField } from '@kbn/data-views-plugin/common';
+import type { FieldListItem } from '../../types';
+import './fields_accordion.scss';
 
-export interface FieldItemSharedProps {
-  core: DatasourceDataPanelProps['core'];
-  fieldFormats: FieldFormatsStart;
-  chartsThemeService: ChartsPluginSetup['theme'];
-  indexPattern: IndexPattern;
-  highlight?: string;
-  query: Query;
-  dateRange: DatasourceDataPanelProps['dateRange'];
-  filters: Filter[];
-}
-
-export interface FieldsAccordionProps {
+export interface FieldsAccordionProps<T extends FieldListItem> {
   initialIsOpen: boolean;
   onToggle: (open: boolean) => void;
   id: string;
@@ -44,23 +29,22 @@ export interface FieldsAccordionProps {
   helpTooltip?: string;
   hasLoaded: boolean;
   fieldsCount: number;
+  hideDetails?: boolean;
   isFiltered: boolean;
-  paginatedFields: IndexPatternField[];
-  fieldProps: FieldItemSharedProps;
-  renderCallout: JSX.Element;
-  exists: (field: IndexPatternField) => boolean;
+  groupIndex: number;
+  paginatedFields: T[];
+  renderFieldItem: (params: {
+    field: T;
+    hideDetails?: boolean;
+    itemIndex: number;
+    groupIndex: number;
+  }) => JSX.Element;
+  renderCallout: () => JSX.Element;
   showExistenceFetchError?: boolean;
   showExistenceFetchTimeout?: boolean;
-  hideDetails?: boolean;
-  groupIndex: number;
-  dropOntoWorkspace: DatasourceDataPanelProps['dropOntoWorkspace'];
-  hasSuggestionForField: DatasourceDataPanelProps['hasSuggestionForField'];
-  editField?: (name: string) => void;
-  removeField?: (name: string) => void;
-  uiActions: UiActionsStart;
 }
 
-export const FieldsAccordion = memo(function InnerFieldsAccordion({
+function InnerFieldsAccordion<T extends FieldListItem = DataViewField>({
   initialIsOpen,
   onToggle,
   id,
@@ -68,56 +52,21 @@ export const FieldsAccordion = memo(function InnerFieldsAccordion({
   helpTooltip,
   hasLoaded,
   fieldsCount,
-  isFiltered,
-  paginatedFields,
-  fieldProps,
-  renderCallout,
-  exists,
   hideDetails,
+  isFiltered,
+  groupIndex,
+  paginatedFields,
+  renderFieldItem,
+  renderCallout,
   showExistenceFetchError,
   showExistenceFetchTimeout,
-  groupIndex,
-  dropOntoWorkspace,
-  hasSuggestionForField,
-  editField,
-  removeField,
-  uiActions,
-}: FieldsAccordionProps) {
-  const renderField = useCallback(
-    (field: IndexPatternField, index) => (
-      <FieldItem
-        {...fieldProps}
-        key={field.name}
-        field={field}
-        exists={exists(field)}
-        hideDetails={hideDetails}
-        itemIndex={index}
-        groupIndex={groupIndex}
-        dropOntoWorkspace={dropOntoWorkspace}
-        hasSuggestionForField={hasSuggestionForField}
-        editField={editField}
-        removeField={removeField}
-        uiActions={uiActions}
-      />
-    ),
-    [
-      fieldProps,
-      exists,
-      hideDetails,
-      dropOntoWorkspace,
-      hasSuggestionForField,
-      groupIndex,
-      editField,
-      removeField,
-      uiActions,
-    ]
-  );
-
+}: FieldsAccordionProps<T>) {
   const renderButton = useMemo(() => {
     const titleClassname = classNames({
       // eslint-disable-next-line @typescript-eslint/naming-convention
-      lnsInnerIndexPatternDataPanel__titleTooltip: !!helpTooltip,
+      unifiedFieldList__fieldsAccordion__titleTooltip: !!helpTooltip,
     });
+
     return (
       <EuiText size="xs">
         <strong className={titleClassname}>{label}</strong>
@@ -142,12 +91,12 @@ export const FieldsAccordion = memo(function InnerFieldsAccordion({
     if (showExistenceFetchError) {
       return (
         <EuiIconTip
-          aria-label={i18n.translate('xpack.lens.indexPattern.existenceErrorAriaLabel', {
+          aria-label={i18n.translate('unifiedFieldList.fieldsAccordion.existenceErrorAriaLabel', {
             defaultMessage: 'Existence fetch failed',
           })}
           type="alert"
           color="warning"
-          content={i18n.translate('xpack.lens.indexPattern.existenceErrorLabel', {
+          content={i18n.translate('unifiedFieldList.fieldsAccordion.existenceErrorLabel', {
             defaultMessage: "Field information can't be loaded",
           })}
         />
@@ -156,12 +105,12 @@ export const FieldsAccordion = memo(function InnerFieldsAccordion({
     if (showExistenceFetchTimeout) {
       return (
         <EuiIconTip
-          aria-label={i18n.translate('xpack.lens.indexPattern.existenceTimeoutAriaLabel', {
+          aria-label={i18n.translate('unifiedFieldList.fieldsAccordion.existenceTimeoutAriaLabel', {
             defaultMessage: 'Existence fetch timed out',
           })}
           type="clock"
           color="warning"
-          content={i18n.translate('xpack.lens.indexPattern.existenceTimeoutLabel', {
+          content={i18n.translate('unifiedFieldList.fieldsAccordion.existenceTimeoutLabel', {
             defaultMessage: 'Field information took too long',
           })}
         />
@@ -194,12 +143,19 @@ export const FieldsAccordion = memo(function InnerFieldsAccordion({
       <EuiSpacer size="s" />
       {hasLoaded &&
         (!!fieldsCount ? (
-          <ul className="lnsInnerIndexPatternDataPanel__fieldItems">
-            {paginatedFields && paginatedFields.map(renderField)}
+          <ul className="unifiedFieldList__fieldsAccordion__fieldItems">
+            {paginatedFields &&
+              paginatedFields.map((field, index) => (
+                <Fragment key={field.name}>
+                  {renderFieldItem({ field, itemIndex: index, groupIndex, hideDetails })}
+                </Fragment>
+              ))}
           </ul>
         ) : (
-          renderCallout
+          renderCallout()
         ))}
     </EuiAccordion>
   );
-});
+}
+
+export const FieldsAccordion = React.memo(InnerFieldsAccordion) as typeof InnerFieldsAccordion;

src/plugins/unified_field_list/public/components/field_list/index.tsx

@@ -0,0 +1,31 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import React, { Fragment } from 'react';
+import { type DataViewField } from '@kbn/data-views-plugin/common';
+import type { FieldListGroupedProps, GenericFieldListGrouped } from './field_list_grouped';
+import { type FieldListItem } from '../../types';
+
+const Fallback = () => <Fragment />;
+
+const LazyFieldListGrouped = React.lazy(
+  () => import('./field_list_grouped')
+) as GenericFieldListGrouped;
+
+function WrappedFieldListGrouped<T extends FieldListItem = DataViewField>(
+  props: FieldListGroupedProps<T>
+) {
+  return (
+    <React.Suspense fallback={<Fallback />}>
+      <LazyFieldListGrouped<T> {...props} />
+    </React.Suspense>
+  );
+}
+
+export const FieldListGrouped = WrappedFieldListGrouped;
+export type { FieldListGroupedProps };

src/plugins/unified_field_list/public/components/field_list/no_fields_callout.test.tsx

@@ -1,17 +1,18 @@
 /*
  * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
  * or more contributor license agreements. Licensed under the Elastic License
- * 2.0; you may not use this file except in compliance with the Elastic License
- * 2.0.
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
  */
 
 import React from 'react';
 import { shallow } from 'enzyme';
 import { NoFieldsCallout } from './no_fields_callout';
 
-describe('NoFieldCallout', () => {
+describe('UnifiedFieldList <NoFieldCallout />', () => {
   it('renders correctly for index with no fields', () => {
-    const component = shallow(<NoFieldsCallout existFieldsInIndex={false} />);
+    const component = shallow(<NoFieldsCallout fieldsExistInIndex={false} />);
     expect(component).toMatchInlineSnapshot(`
       <EuiCallOut
         color="warning"
@@ -21,7 +22,7 @@ describe('NoFieldCallout', () => {
     `);
   });
   it('renders correctly when empty with no filters/timerange reasons', () => {
-    const component = shallow(<NoFieldsCallout existFieldsInIndex={true} />);
+    const component = shallow(<NoFieldsCallout fieldsExistInIndex={true} />);
     expect(component).toMatchInlineSnapshot(`
       <EuiCallOut
         color="warning"
@@ -32,7 +33,7 @@ describe('NoFieldCallout', () => {
   });
   it('renders correctly with passed defaultNoFieldsMessage', () => {
     const component = shallow(
-      <NoFieldsCallout existFieldsInIndex={true} defaultNoFieldsMessage="No empty fields" />
+      <NoFieldsCallout fieldsExistInIndex={true} defaultNoFieldsMessage="No empty fields" />
     );
     expect(component).toMatchInlineSnapshot(`
       <EuiCallOut
@@ -45,7 +46,7 @@ describe('NoFieldCallout', () => {
 
   it('renders properly when affected by field filter', () => {
     const component = shallow(
-      <NoFieldsCallout existFieldsInIndex={true} isAffectedByFieldFilter={true} />
+      <NoFieldsCallout fieldsExistInIndex={true} isAffectedByFieldFilter={true} />
     );
     expect(component).toMatchInlineSnapshot(`
       <EuiCallOut
@@ -68,7 +69,7 @@ describe('NoFieldCallout', () => {
   it('renders correctly when affected by global filters and timerange', () => {
     const component = shallow(
       <NoFieldsCallout
-        existFieldsInIndex={true}
+        fieldsExistInIndex={true}
         isAffectedByTimerange={true}
         isAffectedByGlobalFilter={true}
         defaultNoFieldsMessage="There are no available fields that contain data."
@@ -98,7 +99,7 @@ describe('NoFieldCallout', () => {
   it('renders correctly when affected by global filters and field filters', () => {
     const component = shallow(
       <NoFieldsCallout
-        existFieldsInIndex={true}
+        fieldsExistInIndex={true}
         isAffectedByTimerange={true}
         isAffectedByFieldFilter={true}
         defaultNoFieldsMessage="There are no available fields that contain data."
@@ -128,7 +129,7 @@ describe('NoFieldCallout', () => {
   it('renders correctly when affected by field filters, global filter and timerange', () => {
     const component = shallow(
       <NoFieldsCallout
-        existFieldsInIndex={true}
+        fieldsExistInIndex={true}
         isAffectedByFieldFilter={true}
         isAffectedByTimerange={true}
         isAffectedByGlobalFilter={true}

src/plugins/unified_field_list/public/components/field_list/no_fields_callout.tsx

@@ -1,37 +1,41 @@
 /*
  * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
  * or more contributor license agreements. Licensed under the Elastic License
- * 2.0; you may not use this file except in compliance with the Elastic License
- * 2.0.
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
  */
 
 import React from 'react';
 import { EuiCallOut } from '@elastic/eui';
 import { i18n } from '@kbn/i18n';
 
-const defaultNoFieldsMessageCopy = i18n.translate('xpack.lens.indexPatterns.noDataLabel', {
-  defaultMessage: 'There are no fields.',
-});
+const defaultNoFieldsMessageCopy = i18n.translate(
+  'unifiedFieldList.fieldList.noFieldsCallout.noDataLabel',
+  {
+    defaultMessage: 'There are no fields.',
+  }
+);
 
 export const NoFieldsCallout = ({
-  existFieldsInIndex,
+  fieldsExistInIndex,
   defaultNoFieldsMessage = defaultNoFieldsMessageCopy,
   isAffectedByFieldFilter = false,
   isAffectedByTimerange = false,
   isAffectedByGlobalFilter = false,
 }: {
-  existFieldsInIndex: boolean;
+  fieldsExistInIndex: boolean;
   isAffectedByFieldFilter?: boolean;
   defaultNoFieldsMessage?: string;
   isAffectedByTimerange?: boolean;
   isAffectedByGlobalFilter?: boolean;
 }) => {
-  if (!existFieldsInIndex) {
+  if (!fieldsExistInIndex) {
     return (
       <EuiCallOut
         size="s"
         color="warning"
-        title={i18n.translate('xpack.lens.indexPatterns.noFieldsLabel', {
+        title={i18n.translate('unifiedFieldList.fieldList.noFieldsCallout.noFieldsLabel', {
           defaultMessage: 'No fields exist in this data view.',
         })}
       />
@@ -44,7 +48,7 @@ export const NoFieldsCallout = ({
       color="warning"
       title={
         isAffectedByFieldFilter
-          ? i18n.translate('xpack.lens.indexPatterns.noFilteredFieldsLabel', {
+          ? i18n.translate('unifiedFieldList.fieldList.noFieldsCallout.noFilteredFieldsLabel', {
               defaultMessage: 'No fields match the selected filters.',
             })
           : defaultNoFieldsMessage
@@ -53,30 +57,39 @@ export const NoFieldsCallout = ({
       {(isAffectedByTimerange || isAffectedByFieldFilter || isAffectedByGlobalFilter) && (
         <>
           <strong>
-            {i18n.translate('xpack.lens.indexPatterns.noFields.tryText', {
+            {i18n.translate('unifiedFieldList.fieldList.noFieldsCallout.noFields.tryText', {
               defaultMessage: 'Try:',
             })}
           </strong>
           <ul>
             {isAffectedByTimerange && (
               <li>
-                {i18n.translate('xpack.lens.indexPatterns.noFields.extendTimeBullet', {
-                  defaultMessage: 'Extending the time range',
-                })}
+                {i18n.translate(
+                  'unifiedFieldList.fieldList.noFieldsCallout.noFields.extendTimeBullet',
+                  {
+                    defaultMessage: 'Extending the time range',
+                  }
+                )}
               </li>
             )}
             {isAffectedByFieldFilter && (
               <li>
-                {i18n.translate('xpack.lens.indexPatterns.noFields.fieldTypeFilterBullet', {
-                  defaultMessage: 'Using different field filters',
-                })}
+                {i18n.translate(
+                  'unifiedFieldList.fieldList.noFieldsCallout.noFields.fieldTypeFilterBullet',
+                  {
+                    defaultMessage: 'Using different field filters',
+                  }
+                )}
               </li>
             )}
             {isAffectedByGlobalFilter && (
               <li>
-                {i18n.translate('xpack.lens.indexPatterns.noFields.globalFiltersBullet', {
-                  defaultMessage: 'Changing the global filters',
-                })}
+                {i18n.translate(
+                  'unifiedFieldList.fieldList.noFieldsCallout.noFields.globalFiltersBullet',
+                  {
+                    defaultMessage: 'Changing the global filters',
+                  }
+                )}
               </li>
             )}
           </ul>

src/plugins/unified_field_list/public/components/field_stats/field_stats.tsx

@@ -8,8 +8,8 @@
 
 import React, { useCallback, useEffect, useMemo, useRef, useState } from 'react';
 import {
-  DataView,
-  DataViewField,
+  type DataView,
+  type DataViewField,
   ES_FIELD_TYPES,
   getEsQueryConfig,
   KBN_FIELD_TYPES,

src/plugins/unified_field_list/public/hooks/use_existing_fields.test.tsx

@@ -0,0 +1,536 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { renderHook } from '@testing-library/react-hooks';
+import { stubLogstashDataView as dataView } from '@kbn/data-views-plugin/common/data_view.stub';
+import { createStubDataView, stubFieldSpecMap } from '@kbn/data-plugin/public/stubs';
+import { dataViewPluginMocks } from '@kbn/data-views-plugin/public/mocks';
+import { UI_SETTINGS } from '@kbn/data-plugin/common';
+import {
+  useExistingFieldsFetcher,
+  useExistingFieldsReader,
+  resetExistingFieldsCache,
+  type ExistingFieldsFetcherParams,
+  ExistingFieldsReader,
+} from './use_existing_fields';
+import { dataPluginMock } from '@kbn/data-plugin/public/mocks';
+import { coreMock } from '@kbn/core/public/mocks';
+import * as ExistingFieldsServiceApi from '../services/field_existing/load_field_existing';
+import { ExistenceFetchStatus } from '../types';
+
+const dslQuery = { bool: { must: [], filter: [], should: [], must_not: [] } };
+const rollupAggsMock = {
+  date_histogram: {
+    '@timestamp': {
+      agg: 'date_histogram',
+      fixed_interval: '20m',
+      delay: '10m',
+      time_zone: 'UTC',
+    },
+  },
+};
+
+jest.spyOn(ExistingFieldsServiceApi, 'loadFieldExisting').mockImplementation(async () => ({
+  indexPatternTitle: 'test',
+  existingFieldNames: [],
+}));
+
+describe('UnifiedFieldList useExistingFields', () => {
+  let mockedServices: ExistingFieldsFetcherParams['services'];
+  const anotherDataView = createStubDataView({
+    spec: {
+      id: 'another-data-view',
+      title: 'logstash-0',
+      fields: stubFieldSpecMap,
+    },
+  });
+  const dataViewWithRestrictions = createStubDataView({
+    spec: {
+      id: 'another-data-view-with-restrictions',
+      title: 'logstash-1',
+      fields: stubFieldSpecMap,
+      typeMeta: {
+        aggs: rollupAggsMock,
+      },
+    },
+  });
+  jest.spyOn(dataViewWithRestrictions, 'getAggregationRestrictions');
+
+  beforeEach(() => {
+    const dataViews = dataViewPluginMocks.createStartContract();
+    const core = coreMock.createStart();
+    mockedServices = {
+      dataViews,
+      data: dataPluginMock.createStartContract(),
+      core,
+    };
+
+    core.uiSettings.get.mockImplementation((key: string) => {
+      if (key === UI_SETTINGS.META_FIELDS) {
+        return ['_id'];
+      }
+    });
+
+    dataViews.get.mockImplementation(async (id: string) => {
+      return [dataView, anotherDataView, dataViewWithRestrictions].find((dw) => dw.id === id)!;
+    });
+
+    (ExistingFieldsServiceApi.loadFieldExisting as jest.Mock).mockClear();
+    (dataViewWithRestrictions.getAggregationRestrictions as jest.Mock).mockClear();
+    resetExistingFieldsCache();
+  });
+
+  it('should work correctly based on the specified data view', async () => {
+    const dataViewId = dataView.id!;
+    (ExistingFieldsServiceApi.loadFieldExisting as jest.Mock).mockImplementation(async () => {
+      return {
+        existingFieldNames: [dataView.fields[0].name],
+      };
+    });
+
+    const hookFetcher = renderHook(useExistingFieldsFetcher, {
+      initialProps: {
+        dataViews: [dataView],
+        services: mockedServices,
+        fromDate: '2019-01-01',
+        toDate: '2020-01-01',
+        query: { query: '', language: 'lucene' },
+        filters: [],
+      },
+    });
+
+    const hookReader = renderHook(useExistingFieldsReader);
+
+    await hookFetcher.waitForNextUpdate();
+
+    expect(ExistingFieldsServiceApi.loadFieldExisting).toHaveBeenCalledWith(
+      expect.objectContaining({
+        fromDate: '2019-01-01',
+        toDate: '2020-01-01',
+        dslQuery,
+        dataView,
+        timeFieldName: dataView.timeFieldName,
+      })
+    );
+
+    // has existence info for the loaded data view => works more restrictive
+    expect(hookReader.result.current.isFieldsExistenceInfoUnavailable(dataViewId)).toBe(false);
+    expect(hookReader.result.current.hasFieldData(dataViewId, dataView.fields[0].name)).toBe(true);
+    expect(hookReader.result.current.hasFieldData(dataViewId, dataView.fields[1].name)).toBe(false);
+    expect(hookReader.result.current.getFieldsExistenceStatus(dataViewId)).toBe(
+      ExistenceFetchStatus.succeeded
+    );
+
+    // does not have existence info => works less restrictive
+    const anotherDataViewId = 'test-id';
+    expect(hookReader.result.current.isFieldsExistenceInfoUnavailable(anotherDataViewId)).toBe(
+      false
+    );
+    expect(hookReader.result.current.hasFieldData(anotherDataViewId, dataView.fields[0].name)).toBe(
+      true
+    );
+    expect(hookReader.result.current.hasFieldData(anotherDataViewId, dataView.fields[1].name)).toBe(
+      true
+    );
+    expect(hookReader.result.current.getFieldsExistenceStatus(anotherDataViewId)).toBe(
+      ExistenceFetchStatus.unknown
+    );
+  });
+
+  it('should work correctly with multiple readers', async () => {
+    const dataViewId = dataView.id!;
+    (ExistingFieldsServiceApi.loadFieldExisting as jest.Mock).mockImplementation(async () => {
+      return {
+        existingFieldNames: [dataView.fields[0].name],
+      };
+    });
+
+    const hookFetcher = renderHook(useExistingFieldsFetcher, {
+      initialProps: {
+        dataViews: [dataView],
+        services: mockedServices,
+        fromDate: '2019-01-01',
+        toDate: '2020-01-01',
+        query: { query: '', language: 'lucene' },
+        filters: [],
+      },
+    });
+
+    const hookReader1 = renderHook(useExistingFieldsReader);
+    const hookReader2 = renderHook(useExistingFieldsReader);
+
+    await hookFetcher.waitForNextUpdate();
+
+    expect(ExistingFieldsServiceApi.loadFieldExisting).toHaveBeenCalled();
+
+    const checkResults = (currentResult: ExistingFieldsReader) => {
+      expect(currentResult.isFieldsExistenceInfoUnavailable(dataViewId)).toBe(false);
+      expect(currentResult.hasFieldData(dataViewId, dataView.fields[0].name)).toBe(true);
+      expect(currentResult.hasFieldData(dataViewId, dataView.fields[1].name)).toBe(false);
+      expect(currentResult.getFieldsExistenceStatus(dataViewId)).toBe(
+        ExistenceFetchStatus.succeeded
+      );
+    };
+
+    // both readers should get the same results
+
+    checkResults(hookReader1.result.current);
+    checkResults(hookReader2.result.current);
+
+    // info should be persisted even if the fetcher was unmounted
+
+    hookFetcher.unmount();
+
+    checkResults(hookReader1.result.current);
+    checkResults(hookReader2.result.current);
+  });
+
+  it('should work correctly if load fails', async () => {
+    const dataViewId = dataView.id!;
+    (ExistingFieldsServiceApi.loadFieldExisting as jest.Mock).mockImplementation(async () => {
+      throw new Error('test');
+    });
+
+    const hookFetcher = renderHook(useExistingFieldsFetcher, {
+      initialProps: {
+        dataViews: [dataView],
+        services: mockedServices,
+        fromDate: '2019-01-01',
+        toDate: '2020-01-01',
+        query: { query: '', language: 'lucene' },
+        filters: [],
+      },
+    });
+
+    const hookReader = renderHook(useExistingFieldsReader);
+
+    await hookFetcher.waitForNextUpdate();
+
+    expect(ExistingFieldsServiceApi.loadFieldExisting).toHaveBeenCalled();
+
+    const currentResult = hookReader.result.current;
+    expect(currentResult.isFieldsExistenceInfoUnavailable(dataViewId)).toBe(true);
+    expect(currentResult.hasFieldData(dataViewId, dataView.fields[0].name)).toBe(true);
+    expect(currentResult.getFieldsExistenceStatus(dataViewId)).toBe(ExistenceFetchStatus.failed);
+  });
+
+  it('should work correctly for multiple data views', async () => {
+    (ExistingFieldsServiceApi.loadFieldExisting as jest.Mock).mockImplementation(
+      async ({ dataView: currentDataView }) => {
+        return {
+          existingFieldNames: [currentDataView.fields[0].name],
+        };
+      }
+    );
+
+    const hookFetcher = renderHook(useExistingFieldsFetcher, {
+      initialProps: {
+        dataViews: [dataView, anotherDataView, dataViewWithRestrictions],
+        services: mockedServices,
+        fromDate: '2019-01-01',
+        toDate: '2020-01-01',
+        query: { query: '', language: 'lucene' },
+        filters: [],
+      },
+    });
+
+    const hookReader = renderHook(useExistingFieldsReader);
+    await hookFetcher.waitForNextUpdate();
+
+    const currentResult = hookReader.result.current;
+
+    expect(currentResult.isFieldsExistenceInfoUnavailable(dataView.id!)).toBe(false);
+    expect(currentResult.isFieldsExistenceInfoUnavailable(anotherDataView.id!)).toBe(false);
+    expect(currentResult.isFieldsExistenceInfoUnavailable(dataViewWithRestrictions.id!)).toBe(true);
+    expect(currentResult.isFieldsExistenceInfoUnavailable('test-id')).toBe(false);
+
+    expect(currentResult.hasFieldData(dataView.id!, dataView.fields[0].name)).toBe(true);
+    expect(currentResult.hasFieldData(dataView.id!, dataView.fields[1].name)).toBe(false);
+
+    expect(currentResult.hasFieldData(anotherDataView.id!, anotherDataView.fields[0].name)).toBe(
+      true
+    );
+    expect(currentResult.hasFieldData(anotherDataView.id!, anotherDataView.fields[1].name)).toBe(
+      false
+    );
+
+    expect(
+      currentResult.hasFieldData(
+        dataViewWithRestrictions.id!,
+        dataViewWithRestrictions.fields[0].name
+      )
+    ).toBe(true);
+    expect(
+      currentResult.hasFieldData(
+        dataViewWithRestrictions.id!,
+        dataViewWithRestrictions.fields[1].name
+      )
+    ).toBe(true);
+    expect(currentResult.hasFieldData('test-id', 'test-field')).toBe(true);
+
+    expect(currentResult.getFieldsExistenceStatus(dataView.id!)).toBe(
+      ExistenceFetchStatus.succeeded
+    );
+    expect(currentResult.getFieldsExistenceStatus(anotherDataView.id!)).toBe(
+      ExistenceFetchStatus.succeeded
+    );
+    expect(currentResult.getFieldsExistenceStatus(dataViewWithRestrictions.id!)).toBe(
+      ExistenceFetchStatus.succeeded
+    );
+    expect(currentResult.getFieldsExistenceStatus('test-id')).toBe(ExistenceFetchStatus.unknown);
+
+    expect(dataViewWithRestrictions.getAggregationRestrictions).toHaveBeenCalledTimes(1);
+    expect(ExistingFieldsServiceApi.loadFieldExisting).toHaveBeenCalledTimes(2);
+  });
+
+  it('should work correctly for data views with restrictions', async () => {
+    const dataViewId = dataViewWithRestrictions.id!;
+    (ExistingFieldsServiceApi.loadFieldExisting as jest.Mock).mockImplementation(async () => {
+      throw new Error('test');
+    });
+
+    const hookFetcher = renderHook(useExistingFieldsFetcher, {
+      initialProps: {
+        dataViews: [dataViewWithRestrictions],
+        services: mockedServices,
+        fromDate: '2019-01-01',
+        toDate: '2020-01-01',
+        query: { query: '', language: 'lucene' },
+        filters: [],
+      },
+    });
+
+    const hookReader = renderHook(useExistingFieldsReader);
+    await hookFetcher.waitForNextUpdate();
+    await hookFetcher.waitFor(() => !hookFetcher.result.current.isProcessing);
+
+    expect(dataViewWithRestrictions.getAggregationRestrictions).toHaveBeenCalled();
+    expect(ExistingFieldsServiceApi.loadFieldExisting).not.toHaveBeenCalled();
+
+    const currentResult = hookReader.result.current;
+    expect(currentResult.isFieldsExistenceInfoUnavailable(dataViewId)).toBe(true);
+    expect(currentResult.hasFieldData(dataViewId, dataViewWithRestrictions.fields[0].name)).toBe(
+      true
+    );
+    expect(currentResult.getFieldsExistenceStatus(dataViewId)).toBe(ExistenceFetchStatus.succeeded);
+  });
+
+  it('should work correctly for when data views are changed', async () => {
+    (ExistingFieldsServiceApi.loadFieldExisting as jest.Mock).mockImplementation(
+      async ({ dataView: currentDataView }) => {
+        return {
+          existingFieldNames: [currentDataView.fields[0].name],
+        };
+      }
+    );
+
+    const params: ExistingFieldsFetcherParams = {
+      dataViews: [dataView],
+      services: mockedServices,
+      fromDate: '2019-01-01',
+      toDate: '2020-01-01',
+      query: { query: '', language: 'lucene' },
+      filters: [],
+    };
+    const hookFetcher = renderHook(useExistingFieldsFetcher, {
+      initialProps: params,
+    });
+
+    const hookReader = renderHook(useExistingFieldsReader);
+    await hookFetcher.waitForNextUpdate();
+
+    expect(ExistingFieldsServiceApi.loadFieldExisting).toHaveBeenCalledWith(
+      expect.objectContaining({
+        fromDate: '2019-01-01',
+        toDate: '2020-01-01',
+        dslQuery,
+        dataView,
+        timeFieldName: dataView.timeFieldName,
+      })
+    );
+
+    expect(hookReader.result.current.getFieldsExistenceStatus(dataView.id!)).toBe(
+      ExistenceFetchStatus.succeeded
+    );
+    expect(hookReader.result.current.getFieldsExistenceStatus(anotherDataView.id!)).toBe(
+      ExistenceFetchStatus.unknown
+    );
+
+    hookFetcher.rerender({
+      ...params,
+      dataViews: [dataView, anotherDataView],
+    });
+
+    await hookFetcher.waitForNextUpdate();
+
+    expect(ExistingFieldsServiceApi.loadFieldExisting).toHaveBeenNthCalledWith(
+      2,
+      expect.objectContaining({
+        fromDate: '2019-01-01',
+        toDate: '2020-01-01',
+        dslQuery,
+        dataView,
+        timeFieldName: dataView.timeFieldName,
+      })
+    );
+
+    expect(ExistingFieldsServiceApi.loadFieldExisting).toHaveBeenNthCalledWith(
+      3,
+      expect.objectContaining({
+        fromDate: '2019-01-01',
+        toDate: '2020-01-01',
+        dslQuery,
+        dataView: anotherDataView,
+        timeFieldName: anotherDataView.timeFieldName,
+      })
+    );
+
+    expect(hookReader.result.current.getFieldsExistenceStatus(dataView.id!)).toBe(
+      ExistenceFetchStatus.succeeded
+    );
+    expect(hookReader.result.current.getFieldsExistenceStatus(anotherDataView.id!)).toBe(
+      ExistenceFetchStatus.succeeded
+    );
+  });
+
+  it('should work correctly for when params are changed', async () => {
+    (ExistingFieldsServiceApi.loadFieldExisting as jest.Mock).mockImplementation(
+      async ({ dataView: currentDataView }) => {
+        return {
+          existingFieldNames: [currentDataView.fields[0].name],
+        };
+      }
+    );
+
+    const params: ExistingFieldsFetcherParams = {
+      dataViews: [dataView],
+      services: mockedServices,
+      fromDate: '2019-01-01',
+      toDate: '2020-01-01',
+      query: { query: '', language: 'lucene' },
+      filters: [],
+    };
+    const hookFetcher = renderHook(useExistingFieldsFetcher, {
+      initialProps: params,
+    });
+
+    const hookReader = renderHook(useExistingFieldsReader);
+    await hookFetcher.waitForNextUpdate();
+
+    expect(ExistingFieldsServiceApi.loadFieldExisting).toHaveBeenCalledWith(
+      expect.objectContaining({
+        fromDate: '2019-01-01',
+        toDate: '2020-01-01',
+        dslQuery,
+        dataView,
+        timeFieldName: dataView.timeFieldName,
+      })
+    );
+
+    expect(hookReader.result.current.getFieldsExistenceStatus(dataView.id!)).toBe(
+      ExistenceFetchStatus.succeeded
+    );
+
+    hookFetcher.rerender({
+      ...params,
+      fromDate: '2021-01-01',
+      toDate: '2022-01-01',
+      query: { query: 'test', language: 'kuery' },
+    });
+
+    await hookFetcher.waitForNextUpdate();
+
+    expect(ExistingFieldsServiceApi.loadFieldExisting).toHaveBeenNthCalledWith(
+      2,
+      expect.objectContaining({
+        fromDate: '2021-01-01',
+        toDate: '2022-01-01',
+        dslQuery: {
+          bool: {
+            filter: [
+              {
+                multi_match: {
+                  lenient: true,
+                  query: 'test',
+                  type: 'best_fields',
+                },
+              },
+            ],
+            must: [],
+            must_not: [],
+            should: [],
+          },
+        },
+        dataView,
+        timeFieldName: dataView.timeFieldName,
+      })
+    );
+  });
+
+  it('should call onNoData callback only once', async () => {
+    (ExistingFieldsServiceApi.loadFieldExisting as jest.Mock).mockImplementation(async () => {
+      return {
+        existingFieldNames: ['_id'],
+      };
+    });
+
+    const params: ExistingFieldsFetcherParams = {
+      dataViews: [dataView],
+      services: mockedServices,
+      fromDate: '2019-01-01',
+      toDate: '2020-01-01',
+      query: { query: '', language: 'lucene' },
+      filters: [],
+      onNoData: jest.fn(),
+    };
+    const hookFetcher = renderHook(useExistingFieldsFetcher, {
+      initialProps: params,
+    });
+
+    const hookReader = renderHook(useExistingFieldsReader);
+    await hookFetcher.waitForNextUpdate();
+
+    expect(ExistingFieldsServiceApi.loadFieldExisting).toHaveBeenCalledWith(
+      expect.objectContaining({
+        fromDate: '2019-01-01',
+        toDate: '2020-01-01',
+        dslQuery,
+        dataView,
+        timeFieldName: dataView.timeFieldName,
+      })
+    );
+
+    expect(hookReader.result.current.getFieldsExistenceStatus(dataView.id!)).toBe(
+      ExistenceFetchStatus.succeeded
+    );
+
+    expect(params.onNoData).toHaveBeenCalledWith(dataView.id);
+    expect(params.onNoData).toHaveBeenCalledTimes(1);
+
+    hookFetcher.rerender({
+      ...params,
+      fromDate: '2021-01-01',
+      toDate: '2022-01-01',
+    });
+
+    await hookFetcher.waitForNextUpdate();
+
+    expect(ExistingFieldsServiceApi.loadFieldExisting).toHaveBeenNthCalledWith(
+      2,
+      expect.objectContaining({
+        fromDate: '2021-01-01',
+        toDate: '2022-01-01',
+        dslQuery,
+        dataView,
+        timeFieldName: dataView.timeFieldName,
+      })
+    );
+
+    expect(params.onNoData).toHaveBeenCalledTimes(1); // still 1 time
+  });
+});

src/plugins/unified_field_list/public/hooks/use_existing_fields.ts

@@ -0,0 +1,347 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { useCallback, useEffect, useMemo, useRef, useState } from 'react';
+import { htmlIdGenerator } from '@elastic/eui';
+import { BehaviorSubject } from 'rxjs';
+import { CoreStart } from '@kbn/core/public';
+import type { AggregateQuery, EsQueryConfig, Filter, Query } from '@kbn/es-query';
+import {
+  DataPublicPluginStart,
+  DataViewsContract,
+  getEsQueryConfig,
+  UI_SETTINGS,
+} from '@kbn/data-plugin/public';
+import { type DataView } from '@kbn/data-plugin/common';
+import { loadFieldExisting } from '../services/field_existing';
+import { ExistenceFetchStatus } from '../types';
+
+const getBuildEsQueryAsync = async () => (await import('@kbn/es-query')).buildEsQuery;
+const generateId = htmlIdGenerator();
+
+export interface ExistingFieldsInfo {
+  fetchStatus: ExistenceFetchStatus;
+  existingFieldsByFieldNameMap: Record<string, boolean>;
+  numberOfFetches: number;
+  hasDataViewRestrictions?: boolean;
+}
+
+export interface ExistingFieldsFetcherParams {
+  dataViews: DataView[];
+  fromDate: string;
+  toDate: string;
+  query: Query | AggregateQuery;
+  filters: Filter[];
+  services: {
+    core: Pick<CoreStart, 'uiSettings'>;
+    data: DataPublicPluginStart;
+    dataViews: DataViewsContract;
+  };
+  onNoData?: (dataViewId: string) => unknown;
+}
+
+type ExistingFieldsByDataViewMap = Record<string, ExistingFieldsInfo>;
+
+export interface ExistingFieldsFetcher {
+  refetchFieldsExistenceInfo: (dataViewId?: string) => Promise<void>;
+  isProcessing: boolean;
+}
+
+export interface ExistingFieldsReader {
+  hasFieldData: (dataViewId: string, fieldName: string) => boolean;
+  getFieldsExistenceStatus: (dataViewId: string) => ExistenceFetchStatus;
+  isFieldsExistenceInfoUnavailable: (dataViewId: string) => boolean;
+}
+
+const initialData: ExistingFieldsByDataViewMap = {};
+const unknownInfo: ExistingFieldsInfo = {
+  fetchStatus: ExistenceFetchStatus.unknown,
+  existingFieldsByFieldNameMap: {},
+  numberOfFetches: 0,
+};
+
+const globalMap$ = new BehaviorSubject<ExistingFieldsByDataViewMap>(initialData); // for syncing between hooks
+let lastFetchId: string = ''; // persist last fetch id to skip older requests/responses if any
+
+export const useExistingFieldsFetcher = (
+  params: ExistingFieldsFetcherParams
+): ExistingFieldsFetcher => {
+  const mountedRef = useRef<boolean>(true);
+  const [activeRequests, setActiveRequests] = useState<number>(0);
+  const isProcessing = activeRequests > 0;
+
+  const fetchFieldsExistenceInfo = useCallback(
+    async ({
+      dataViewId,
+      query,
+      filters,
+      fromDate,
+      toDate,
+      services: { dataViews, data, core },
+      onNoData,
+      fetchId,
+    }: ExistingFieldsFetcherParams & {
+      dataViewId: string | undefined;
+      fetchId: string;
+    }): Promise<void> => {
+      if (!dataViewId) {
+        return;
+      }
+
+      const currentInfo = globalMap$.getValue()?.[dataViewId];
+
+      if (!mountedRef.current) {
+        return;
+      }
+
+      const numberOfFetches = (currentInfo?.numberOfFetches ?? 0) + 1;
+      const dataView = await dataViews.get(dataViewId);
+
+      if (!dataView?.title) {
+        return;
+      }
+
+      setActiveRequests((value) => value + 1);
+
+      const hasRestrictions = Boolean(dataView.getAggregationRestrictions?.());
+      const info: ExistingFieldsInfo = {
+        ...unknownInfo,
+        numberOfFetches,
+      };
+
+      if (hasRestrictions) {
+        info.fetchStatus = ExistenceFetchStatus.succeeded;
+        info.hasDataViewRestrictions = true;
+      } else {
+        try {
+          const result = await loadFieldExisting({
+            dslQuery: await buildSafeEsQuery(
+              dataView,
+              query,
+              filters,
+              getEsQueryConfig(core.uiSettings)
+            ),
+            fromDate,
+            toDate,
+            timeFieldName: dataView.timeFieldName,
+            data,
+            uiSettingsClient: core.uiSettings,
+            dataViewsService: dataViews,
+            dataView,
+          });
+
+          const existingFieldNames = result?.existingFieldNames || [];
+
+          const metaFields = core.uiSettings.get(UI_SETTINGS.META_FIELDS) || [];
+          if (
+            !existingFieldNames.filter((fieldName) => !metaFields.includes?.(fieldName)).length &&
+            numberOfFetches === 1 &&
+            onNoData
+          ) {
+            onNoData(dataViewId);
+          }
+
+          info.existingFieldsByFieldNameMap = booleanMap(existingFieldNames);
+          info.fetchStatus = ExistenceFetchStatus.succeeded;
+        } catch (error) {
+          // eslint-disable-next-line no-console
+          console.error(error);
+          info.fetchStatus = ExistenceFetchStatus.failed;
+        }
+      }
+
+      // skip redundant and older results
+      if (mountedRef.current && fetchId === lastFetchId) {
+        globalMap$.next({
+          ...globalMap$.getValue(),
+          [dataViewId]: info,
+        });
+      }
+
+      setActiveRequests((value) => value - 1);
+    },
+    [mountedRef, setActiveRequests]
+  );
+
+  const dataViewsHash = getDataViewsHash(params.dataViews);
+  const refetchFieldsExistenceInfo = useCallback(
+    async (dataViewId?: string) => {
+      const fetchId = generateId();
+      lastFetchId = fetchId;
+      // refetch only for the specified data view
+      if (dataViewId) {
+        await fetchFieldsExistenceInfo({
+          fetchId,
+          dataViewId,
+          ...params,
+        });
+        return;
+      }
+      // refetch for all mentioned data views
+      await Promise.all(
+        params.dataViews.map((dataView) =>
+          fetchFieldsExistenceInfo({
+            fetchId,
+            dataViewId: dataView.id,
+            ...params,
+          })
+        )
+      );
+    },
+    // eslint-disable-next-line react-hooks/exhaustive-deps
+    [
+      fetchFieldsExistenceInfo,
+      dataViewsHash,
+      params.query,
+      params.filters,
+      params.fromDate,
+      params.toDate,
+    ]
+  );
+
+  useEffect(() => {
+    refetchFieldsExistenceInfo();
+  }, [refetchFieldsExistenceInfo]);
+
+  useEffect(() => {
+    return () => {
+      mountedRef.current = false;
+      globalMap$.next({}); // reset the cache (readers will continue using their own data slice until they are unmounted too)
+    };
+  }, [mountedRef]);
+
+  return useMemo(
+    () => ({
+      refetchFieldsExistenceInfo,
+      isProcessing,
+    }),
+    [refetchFieldsExistenceInfo, isProcessing]
+  );
+};
+
+export const useExistingFieldsReader: () => ExistingFieldsReader = () => {
+  const mountedRef = useRef<boolean>(true);
+  const [existingFieldsByDataViewMap, setExistingFieldsByDataViewMap] =
+    useState<ExistingFieldsByDataViewMap>(globalMap$.getValue());
+
+  useEffect(() => {
+    const subscription = globalMap$.subscribe((data) => {
+      if (mountedRef.current && Object.keys(data).length > 0) {
+        setExistingFieldsByDataViewMap((savedData) => ({
+          ...savedData,
+          ...data,
+        }));
+      }
+    });
+
+    return () => {
+      subscription.unsubscribe();
+    };
+  }, [setExistingFieldsByDataViewMap, mountedRef]);
+
+  const hasFieldData = useCallback(
+    (dataViewId: string, fieldName: string) => {
+      const info = existingFieldsByDataViewMap[dataViewId];
+
+      if (info?.fetchStatus === ExistenceFetchStatus.succeeded) {
+        return (
+          info?.hasDataViewRestrictions || Boolean(info?.existingFieldsByFieldNameMap[fieldName])
+        );
+      }
+
+      return true;
+    },
+    [existingFieldsByDataViewMap]
+  );
+
+  const getFieldsExistenceInfo = useCallback(
+    (dataViewId: string) => {
+      return dataViewId ? existingFieldsByDataViewMap[dataViewId] : unknownInfo;
+    },
+    [existingFieldsByDataViewMap]
+  );
+
+  const getFieldsExistenceStatus = useCallback(
+    (dataViewId: string): ExistenceFetchStatus => {
+      return getFieldsExistenceInfo(dataViewId)?.fetchStatus || ExistenceFetchStatus.unknown;
+    },
+    [getFieldsExistenceInfo]
+  );
+
+  const isFieldsExistenceInfoUnavailable = useCallback(
+    (dataViewId: string): boolean => {
+      const info = getFieldsExistenceInfo(dataViewId);
+      return Boolean(
+        info?.fetchStatus === ExistenceFetchStatus.failed || info?.hasDataViewRestrictions
+      );
+    },
+    [getFieldsExistenceInfo]
+  );
+
+  useEffect(() => {
+    return () => {
+      mountedRef.current = false;
+    };
+  }, [mountedRef]);
+
+  return useMemo(
+    () => ({
+      hasFieldData,
+      getFieldsExistenceStatus,
+      isFieldsExistenceInfoUnavailable,
+    }),
+    [hasFieldData, getFieldsExistenceStatus, isFieldsExistenceInfoUnavailable]
+  );
+};
+
+export const resetExistingFieldsCache = () => {
+  globalMap$.next(initialData);
+};
+
+function getDataViewsHash(dataViews: DataView[]): string {
+  return (
+    dataViews
+      // From Lens it's coming as IndexPattern type and not the real DataView type
+      .map(
+        (dataView) =>
+          `${dataView.id}:${dataView.title}:${dataView.timeFieldName || 'no-timefield'}:${
+            dataView.fields?.length ?? 0 // adding a field will also trigger a refetch of fields existence data
+          }`
+      )
+      .join(',')
+  );
+}
+
+// Wrapper around buildEsQuery, handling errors (e.g. because a query can't be parsed) by
+// returning a query dsl object not matching anything
+async function buildSafeEsQuery(
+  dataView: DataView,
+  query: Query | AggregateQuery,
+  filters: Filter[],
+  queryConfig: EsQueryConfig
+) {
+  const buildEsQuery = await getBuildEsQueryAsync();
+  try {
+    return buildEsQuery(dataView, query, filters, queryConfig);
+  } catch (e) {
+    return {
+      bool: {
+        must_not: {
+          match_all: {},
+        },
+      },
+    };
+  }
+}
+
+function booleanMap(keys: string[]) {
+  return keys.reduce((acc, key) => {
+    acc[key] = true;
+    return acc;
+  }, {} as Record<string, boolean>);
+}

src/plugins/unified_field_list/public/hooks/use_grouped_fields.test.tsx

@@ -0,0 +1,272 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { renderHook } from '@testing-library/react-hooks';
+import {
+  stubDataViewWithoutTimeField,
+  stubLogstashDataView as dataView,
+} from '@kbn/data-views-plugin/common/data_view.stub';
+import { createStubDataView, stubFieldSpecMap } from '@kbn/data-plugin/public/stubs';
+import { DataViewField } from '@kbn/data-views-plugin/common';
+import { dataViewPluginMocks } from '@kbn/data-views-plugin/public/mocks';
+import { type GroupedFieldsParams, useGroupedFields } from './use_grouped_fields';
+import { ExistenceFetchStatus, FieldListGroups, FieldsGroupNames } from '../types';
+
+describe('UnifiedFieldList useGroupedFields()', () => {
+  let mockedServices: GroupedFieldsParams<DataViewField>['services'];
+  const allFields = dataView.fields;
+  const anotherDataView = createStubDataView({
+    spec: {
+      id: 'another-data-view',
+      title: 'logstash-0',
+      fields: stubFieldSpecMap,
+    },
+  });
+
+  beforeEach(() => {
+    const dataViews = dataViewPluginMocks.createStartContract();
+    mockedServices = {
+      dataViews,
+    };
+
+    dataViews.get.mockImplementation(async (id: string) => {
+      return [dataView, stubDataViewWithoutTimeField].find((dw) => dw.id === id)!;
+    });
+  });
+
+  it('should work correctly for no data', async () => {
+    const { result, waitForNextUpdate } = renderHook(() =>
+      useGroupedFields({
+        dataViewId: dataView.id!,
+        allFields: [],
+        services: mockedServices,
+      })
+    );
+
+    await waitForNextUpdate();
+
+    const fieldGroups = result.current.fieldGroups;
+
+    expect(
+      Object.keys(fieldGroups!).map(
+        (key) => `${key}-${fieldGroups![key as FieldsGroupNames]?.fields.length}`
+      )
+    ).toStrictEqual([
+      'SpecialFields-0',
+      'SelectedFields-0',
+      'AvailableFields-0',
+      'EmptyFields-0',
+      'MetaFields-0',
+    ]);
+  });
+
+  it('should work correctly with fields', async () => {
+    const { result, waitForNextUpdate } = renderHook(() =>
+      useGroupedFields({
+        dataViewId: dataView.id!,
+        allFields,
+        services: mockedServices,
+      })
+    );
+
+    await waitForNextUpdate();
+
+    const fieldGroups = result.current.fieldGroups;
+
+    expect(
+      Object.keys(fieldGroups!).map(
+        (key) => `${key}-${fieldGroups![key as FieldsGroupNames]?.fields.length}`
+      )
+    ).toStrictEqual([
+      'SpecialFields-0',
+      'SelectedFields-0',
+      'AvailableFields-25',
+      'EmptyFields-0',
+      'MetaFields-3',
+    ]);
+  });
+
+  it('should work correctly when filtered', async () => {
+    const { result, waitForNextUpdate } = renderHook(() =>
+      useGroupedFields({
+        dataViewId: dataView.id!,
+        allFields,
+        services: mockedServices,
+        onFilterField: (field: DataViewField) => field.name.startsWith('@'),
+      })
+    );
+
+    await waitForNextUpdate();
+
+    const fieldGroups = result.current.fieldGroups;
+
+    expect(
+      Object.keys(fieldGroups!).map(
+        (key) => `${key}-${fieldGroups![key as FieldsGroupNames]?.fields.length}`
+      )
+    ).toStrictEqual([
+      'SpecialFields-0',
+      'SelectedFields-0',
+      'AvailableFields-2',
+      'EmptyFields-0',
+      'MetaFields-0',
+    ]);
+  });
+
+  it('should work correctly when custom unsupported fields are skipped', async () => {
+    const { result, waitForNextUpdate } = renderHook(() =>
+      useGroupedFields({
+        dataViewId: dataView.id!,
+        allFields,
+        services: mockedServices,
+        onSupportedFieldFilter: (field: DataViewField) => field.aggregatable,
+      })
+    );
+
+    await waitForNextUpdate();
+
+    const fieldGroups = result.current.fieldGroups;
+
+    expect(
+      Object.keys(fieldGroups!).map(
+        (key) => `${key}-${fieldGroups![key as FieldsGroupNames]?.fields.length}`
+      )
+    ).toStrictEqual([
+      'SpecialFields-0',
+      'SelectedFields-0',
+      'AvailableFields-23',
+      'EmptyFields-0',
+      'MetaFields-3',
+    ]);
+  });
+
+  it('should work correctly when selected fields are present', async () => {
+    const { result, waitForNextUpdate } = renderHook(() =>
+      useGroupedFields({
+        dataViewId: dataView.id!,
+        allFields,
+        services: mockedServices,
+        onSelectedFieldFilter: (field: DataViewField) =>
+          ['bytes', 'extension', '_id', '@timestamp'].includes(field.name),
+      })
+    );
+
+    await waitForNextUpdate();
+
+    const fieldGroups = result.current.fieldGroups;
+
+    expect(
+      Object.keys(fieldGroups!).map(
+        (key) => `${key}-${fieldGroups![key as FieldsGroupNames]?.fields.length}`
+      )
+    ).toStrictEqual([
+      'SpecialFields-0',
+      'SelectedFields-4',
+      'AvailableFields-25',
+      'EmptyFields-0',
+      'MetaFields-3',
+    ]);
+  });
+
+  it('should work correctly for text-based queries (no data view)', async () => {
+    const { result } = renderHook(() =>
+      useGroupedFields({
+        dataViewId: null,
+        allFields,
+        services: mockedServices,
+      })
+    );
+
+    const fieldGroups = result.current.fieldGroups;
+
+    expect(
+      Object.keys(fieldGroups!).map(
+        (key) => `${key}-${fieldGroups![key as FieldsGroupNames]?.fields.length}`
+      )
+    ).toStrictEqual(['SpecialFields-0', 'SelectedFields-0', 'AvailableFields-28', 'MetaFields-0']);
+  });
+
+  it('should work correctly when details are overwritten', async () => {
+    const { result, waitForNextUpdate } = renderHook(() =>
+      useGroupedFields({
+        dataViewId: dataView.id!,
+        allFields,
+        services: mockedServices,
+        onOverrideFieldGroupDetails: (groupName) => {
+          if (groupName === FieldsGroupNames.SelectedFields) {
+            return {
+              helpText: 'test',
+            };
+          }
+        },
+      })
+    );
+
+    await waitForNextUpdate();
+
+    const fieldGroups = result.current.fieldGroups;
+
+    expect(fieldGroups[FieldsGroupNames.SelectedFields]?.helpText).toBe('test');
+    expect(fieldGroups[FieldsGroupNames.AvailableFields]?.helpText).not.toBe('test');
+  });
+
+  it('should work correctly when changing a data view and existence info is available only for one of them', async () => {
+    const knownDataViewId = dataView.id!;
+    let fieldGroups: FieldListGroups<DataViewField>;
+    const props: GroupedFieldsParams<DataViewField> = {
+      dataViewId: dataView.id!,
+      allFields,
+      services: mockedServices,
+      fieldsExistenceReader: {
+        hasFieldData: (dataViewId, fieldName) => {
+          return dataViewId === knownDataViewId && ['bytes', 'extension'].includes(fieldName);
+        },
+        getFieldsExistenceStatus: (dataViewId) =>
+          dataViewId === knownDataViewId
+            ? ExistenceFetchStatus.succeeded
+            : ExistenceFetchStatus.unknown,
+        isFieldsExistenceInfoUnavailable: (dataViewId) => dataViewId !== knownDataViewId,
+      },
+    };
+
+    const { result, waitForNextUpdate, rerender } = renderHook(useGroupedFields, {
+      initialProps: props,
+    });
+    await waitForNextUpdate();
+
+    fieldGroups = result.current.fieldGroups;
+
+    expect(
+      Object.keys(fieldGroups!).map(
+        (key) => `${key}-${fieldGroups![key as FieldsGroupNames]?.fields.length}`
+      )
+    ).toStrictEqual([
+      'SpecialFields-0',
+      'SelectedFields-0',
+      'AvailableFields-2',
+      'EmptyFields-23',
+      'MetaFields-3',
+    ]);
+
+    rerender({
+      ...props,
+      dataViewId: anotherDataView.id!,
+      allFields: anotherDataView.fields,
+    });
+
+    await waitForNextUpdate();
+
+    fieldGroups = result.current.fieldGroups;
+
+    expect(
+      Object.keys(fieldGroups!).map(
+        (key) => `${key}-${fieldGroups![key as FieldsGroupNames]?.fields.length}`
+      )
+    ).toStrictEqual(['SpecialFields-0', 'SelectedFields-0', 'AvailableFields-8', 'MetaFields-0']);
+  });
+});

src/plugins/unified_field_list/public/hooks/use_grouped_fields.ts

@@ -0,0 +1,267 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { groupBy } from 'lodash';
+import { useEffect, useMemo, useState } from 'react';
+import { i18n } from '@kbn/i18n';
+import { type DataView, type DataViewField } from '@kbn/data-views-plugin/common';
+import { type DataViewsContract } from '@kbn/data-views-plugin/public';
+import {
+  type FieldListGroups,
+  type FieldsGroupDetails,
+  type FieldsGroup,
+  type FieldListItem,
+  FieldsGroupNames,
+} from '../types';
+import { type ExistingFieldsReader } from './use_existing_fields';
+
+export interface GroupedFieldsParams<T extends FieldListItem> {
+  dataViewId: string | null; // `null` is for text-based queries
+  allFields: T[];
+  services: {
+    dataViews: DataViewsContract;
+  };
+  fieldsExistenceReader?: ExistingFieldsReader;
+  onOverrideFieldGroupDetails?: (
+    groupName: FieldsGroupNames
+  ) => Partial<FieldsGroupDetails> | undefined | null;
+  onSupportedFieldFilter?: (field: T) => boolean;
+  onSelectedFieldFilter?: (field: T) => boolean;
+  onFilterField?: (field: T) => boolean;
+}
+
+export interface GroupedFieldsResult<T extends FieldListItem> {
+  fieldGroups: FieldListGroups<T>;
+}
+
+export function useGroupedFields<T extends FieldListItem = DataViewField>({
+  dataViewId,
+  allFields,
+  services,
+  fieldsExistenceReader,
+  onOverrideFieldGroupDetails,
+  onSupportedFieldFilter,
+  onSelectedFieldFilter,
+  onFilterField,
+}: GroupedFieldsParams<T>): GroupedFieldsResult<T> {
+  const [dataView, setDataView] = useState<DataView | null>(null);
+  const fieldsExistenceInfoUnavailable: boolean = dataViewId
+    ? fieldsExistenceReader?.isFieldsExistenceInfoUnavailable(dataViewId) ?? false
+    : true;
+  const hasFieldDataHandler =
+    dataViewId && fieldsExistenceReader
+      ? fieldsExistenceReader.hasFieldData
+      : hasFieldDataByDefault;
+
+  useEffect(() => {
+    const getDataView = async () => {
+      if (dataViewId) {
+        setDataView(await services.dataViews.get(dataViewId));
+      }
+    };
+    getDataView();
+    // if field existence information changed, reload the data view too
+  }, [dataViewId, services.dataViews, setDataView, hasFieldDataHandler]);
+
+  const unfilteredFieldGroups: FieldListGroups<T> = useMemo(() => {
+    const containsData = (field: T) => {
+      if (!dataViewId || !dataView) {
+        return true;
+      }
+      const overallField = dataView.getFieldByName?.(field.name);
+      return Boolean(overallField && hasFieldDataHandler(dataViewId, overallField.name));
+    };
+
+    const fields = allFields || [];
+    const allSupportedTypesFields = onSupportedFieldFilter
+      ? fields.filter(onSupportedFieldFilter)
+      : fields;
+    const sortedFields = [...allSupportedTypesFields].sort(sortFields);
+    const groupedFields = {
+      ...getDefaultFieldGroups(),
+      ...groupBy(sortedFields, (field) => {
+        if (field.type === 'document') {
+          return 'specialFields';
+        } else if (dataView?.metaFields?.includes(field.name)) {
+          return 'metaFields';
+        } else if (containsData(field)) {
+          return 'availableFields';
+        } else return 'emptyFields';
+      }),
+    };
+    const selectedFields = onSelectedFieldFilter ? sortedFields.filter(onSelectedFieldFilter) : [];
+
+    let fieldGroupDefinitions: FieldListGroups<T> = {
+      SpecialFields: {
+        fields: groupedFields.specialFields,
+        fieldCount: groupedFields.specialFields.length,
+        isAffectedByGlobalFilter: false,
+        isAffectedByTimeFilter: false,
+        isInitiallyOpen: false,
+        showInAccordion: false,
+        title: '',
+        hideDetails: true,
+      },
+      SelectedFields: {
+        fields: selectedFields,
+        fieldCount: selectedFields.length,
+        isInitiallyOpen: true,
+        showInAccordion: true,
+        title: i18n.translate('unifiedFieldList.useGroupedFields.selectedFieldsLabel', {
+          defaultMessage: 'Selected fields',
+        }),
+        isAffectedByGlobalFilter: false,
+        isAffectedByTimeFilter: true,
+        hideDetails: false,
+        hideIfEmpty: true,
+      },
+      AvailableFields: {
+        fields: groupedFields.availableFields,
+        fieldCount: groupedFields.availableFields.length,
+        isInitiallyOpen: true,
+        showInAccordion: true,
+        title:
+          dataViewId && fieldsExistenceInfoUnavailable
+            ? i18n.translate('unifiedFieldList.useGroupedFields.allFieldsLabel', {
+                defaultMessage: 'All fields',
+              })
+            : i18n.translate('unifiedFieldList.useGroupedFields.availableFieldsLabel', {
+                defaultMessage: 'Available fields',
+              }),
+        isAffectedByGlobalFilter: false,
+        isAffectedByTimeFilter: true,
+        // Show details on timeout but not failure
+        // hideDetails: fieldsExistenceInfoUnavailable && !existenceFetchTimeout, // TODO: is this check still necessary?
+        hideDetails: fieldsExistenceInfoUnavailable,
+        defaultNoFieldsMessage: i18n.translate(
+          'unifiedFieldList.useGroupedFields.noAvailableDataLabel',
+          {
+            defaultMessage: `There are no available fields that contain data.`,
+          }
+        ),
+      },
+      EmptyFields: {
+        fields: groupedFields.emptyFields,
+        fieldCount: groupedFields.emptyFields.length,
+        isAffectedByGlobalFilter: false,
+        isAffectedByTimeFilter: false,
+        isInitiallyOpen: false,
+        showInAccordion: true,
+        hideDetails: false,
+        hideIfEmpty: !dataViewId,
+        title: i18n.translate('unifiedFieldList.useGroupedFields.emptyFieldsLabel', {
+          defaultMessage: 'Empty fields',
+        }),
+        defaultNoFieldsMessage: i18n.translate(
+          'unifiedFieldList.useGroupedFields.noEmptyDataLabel',
+          {
+            defaultMessage: `There are no empty fields.`,
+          }
+        ),
+        helpText: i18n.translate('unifiedFieldList.useGroupedFields.emptyFieldsLabelHelp', {
+          defaultMessage: 'Empty fields did not contain any values based on your filters.',
+        }),
+      },
+      MetaFields: {
+        fields: groupedFields.metaFields,
+        fieldCount: groupedFields.metaFields.length,
+        isAffectedByGlobalFilter: false,
+        isAffectedByTimeFilter: false,
+        isInitiallyOpen: false,
+        showInAccordion: true,
+        hideDetails: false,
+        hideIfEmpty: !dataViewId,
+        title: i18n.translate('unifiedFieldList.useGroupedFields.metaFieldsLabel', {
+          defaultMessage: 'Meta fields',
+        }),
+        defaultNoFieldsMessage: i18n.translate(
+          'unifiedFieldList.useGroupedFields.noMetaDataLabel',
+          {
+            defaultMessage: `There are no meta fields.`,
+          }
+        ),
+      },
+    };
+
+    // do not show empty field accordion if there is no existence information
+    if (fieldsExistenceInfoUnavailable) {
+      delete fieldGroupDefinitions.EmptyFields;
+    }
+
+    if (onOverrideFieldGroupDetails) {
+      fieldGroupDefinitions = Object.keys(fieldGroupDefinitions).reduce<FieldListGroups<T>>(
+        (definitions, name) => {
+          const groupName = name as FieldsGroupNames;
+          const group: FieldsGroup<T> | undefined = fieldGroupDefinitions[groupName];
+          if (group) {
+            definitions[groupName] = {
+              ...group,
+              ...(onOverrideFieldGroupDetails(groupName) || {}),
+            };
+          }
+          return definitions;
+        },
+        {} as FieldListGroups<T>
+      );
+    }
+
+    return fieldGroupDefinitions;
+  }, [
+    allFields,
+    onSupportedFieldFilter,
+    onSelectedFieldFilter,
+    onOverrideFieldGroupDetails,
+    dataView,
+    dataViewId,
+    hasFieldDataHandler,
+    fieldsExistenceInfoUnavailable,
+  ]);
+
+  const fieldGroups: FieldListGroups<T> = useMemo(() => {
+    if (!onFilterField) {
+      return unfilteredFieldGroups;
+    }
+
+    return Object.fromEntries(
+      Object.entries(unfilteredFieldGroups).map(([name, group]) => [
+        name,
+        { ...group, fields: group.fields.filter(onFilterField) },
+      ])
+    ) as FieldListGroups<T>;
+  }, [unfilteredFieldGroups, onFilterField]);
+
+  return useMemo(
+    () => ({
+      fieldGroups,
+    }),
+    [fieldGroups]
+  );
+}
+
+function sortFields<T extends FieldListItem>(fieldA: T, fieldB: T) {
+  return (fieldA.displayName || fieldA.name).localeCompare(
+    fieldB.displayName || fieldB.name,
+    undefined,
+    {
+      sensitivity: 'base',
+    }
+  );
+}
+
+function hasFieldDataByDefault(): boolean {
+  return true;
+}
+
+function getDefaultFieldGroups() {
+  return {
+    specialFields: [],
+    availableFields: [],
+    emptyFields: [],
+    metaFields: [],
+  };
+}

src/plugins/unified_field_list/public/index.ts

@@ -14,6 +14,7 @@ export type {
   NumberStatsResult,
   TopValuesResult,
 } from '../common/types';
+export { FieldListGrouped, type FieldListGroupedProps } from './components/field_list';
 export type { FieldStatsProps, FieldStatsServices } from './components/field_stats';
 export { FieldStats } from './components/field_stats';
 export {
@@ -44,4 +45,23 @@ export type {
   UnifiedFieldListPluginSetup,
   UnifiedFieldListPluginStart,
   AddFieldFilterHandler,
+  FieldListGroups,
+  FieldsGroupDetails,
 } from './types';
+export { ExistenceFetchStatus, FieldsGroupNames } from './types';
+
+export {
+  useExistingFieldsFetcher,
+  useExistingFieldsReader,
+  resetExistingFieldsCache,
+  type ExistingFieldsInfo,
+  type ExistingFieldsFetcherParams,
+  type ExistingFieldsFetcher,
+  type ExistingFieldsReader,
+} from './hooks/use_existing_fields';
+
+export {
+  useGroupedFields,
+  type GroupedFieldsParams,
+  type GroupedFieldsResult,
+} from './hooks/use_grouped_fields';

src/plugins/unified_field_list/public/services/field_existing/index.ts

@@ -6,4 +6,9 @@
  * Side Public License, v 1.
  */
 
-export { loadFieldExisting } from './load_field_existing';
+import type { LoadFieldExistingHandler } from './load_field_existing';
+
+export const loadFieldExisting: LoadFieldExistingHandler = async (params) => {
+  const { loadFieldExisting: loadFieldExistingHandler } = await import('./load_field_existing');
+  return await loadFieldExistingHandler(params);
+};

src/plugins/unified_field_list/public/services/field_existing/load_field_existing.ts

@@ -24,7 +24,12 @@ interface FetchFieldExistenceParams {
   uiSettingsClient: IUiSettingsClient;
 }
 
-export async function loadFieldExisting({
+export type LoadFieldExistingHandler = (params: FetchFieldExistenceParams) => Promise<{
+  existingFieldNames: string[];
+  indexPatternTitle: string;
+}>;
+
+export const loadFieldExisting: LoadFieldExistingHandler = async ({
   data,
   dslQuery,
   fromDate,
@@ -33,7 +38,7 @@ export async function loadFieldExisting({
   dataViewsService,
   uiSettingsClient,
   dataView,
-}: FetchFieldExistenceParams) {
+}) => {
   const includeFrozen = uiSettingsClient.get(UI_SETTINGS.SEARCH_INCLUDE_FROZEN);
   const useSampling = uiSettingsClient.get(FIELD_EXISTENCE_SETTING);
   const metaFields = uiSettingsClient.get(UI_SETTINGS.META_FIELDS);
@@ -53,4 +58,4 @@ export async function loadFieldExisting({
       return response.rawResponse;
     },
   });
-}
+};

src/plugins/unified_field_list/public/types.ts

@@ -19,3 +19,44 @@ export type AddFieldFilterHandler = (
   value: unknown,
   type: '+' | '-'
 ) => void;
+
+export enum ExistenceFetchStatus {
+  failed = 'failed',
+  succeeded = 'succeeded',
+  unknown = 'unknown',
+}
+
+export interface FieldListItem {
+  name: DataViewField['name'];
+  type?: DataViewField['type'];
+  displayName?: DataViewField['displayName'];
+}
+
+export enum FieldsGroupNames {
+  SpecialFields = 'SpecialFields',
+  SelectedFields = 'SelectedFields',
+  AvailableFields = 'AvailableFields',
+  EmptyFields = 'EmptyFields',
+  MetaFields = 'MetaFields',
+}
+
+export interface FieldsGroupDetails {
+  showInAccordion: boolean;
+  isInitiallyOpen: boolean;
+  title: string;
+  helpText?: string;
+  isAffectedByGlobalFilter: boolean;
+  isAffectedByTimeFilter: boolean;
+  hideDetails?: boolean;
+  defaultNoFieldsMessage?: string;
+  hideIfEmpty?: boolean;
+}
+
+export interface FieldsGroup<T extends FieldListItem> extends FieldsGroupDetails {
+  fields: T[];
+  fieldCount: number;
+}
+
+export type FieldListGroups<T extends FieldListItem> = {
+  [key in FieldsGroupNames]?: FieldsGroup<T>;
+};

x-pack/plugins/lens/common/types.ts

@@ -20,11 +20,6 @@ export type { OriginalColumn } from './expressions/map_to_columns';
 
 export type FormatFactory = (mapping?: SerializedFieldFormat) => IFieldFormat;
 
-export interface ExistingFields {
-  indexPatternTitle: string;
-  existingFieldNames: string[];
-}
-
 export interface DateRange {
   fromDate: string;
   toDate: string;

x-pack/plugins/lens/public/data_views_service/loader.test.ts

@@ -5,22 +5,10 @@
  * 2.0.
  */
 
-import { DataViewsContract, DataViewSpec, FieldSpec } from '@kbn/data-views-plugin/public';
-import { IndexPattern, IndexPatternField } from '../types';
-import {
-  ensureIndexPattern,
-  loadIndexPatternRefs,
-  loadIndexPatterns,
-  syncExistingFields,
-} from './loader';
+import { DataViewsContract } from '@kbn/data-views-plugin/public';
+import { ensureIndexPattern, loadIndexPatternRefs, loadIndexPatterns } from './loader';
 import { sampleIndexPatterns, mockDataViewsService } from './mocks';
 import { documentField } from '../datasources/form_based/document_field';
-import { coreMock } from '@kbn/core/public/mocks';
-import { dataViewPluginMocks } from '@kbn/data-views-plugin/public/mocks';
-import type { DataView } from '@kbn/data-views-plugin/public';
-import { dataPluginMock } from '@kbn/data-plugin/public/mocks';
-import { UI_SETTINGS } from '@kbn/data-plugin/public';
-import { createHttpFetchError } from '@kbn/core-http-browser-mocks';
 
 describe('loader', () => {
   describe('loadIndexPatternRefs', () => {
@@ -266,218 +254,4 @@ describe('loader', () => {
       expect(onError).not.toHaveBeenCalled();
     });
   });
-
-  describe('syncExistingFields', () => {
-    const core = coreMock.createStart();
-    const dataViews = dataViewPluginMocks.createStartContract();
-    const data = dataPluginMock.createStartContract();
-
-    const dslQuery = {
-      bool: {
-        must: [],
-        filter: [{ match_all: {} }],
-        should: [],
-        must_not: [],
-      },
-    };
-
-    function getIndexPatternList() {
-      return [
-        {
-          id: '1',
-          title: '1',
-          fields: [{ name: 'ip1_field_1' }, { name: 'ip1_field_2' }],
-          hasRestrictions: false,
-        },
-        {
-          id: '2',
-          title: '2',
-          fields: [{ name: 'ip2_field_1' }, { name: 'ip2_field_2' }],
-          hasRestrictions: false,
-        },
-        {
-          id: '3',
-          title: '3',
-          fields: [{ name: 'ip3_field_1' }, { name: 'ip3_field_2' }],
-          hasRestrictions: false,
-        },
-      ] as unknown as IndexPattern[];
-    }
-
-    beforeEach(() => {
-      core.uiSettings.get.mockImplementation((key: string) => {
-        if (key === UI_SETTINGS.META_FIELDS) {
-          return [];
-        }
-      });
-      dataViews.get.mockImplementation((id: string) =>
-        Promise.resolve(
-          getIndexPatternList().find(
-            (indexPattern) => indexPattern.id === id
-          ) as unknown as DataView
-        )
-      );
-    });
-
-    it('should call once for each index pattern', async () => {
-      const updateIndexPatterns = jest.fn();
-      dataViews.getFieldsForIndexPattern.mockImplementation(
-        (dataView: DataViewSpec | DataView) =>
-          Promise.resolve(dataView.fields) as Promise<FieldSpec[]>
-      );
-
-      await syncExistingFields({
-        dateRange: { fromDate: '1900-01-01', toDate: '2000-01-01' },
-        indexPatternList: getIndexPatternList(),
-        updateIndexPatterns,
-        dslQuery,
-        onNoData: jest.fn(),
-        currentIndexPatternTitle: 'abc',
-        isFirstExistenceFetch: false,
-        existingFields: {},
-        core,
-        data,
-        dataViews,
-      });
-
-      expect(dataViews.get).toHaveBeenCalledTimes(3);
-      expect(dataViews.getFieldsForIndexPattern).toHaveBeenCalledTimes(3);
-      expect(updateIndexPatterns).toHaveBeenCalledTimes(1);
-
-      const [newState, options] = updateIndexPatterns.mock.calls[0];
-      expect(options).toEqual({ applyImmediately: true });
-
-      expect(newState).toEqual({
-        isFirstExistenceFetch: false,
-        existingFields: {
-          '1': { ip1_field_1: true, ip1_field_2: true },
-          '2': { ip2_field_1: true, ip2_field_2: true },
-          '3': { ip3_field_1: true, ip3_field_2: true },
-        },
-      });
-    });
-
-    it('should call onNoData callback if current index pattern returns no fields', async () => {
-      const updateIndexPatterns = jest.fn();
-      const onNoData = jest.fn();
-      dataViews.getFieldsForIndexPattern.mockImplementation(
-        async (dataView: DataViewSpec | DataView) => {
-          return (dataView.title === '1'
-            ? [{ name: `${dataView.title}_field_1` }, { name: `${dataView.title}_field_2` }]
-            : []) as unknown as Promise<FieldSpec[]>;
-        }
-      );
-
-      const args = {
-        dateRange: { fromDate: '1900-01-01', toDate: '2000-01-01' },
-        indexPatternList: getIndexPatternList(),
-        updateIndexPatterns,
-        dslQuery,
-        onNoData,
-        currentIndexPatternTitle: 'abc',
-        isFirstExistenceFetch: false,
-        existingFields: {},
-        core,
-        data,
-        dataViews,
-      };
-
-      await syncExistingFields(args);
-
-      expect(onNoData).not.toHaveBeenCalled();
-
-      await syncExistingFields({ ...args, isFirstExistenceFetch: true });
-      expect(onNoData).not.toHaveBeenCalled();
-    });
-
-    it('should set all fields to available and existence error flag if the request fails', async () => {
-      const updateIndexPatterns = jest.fn();
-      dataViews.getFieldsForIndexPattern.mockImplementation(() => {
-        return new Promise((_, reject) => {
-          reject(new Error());
-        });
-      });
-
-      const args = {
-        dateRange: { fromDate: '1900-01-01', toDate: '2000-01-01' },
-        indexPatternList: [
-          {
-            id: '1',
-            title: '1',
-            hasRestrictions: false,
-            fields: [{ name: 'field1' }, { name: 'field2' }] as IndexPatternField[],
-          },
-        ] as IndexPattern[],
-        updateIndexPatterns,
-        dslQuery,
-        onNoData: jest.fn(),
-        currentIndexPatternTitle: 'abc',
-        isFirstExistenceFetch: false,
-        existingFields: {},
-        core,
-        data,
-        dataViews,
-      };
-
-      await syncExistingFields(args);
-
-      const [newState, options] = updateIndexPatterns.mock.calls[0];
-      expect(options).toEqual({ applyImmediately: true });
-
-      expect(newState.existenceFetchFailed).toEqual(true);
-      expect(newState.existenceFetchTimeout).toEqual(false);
-      expect(newState.existingFields['1']).toEqual({
-        field1: true,
-        field2: true,
-      });
-    });
-
-    it('should set all fields to available and existence error flag if the request times out', async () => {
-      const updateIndexPatterns = jest.fn();
-      dataViews.getFieldsForIndexPattern.mockImplementation(() => {
-        return new Promise((_, reject) => {
-          const error = createHttpFetchError(
-            'timeout',
-            'error',
-            {} as Request,
-            { status: 408 } as Response
-          );
-          reject(error);
-        });
-      });
-
-      const args = {
-        dateRange: { fromDate: '1900-01-01', toDate: '2000-01-01' },
-        indexPatternList: [
-          {
-            id: '1',
-            title: '1',
-            hasRestrictions: false,
-            fields: [{ name: 'field1' }, { name: 'field2' }] as IndexPatternField[],
-          },
-        ] as IndexPattern[],
-        updateIndexPatterns,
-        dslQuery,
-        onNoData: jest.fn(),
-        currentIndexPatternTitle: 'abc',
-        isFirstExistenceFetch: false,
-        existingFields: {},
-        core,
-        data,
-        dataViews,
-      };
-
-      await syncExistingFields(args);
-
-      const [newState, options] = updateIndexPatterns.mock.calls[0];
-      expect(options).toEqual({ applyImmediately: true });
-
-      expect(newState.existenceFetchFailed).toEqual(false);
-      expect(newState.existenceFetchTimeout).toEqual(true);
-      expect(newState.existingFields['1']).toEqual({
-        field1: true,
-        field2: true,
-      });
-    });
-  });
 });

x-pack/plugins/lens/public/data_views_service/loader.ts

@@ -8,13 +8,8 @@
 import { isNestedField } from '@kbn/data-views-plugin/common';
 import type { DataViewsContract, DataView, DataViewSpec } from '@kbn/data-views-plugin/public';
 import { keyBy } from 'lodash';
-import { CoreStart } from '@kbn/core/public';
-import { DataPublicPluginStart } from '@kbn/data-plugin/public';
-import { loadFieldExisting } from '@kbn/unified-field-list-plugin/public';
 import { IndexPattern, IndexPatternField, IndexPatternMap, IndexPatternRef } from '../types';
 import { documentField } from '../datasources/form_based/document_field';
-import { DateRange } from '../../common';
-import { DataViewsState } from '../state_management';
 
 type ErrorHandler = (err: Error) => void;
 type MinimalDataViewsContract = Pick<DataViewsContract, 'get' | 'getIdsWithTitle' | 'create'>;
@@ -247,120 +242,3 @@ export async function ensureIndexPattern({
   };
   return newIndexPatterns;
 }
-
-async function refreshExistingFields({
-  dateRange,
-  indexPatternList,
-  dslQuery,
-  core,
-  data,
-  dataViews,
-}: {
-  dateRange: DateRange;
-  indexPatternList: IndexPattern[];
-  dslQuery: object;
-  core: Pick<CoreStart, 'http' | 'notifications' | 'uiSettings'>;
-  data: DataPublicPluginStart;
-  dataViews: DataViewsContract;
-}) {
-  try {
-    const emptinessInfo = await Promise.all(
-      indexPatternList.map(async (pattern) => {
-        if (pattern.hasRestrictions) {
-          return {
-            indexPatternTitle: pattern.title,
-            existingFieldNames: pattern.fields.map((field) => field.name),
-          };
-        }
-
-        const dataView = await dataViews.get(pattern.id);
-        return await loadFieldExisting({
-          dslQuery,
-          fromDate: dateRange.fromDate,
-          toDate: dateRange.toDate,
-          timeFieldName: pattern.timeFieldName,
-          data,
-          uiSettingsClient: core.uiSettings,
-          dataViewsService: dataViews,
-          dataView,
-        });
-      })
-    );
-    return { result: emptinessInfo, status: 200 };
-  } catch (e) {
-    return { result: undefined, status: e.res?.status as number };
-  }
-}
-
-type FieldsPropsFromDataViewsState = Pick<
-  DataViewsState,
-  'existingFields' | 'isFirstExistenceFetch' | 'existenceFetchTimeout' | 'existenceFetchFailed'
->;
-export async function syncExistingFields({
-  updateIndexPatterns,
-  isFirstExistenceFetch,
-  currentIndexPatternTitle,
-  onNoData,
-  existingFields,
-  ...requestOptions
-}: {
-  dateRange: DateRange;
-  indexPatternList: IndexPattern[];
-  existingFields: Record<string, Record<string, boolean>>;
-  updateIndexPatterns: (
-    newFieldState: FieldsPropsFromDataViewsState,
-    options: { applyImmediately: boolean }
-  ) => void;
-  isFirstExistenceFetch: boolean;
-  currentIndexPatternTitle: string;
-  dslQuery: object;
-  onNoData?: () => void;
-  core: Pick<CoreStart, 'http' | 'notifications' | 'uiSettings'>;
-  data: DataPublicPluginStart;
-  dataViews: DataViewsContract;
-}) {
-  const { indexPatternList } = requestOptions;
-  const newExistingFields = { ...existingFields };
-
-  const { result, status } = await refreshExistingFields(requestOptions);
-
-  if (result) {
-    if (isFirstExistenceFetch) {
-      const fieldsCurrentIndexPattern = result.find(
-        (info) => info.indexPatternTitle === currentIndexPatternTitle
-      );
-      if (fieldsCurrentIndexPattern && fieldsCurrentIndexPattern.existingFieldNames.length === 0) {
-        onNoData?.();
-      }
-    }
-
-    for (const { indexPatternTitle, existingFieldNames } of result) {
-      newExistingFields[indexPatternTitle] = booleanMap(existingFieldNames);
-    }
-  } else {
-    for (const { title, fields } of indexPatternList) {
-      newExistingFields[title] = booleanMap(fields.map((field) => field.name));
-    }
-  }
-
-  updateIndexPatterns(
-    {
-      existingFields: newExistingFields,
-      ...(result
-        ? { isFirstExistenceFetch: status !== 200 }
-        : {
-            isFirstExistenceFetch,
-            existenceFetchFailed: status !== 408,
-            existenceFetchTimeout: status === 408,
-          }),
-    },
-    { applyImmediately: true }
-  );
-}
-
-function booleanMap(keys: string[]) {
-  return keys.reduce((acc, key) => {
-    acc[key] = true;
-    return acc;
-  }, {} as Record<string, boolean>);
-}

x-pack/plugins/lens/public/data_views_service/mocks.ts

@@ -12,7 +12,7 @@ import {
   createMockedRestrictedIndexPattern,
 } from '../datasources/form_based/mocks';
 import { DataViewsState } from '../state_management';
-import { ExistingFieldsMap, IndexPattern } from '../types';
+import { IndexPattern } from '../types';
 import { getFieldByNameFactory } from './loader';
 
 /**
@@ -22,25 +22,13 @@ import { getFieldByNameFactory } from './loader';
 export const createMockDataViewsState = ({
   indexPatterns,
   indexPatternRefs,
-  isFirstExistenceFetch,
-  existingFields,
 }: Partial<DataViewsState> = {}): DataViewsState => {
   const refs =
     indexPatternRefs ??
     Object.values(indexPatterns ?? {}).map(({ id, title, name }) => ({ id, title, name }));
-  const allFields =
-    existingFields ??
-    refs.reduce((acc, { id, title }) => {
-      if (indexPatterns && id in indexPatterns) {
-        acc[title] = Object.fromEntries(indexPatterns[id].fields.map((f) => [f.displayName, true]));
-      }
-      return acc;
-    }, {} as ExistingFieldsMap);
   return {
     indexPatterns: indexPatterns ?? {},
     indexPatternRefs: refs,
-    isFirstExistenceFetch: Boolean(isFirstExistenceFetch),
-    existingFields: allFields,
   };
 };
 

x-pack/plugins/lens/public/data_views_service/service.ts

@@ -14,14 +14,8 @@ import {
   UPDATE_FILTER_REFERENCES_ACTION,
   UPDATE_FILTER_REFERENCES_TRIGGER,
 } from '@kbn/unified-search-plugin/public';
-import type { DateRange } from '../../common';
 import type { IndexPattern, IndexPatternMap, IndexPatternRef } from '../types';
-import {
-  ensureIndexPattern,
-  loadIndexPatternRefs,
-  loadIndexPatterns,
-  syncExistingFields,
-} from './loader';
+import { ensureIndexPattern, loadIndexPatternRefs, loadIndexPatterns } from './loader';
 import type { DataViewsState } from '../state_management';
 import { generateId } from '../id_generator';
 
@@ -71,18 +65,6 @@ export interface IndexPatternServiceAPI {
     id: string;
     cache: IndexPatternMap;
   }) => Promise<IndexPatternMap | undefined>;
-  /**
-   * Loads the existingFields map given the current context
-   */
-  refreshExistingFields: (args: {
-    dateRange: DateRange;
-    currentIndexPatternTitle: string;
-    dslQuery: object;
-    onNoData?: () => void;
-    existingFields: Record<string, Record<string, boolean>>;
-    indexPatternList: IndexPattern[];
-    isFirstExistenceFetch: boolean;
-  }) => Promise<void>;
 
   replaceDataViewId: (newDataView: DataView) => Promise<void>;
   /**
@@ -150,14 +132,6 @@ export function createIndexPatternService({
     },
     ensureIndexPattern: (args) =>
       ensureIndexPattern({ onError: onChangeError, dataViews, ...args }),
-    refreshExistingFields: (args) =>
-      syncExistingFields({
-        updateIndexPatterns,
-        ...args,
-        data,
-        dataViews,
-        core,
-      }),
     loadIndexPatternRefs: async ({ isFullEditor }) =>
       isFullEditor ? loadIndexPatternRefs(dataViews) : [],
     getDefaultIndex: () => core.uiSettings.get('defaultIndex'),

x-pack/plugins/lens/public/datasources/form_based/__mocks__/loader.ts

@@ -28,11 +28,9 @@ export function loadInitialDataViews() {
   const restricted = createMockedRestrictedIndexPattern();
   return {
     indexPatternRefs: [],
-    existingFields: {},
     indexPatterns: {
       [indexPattern.id]: indexPattern,
       [restricted.id]: restricted,
     },
-    isFirstExistenceFetch: false,
   };
 }

x-pack/plugins/lens/public/datasources/form_based/datapanel.scss

@@ -14,15 +14,6 @@
   margin-bottom: $euiSizeS;
 }
 
-.lnsInnerIndexPatternDataPanel__titleTooltip {
-  margin-right: $euiSizeXS;
-}
-
-.lnsInnerIndexPatternDataPanel__fieldItems {
-  // Quick fix for making sure the shadow and focus rings are visible outside the accordion bounds
-  padding: $euiSizeXS;
-}
-
 .lnsInnerIndexPatternDataPanel__textField {
   @include euiFormControlLayoutPadding(1, 'right');
   @include euiFormControlLayoutPadding(1, 'left');
@@ -60,4 +51,4 @@
 
 .lnsFilterButton .euiFilterButton__textShift {
   min-width: 0;
-}
+}

x-pack/plugins/lens/public/datasources/form_based/datapanel.tsx

@@ -6,32 +6,38 @@
  */
 
 import './datapanel.scss';
-import { uniq, groupBy } from 'lodash';
-import React, { useState, memo, useCallback, useMemo, useRef, useEffect } from 'react';
+import { uniq } from 'lodash';
+import React, { memo, useCallback, useEffect, useMemo, useRef, useState } from 'react';
 import {
+  EuiCallOut,
+  EuiContextMenuItem,
+  EuiContextMenuPanel,
+  EuiFilterButton,
   EuiFlexGroup,
   EuiFlexItem,
-  EuiContextMenuPanel,
-  EuiContextMenuItem,
-  EuiPopover,
-  EuiCallOut,
   EuiFormControlLayout,
-  EuiFilterButton,
-  EuiScreenReaderOnly,
   EuiIcon,
+  EuiPopover,
+  EuiProgress,
+  htmlIdGenerator,
 } from '@elastic/eui';
 import { i18n } from '@kbn/i18n';
-import type { EsQueryConfig, Query, Filter } from '@kbn/es-query';
 import { FormattedMessage } from '@kbn/i18n-react';
 import type { CoreStart } from '@kbn/core/public';
 import type { DataPublicPluginStart } from '@kbn/data-plugin/public';
+import { type DataView } from '@kbn/data-plugin/common';
 import type { FieldFormatsStart } from '@kbn/field-formats-plugin/public';
-import { htmlIdGenerator } from '@elastic/eui';
-import { buildEsQuery } from '@kbn/es-query';
-import { getEsQueryConfig } from '@kbn/data-plugin/public';
 import { IndexPatternFieldEditorStart } from '@kbn/data-view-field-editor-plugin/public';
 import { VISUALIZE_GEO_FIELD_TRIGGER } from '@kbn/ui-actions-plugin/public';
 import type { DataViewsPublicPluginStart } from '@kbn/data-views-plugin/public';
+import {
+  FieldsGroupNames,
+  FieldListGrouped,
+  type FieldListGroupedProps,
+  useExistingFieldsFetcher,
+  useGroupedFields,
+  useExistingFieldsReader,
+} from '@kbn/unified-field-list-plugin/public';
 import { ChartsPluginSetup } from '@kbn/charts-plugin/public';
 import type {
   DatasourceDataPanelProps,
@@ -42,12 +48,11 @@ import type {
 } from '../../types';
 import { ChildDragDropProvider, DragContextState } from '../../drag_drop';
 import type { FormBasedPrivateState } from './types';
-import { Loader } from '../../loader';
 import { LensFieldIcon } from '../../shared_components/field_picker/lens_field_icon';
 import { getFieldType } from './pure_utils';
-import { FieldGroups, FieldList } from './field_list';
-import { fieldContainsData, fieldExists } from '../../shared_components';
+import { fieldContainsData } from '../../shared_components';
 import { IndexPatternServiceAPI } from '../../data_views_service/service';
+import { FieldItem } from './field_item';
 
 export type Props = Omit<
   DatasourceDataPanelProps<FormBasedPrivateState>,
@@ -65,10 +70,6 @@ export type Props = Omit<
   layerFields?: string[];
 };
 
-function sortFields(fieldA: IndexPatternField, fieldB: IndexPatternField) {
-  return fieldA.displayName.localeCompare(fieldB.displayName, undefined, { sensitivity: 'base' });
-}
-
 const supportedFieldTypes = new Set([
   'string',
   'number',
@@ -104,25 +105,8 @@ const fieldTypeNames: Record<DataType, string> = {
   murmur3: i18n.translate('xpack.lens.datatypes.murmur3', { defaultMessage: 'murmur3' }),
 };
 
-// Wrapper around buildEsQuery, handling errors (e.g. because a query can't be parsed) by
-// returning a query dsl object not matching anything
-function buildSafeEsQuery(
-  indexPattern: IndexPattern,
-  query: Query,
-  filters: Filter[],
-  queryConfig: EsQueryConfig
-) {
-  try {
-    return buildEsQuery(indexPattern, query, filters, queryConfig);
-  } catch (e) {
-    return {
-      bool: {
-        must_not: {
-          match_all: {},
-        },
-      },
-    };
-  }
+function onSupportedFieldFilter(field: IndexPatternField): boolean {
+  return supportedFieldTypes.has(field.type);
 }
 
 export function FormBasedDataPanel({
@@ -147,51 +131,22 @@ export function FormBasedDataPanel({
   usedIndexPatterns,
   layerFields,
 }: Props) {
-  const { indexPatterns, indexPatternRefs, existingFields, isFirstExistenceFetch } =
-    frame.dataViews;
+  const { indexPatterns, indexPatternRefs } = frame.dataViews;
   const { currentIndexPatternId } = state;
 
-  const indexPatternList = uniq(
-    (
-      usedIndexPatterns ?? Object.values(state.layers).map(({ indexPatternId }) => indexPatternId)
-    ).concat(currentIndexPatternId)
-  )
-    .filter((id) => !!indexPatterns[id])
-    .sort()
-    .map((id) => indexPatterns[id]);
-
-  const dslQuery = buildSafeEsQuery(
-    indexPatterns[currentIndexPatternId],
-    query,
-    filters,
-    getEsQueryConfig(core.uiSettings)
-  );
+  const activeIndexPatterns = useMemo(() => {
+    return uniq(
+      (
+        usedIndexPatterns ?? Object.values(state.layers).map(({ indexPatternId }) => indexPatternId)
+      ).concat(currentIndexPatternId)
+    )
+      .filter((id) => !!indexPatterns[id])
+      .sort()
+      .map((id) => indexPatterns[id]);
+  }, [usedIndexPatterns, indexPatterns, state.layers, currentIndexPatternId]);
 
   return (
     <>
-      <Loader
-        load={() =>
-          indexPatternService.refreshExistingFields({
-            dateRange,
-            currentIndexPatternTitle: indexPatterns[currentIndexPatternId]?.title || '',
-            onNoData: showNoDataPopover,
-            dslQuery,
-            indexPatternList,
-            isFirstExistenceFetch,
-            existingFields,
-          })
-        }
-        loadDeps={[
-          query,
-          filters,
-          dateRange.fromDate,
-          dateRange.toDate,
-          indexPatternList.map((x) => `${x.title}:${x.timeFieldName}`).join(','),
-          // important here to rerun the fields existence on indexPattern change (i.e. add new fields in place)
-          frame.dataViews.indexPatterns,
-        ]}
-      />
-
       {Object.keys(indexPatterns).length === 0 && indexPatternRefs.length === 0 ? (
         <EuiFlexGroup
           gutterSize="m"
@@ -237,6 +192,8 @@ export function FormBasedDataPanel({
           onIndexPatternRefresh={onIndexPatternRefresh}
           frame={frame}
           layerFields={layerFields}
+          showNoDataPopover={showNoDataPopover}
+          activeIndexPatterns={activeIndexPatterns}
         />
       )}
     </>
@@ -252,18 +209,6 @@ interface DataPanelState {
   isMetaAccordionOpen: boolean;
 }
 
-const defaultFieldGroups: {
-  specialFields: IndexPatternField[];
-  availableFields: IndexPatternField[];
-  emptyFields: IndexPatternField[];
-  metaFields: IndexPatternField[];
-} = {
-  specialFields: [],
-  availableFields: [],
-  emptyFields: [],
-  metaFields: [],
-};
-
 const htmlId = htmlIdGenerator('datapanel');
 const fieldSearchDescriptionId = htmlId();
 
@@ -286,9 +231,11 @@ export const InnerFormBasedDataPanel = function InnerFormBasedDataPanel({
   frame,
   onIndexPatternRefresh,
   layerFields,
+  showNoDataPopover,
+  activeIndexPatterns,
 }: Omit<
   DatasourceDataPanelProps,
-  'state' | 'setState' | 'showNoDataPopover' | 'core' | 'onChangeIndexPattern' | 'usedIndexPatterns'
+  'state' | 'setState' | 'core' | 'onChangeIndexPattern' | 'usedIndexPatterns'
 > & {
   data: DataPublicPluginStart;
   dataViews: DataViewsPublicPluginStart;
@@ -301,6 +248,7 @@ export const InnerFormBasedDataPanel = function InnerFormBasedDataPanel({
   indexPatternFieldEditor: IndexPatternFieldEditorStart;
   onIndexPatternRefresh: () => void;
   layerFields?: string[];
+  activeIndexPatterns: IndexPattern[];
 }) {
   const [localState, setLocalState] = useState<DataPanelState>({
     nameFilter: '',
@@ -310,10 +258,30 @@ export const InnerFormBasedDataPanel = function InnerFormBasedDataPanel({
     isEmptyAccordionOpen: false,
     isMetaAccordionOpen: false,
   });
-  const { existenceFetchFailed, existenceFetchTimeout, indexPatterns, existingFields } =
-    frame.dataViews;
+  const { indexPatterns } = frame.dataViews;
   const currentIndexPattern = indexPatterns[currentIndexPatternId];
-  const existingFieldsForIndexPattern = existingFields[currentIndexPattern?.title];
+
+  const { refetchFieldsExistenceInfo, isProcessing } = useExistingFieldsFetcher({
+    dataViews: activeIndexPatterns as unknown as DataView[],
+    query,
+    filters,
+    fromDate: dateRange.fromDate,
+    toDate: dateRange.toDate,
+    services: {
+      data,
+      dataViews,
+      core,
+    },
+    onNoData: (dataViewId) => {
+      if (dataViewId === currentIndexPatternId) {
+        showNoDataPopover();
+      }
+    },
+  });
+  const fieldsExistenceReader = useExistingFieldsReader();
+  const fieldsExistenceStatus =
+    fieldsExistenceReader.getFieldsExistenceStatus(currentIndexPatternId);
+
   const visualizeGeoFieldTrigger = uiActions.getTrigger(VISUALIZE_GEO_FIELD_TRIGGER);
   const allFields = useMemo(() => {
     if (!currentIndexPattern) return [];
@@ -331,186 +299,73 @@ export const InnerFormBasedDataPanel = function InnerFormBasedDataPanel({
     ...localState.typeFilter,
   ]);
 
-  const fieldInfoUnavailable =
-    existenceFetchFailed || existenceFetchTimeout || currentIndexPattern?.hasRestrictions;
-
   const editPermission =
     indexPatternFieldEditor.userPermissions.editIndexPattern() || !currentIndexPattern.isPersisted;
 
-  const unfilteredFieldGroups: FieldGroups = useMemo(() => {
-    const containsData = (field: IndexPatternField) => {
-      const overallField = currentIndexPattern?.getFieldByName(field.name);
-      return (
-        overallField &&
-        existingFieldsForIndexPattern &&
-        fieldExists(existingFieldsForIndexPattern, overallField.name)
-      );
-    };
+  const onSelectedFieldFilter = useCallback(
+    (field: IndexPatternField): boolean => {
+      return Boolean(layerFields?.includes(field.name));
+    },
+    [layerFields]
+  );
 
-    const allSupportedTypesFields = allFields.filter((field) =>
-      supportedFieldTypes.has(field.type)
-    );
-    const usedByLayersFields = allFields.filter((field) => layerFields?.includes(field.name));
-    const sorted = allSupportedTypesFields.sort(sortFields);
-    const groupedFields = {
-      ...defaultFieldGroups,
-      ...groupBy(sorted, (field) => {
-        if (field.type === 'document') {
-          return 'specialFields';
-        } else if (field.meta) {
-          return 'metaFields';
-        } else if (containsData(field)) {
-          return 'availableFields';
-        } else return 'emptyFields';
-      }),
-    };
+  const onFilterField = useCallback(
+    (field: IndexPatternField) => {
+      if (
+        localState.nameFilter.length &&
+        !field.name.toLowerCase().includes(localState.nameFilter.toLowerCase()) &&
+        !field.displayName.toLowerCase().includes(localState.nameFilter.toLowerCase())
+      ) {
+        return false;
+      }
+      if (localState.typeFilter.length > 0) {
+        return localState.typeFilter.includes(getFieldType(field) as DataType);
+      }
+      return true;
+    },
+    [localState]
+  );
 
-    const isUsingSampling = core.uiSettings.get('lens:useFieldExistenceSampling');
+  const hasFilters = Boolean(filters.length);
+  const onOverrideFieldGroupDetails = useCallback(
+    (groupName) => {
+      if (groupName === FieldsGroupNames.AvailableFields) {
+        const isUsingSampling = core.uiSettings.get('lens:useFieldExistenceSampling');
 
-    const fieldGroupDefinitions: FieldGroups = {
-      SpecialFields: {
-        fields: groupedFields.specialFields,
-        fieldCount: 1,
-        isAffectedByGlobalFilter: false,
-        isAffectedByTimeFilter: false,
-        isInitiallyOpen: false,
-        showInAccordion: false,
-        title: '',
-        hideDetails: true,
-      },
-      SelectedFields: {
-        fields: usedByLayersFields,
-        fieldCount: usedByLayersFields.length,
-        isInitiallyOpen: true,
-        showInAccordion: true,
-        title: i18n.translate('xpack.lens.indexPattern.selectedFieldsLabel', {
-          defaultMessage: 'Selected fields',
-        }),
-        isAffectedByGlobalFilter: !!filters.length,
-        isAffectedByTimeFilter: true,
-        hideDetails: false,
-        hideIfEmpty: true,
-      },
-      AvailableFields: {
-        fields: groupedFields.availableFields,
-        fieldCount: groupedFields.availableFields.length,
-        isInitiallyOpen: true,
-        showInAccordion: true,
-        title: fieldInfoUnavailable
-          ? i18n.translate('xpack.lens.indexPattern.allFieldsLabel', {
-              defaultMessage: 'All fields',
-            })
-          : i18n.translate('xpack.lens.indexPattern.availableFieldsLabel', {
-              defaultMessage: 'Available fields',
-            }),
-        helpText: isUsingSampling
-          ? i18n.translate('xpack.lens.indexPattern.allFieldsSamplingLabelHelp', {
-              defaultMessage:
-                'Available fields contain the data in the first 500 documents that match your filters. To view all fields, expand Empty fields. You are unable to create visualizations with full text, geographic, flattened, and object fields.',
-            })
-          : i18n.translate('xpack.lens.indexPattern.allFieldsLabelHelp', {
-              defaultMessage:
-                'Drag and drop available fields to the workspace and create visualizations. To change the available fields, select a different data view, edit your queries, or use a different time range. Some field types cannot be visualized in Lens, including full text and geographic fields.',
-            }),
-        isAffectedByGlobalFilter: !!filters.length,
-        isAffectedByTimeFilter: true,
-        // Show details on timeout but not failure
-        hideDetails: fieldInfoUnavailable && !existenceFetchTimeout,
-        defaultNoFieldsMessage: i18n.translate('xpack.lens.indexPatterns.noAvailableDataLabel', {
-          defaultMessage: `There are no available fields that contain data.`,
-        }),
-      },
-      EmptyFields: {
-        fields: groupedFields.emptyFields,
-        fieldCount: groupedFields.emptyFields.length,
-        isAffectedByGlobalFilter: false,
-        isAffectedByTimeFilter: false,
-        isInitiallyOpen: false,
-        showInAccordion: true,
-        hideDetails: false,
-        title: i18n.translate('xpack.lens.indexPattern.emptyFieldsLabel', {
-          defaultMessage: 'Empty fields',
-        }),
-        defaultNoFieldsMessage: i18n.translate('xpack.lens.indexPatterns.noEmptyDataLabel', {
-          defaultMessage: `There are no empty fields.`,
-        }),
-        helpText: i18n.translate('xpack.lens.indexPattern.emptyFieldsLabelHelp', {
-          defaultMessage:
-            'Empty fields did not contain any values in the first 500 documents based on your filters.',
-        }),
-      },
-      MetaFields: {
-        fields: groupedFields.metaFields,
-        fieldCount: groupedFields.metaFields.length,
-        isAffectedByGlobalFilter: false,
-        isAffectedByTimeFilter: false,
-        isInitiallyOpen: false,
-        showInAccordion: true,
-        hideDetails: false,
-        title: i18n.translate('xpack.lens.indexPattern.metaFieldsLabel', {
-          defaultMessage: 'Meta fields',
-        }),
-        defaultNoFieldsMessage: i18n.translate('xpack.lens.indexPatterns.noMetaDataLabel', {
-          defaultMessage: `There are no meta fields.`,
-        }),
-      },
-    };
+        return {
+          helpText: isUsingSampling
+            ? i18n.translate('xpack.lens.indexPattern.allFieldsSamplingLabelHelp', {
+                defaultMessage:
+                  'Available fields contain the data in the first 500 documents that match your filters. To view all fields, expand Empty fields. You are unable to create visualizations with full text, geographic, flattened, and object fields.',
+              })
+            : i18n.translate('xpack.lens.indexPattern.allFieldsLabelHelp', {
+                defaultMessage:
+                  'Drag and drop available fields to the workspace and create visualizations. To change the available fields, select a different data view, edit your queries, or use a different time range. Some field types cannot be visualized in Lens, including full text and geographic fields.',
+              }),
+          isAffectedByGlobalFilter: hasFilters,
+        };
+      }
+      if (groupName === FieldsGroupNames.SelectedFields) {
+        return {
+          isAffectedByGlobalFilter: hasFilters,
+        };
+      }
+    },
+    [core.uiSettings, hasFilters]
+  );
 
-    // do not show empty field accordion if there is no existence information
-    if (fieldInfoUnavailable) {
-      delete fieldGroupDefinitions.EmptyFields;
-    }
-
-    return fieldGroupDefinitions;
-  }, [
+  const { fieldGroups } = useGroupedFields<IndexPatternField>({
+    dataViewId: currentIndexPatternId,
     allFields,
-    core.uiSettings,
-    fieldInfoUnavailable,
-    filters.length,
-    existenceFetchTimeout,
-    currentIndexPattern,
-    existingFieldsForIndexPattern,
-    layerFields,
-  ]);
-
-  const fieldGroups: FieldGroups = useMemo(() => {
-    const filterFieldGroup = (fieldGroup: IndexPatternField[]) =>
-      fieldGroup.filter((field) => {
-        if (
-          localState.nameFilter.length &&
-          !field.name.toLowerCase().includes(localState.nameFilter.toLowerCase()) &&
-          !field.displayName.toLowerCase().includes(localState.nameFilter.toLowerCase())
-        ) {
-          return false;
-        }
-        if (localState.typeFilter.length > 0) {
-          return localState.typeFilter.includes(getFieldType(field) as DataType);
-        }
-        return true;
-      });
-    return Object.fromEntries(
-      Object.entries(unfilteredFieldGroups).map(([name, group]) => [
-        name,
-        { ...group, fields: filterFieldGroup(group.fields) },
-      ])
-    );
-  }, [unfilteredFieldGroups, localState.nameFilter, localState.typeFilter]);
-
-  const checkFieldExists = useCallback(
-    (field: IndexPatternField) =>
-      fieldContainsData(field.name, currentIndexPattern, existingFieldsForIndexPattern),
-    [currentIndexPattern, existingFieldsForIndexPattern]
-  );
-
-  const { nameFilter, typeFilter } = localState;
-
-  const filter = useMemo(
-    () => ({
-      nameFilter,
-      typeFilter,
-    }),
-    [nameFilter, typeFilter]
-  );
+    services: {
+      dataViews,
+    },
+    fieldsExistenceReader,
+    onFilterField,
+    onSupportedFieldFilter,
+    onSelectedFieldFilter,
+    onOverrideFieldGroupDetails,
+  });
 
   const closeFieldEditor = useRef<() => void | undefined>();
 
@@ -560,6 +415,7 @@ export const InnerFormBasedDataPanel = function InnerFormBasedDataPanel({
               onSave: () => {
                 if (indexPatternInstance.isPersisted()) {
                   refreshFieldList();
+                  refetchFieldsExistenceInfo(indexPatternInstance.id);
                 } else {
                   indexPatternService.replaceDataViewId(indexPatternInstance);
                 }
@@ -574,6 +430,7 @@ export const InnerFormBasedDataPanel = function InnerFormBasedDataPanel({
       indexPatternFieldEditor,
       refreshFieldList,
       indexPatternService,
+      refetchFieldsExistenceInfo,
     ]
   );
 
@@ -590,6 +447,7 @@ export const InnerFormBasedDataPanel = function InnerFormBasedDataPanel({
               onDelete: () => {
                 if (indexPatternInstance.isPersisted()) {
                   refreshFieldList();
+                  refetchFieldsExistenceInfo(indexPatternInstance.id);
                 } else {
                   indexPatternService.replaceDataViewId(indexPatternInstance);
                 }
@@ -604,24 +462,39 @@ export const InnerFormBasedDataPanel = function InnerFormBasedDataPanel({
       indexPatternFieldEditor,
       indexPatternService,
       refreshFieldList,
+      refetchFieldsExistenceInfo,
     ]
   );
 
-  const fieldProps = useMemo(
-    () => ({
-      core,
-      data,
-      fieldFormats,
-      indexPattern: currentIndexPattern,
-      highlight: localState.nameFilter.toLowerCase(),
-      dateRange,
-      query,
-      filters,
-      chartsThemeService: charts.theme,
-    }),
+  const renderFieldItem: FieldListGroupedProps<IndexPatternField>['renderFieldItem'] = useCallback(
+    ({ field, itemIndex, groupIndex, hideDetails }) => (
+      <FieldItem
+        field={field}
+        exists={fieldContainsData(
+          field.name,
+          currentIndexPattern,
+          fieldsExistenceReader.hasFieldData
+        )}
+        hideDetails={hideDetails || field.type === 'document'}
+        itemIndex={itemIndex}
+        groupIndex={groupIndex}
+        dropOntoWorkspace={dropOntoWorkspace}
+        hasSuggestionForField={hasSuggestionForField}
+        editField={editField}
+        removeField={removeField}
+        uiActions={uiActions}
+        core={core}
+        fieldFormats={fieldFormats}
+        indexPattern={currentIndexPattern}
+        highlight={localState.nameFilter.toLowerCase()}
+        dateRange={dateRange}
+        query={query}
+        filters={filters}
+        chartsThemeService={charts.theme}
+      />
+    ),
     [
       core,
-      data,
       fieldFormats,
       currentIndexPattern,
       dateRange,
@@ -629,6 +502,12 @@ export const InnerFormBasedDataPanel = function InnerFormBasedDataPanel({
       filters,
       localState.nameFilter,
       charts.theme,
+      fieldsExistenceReader.hasFieldData,
+      dropOntoWorkspace,
+      hasSuggestionForField,
+      editField,
+      removeField,
+      uiActions,
     ]
   );
 
@@ -640,6 +519,7 @@ export const InnerFormBasedDataPanel = function InnerFormBasedDataPanel({
         direction="column"
         responsive={false}
       >
+        {isProcessing && <EuiProgress size="xs" color="accent" position="absolute" />}
         <EuiFlexItem grow={false}>
           <EuiFormControlLayout
             icon="search"
@@ -734,36 +614,14 @@ export const InnerFormBasedDataPanel = function InnerFormBasedDataPanel({
             />
           </EuiFormControlLayout>
         </EuiFlexItem>
-        <EuiScreenReaderOnly>
-          <div aria-live="polite" id={fieldSearchDescriptionId}>
-            {i18n.translate('xpack.lens.indexPatterns.fieldSearchLiveRegion', {
-              defaultMessage:
-                '{availableFields} available {availableFields, plural, one {field} other {fields}}. {emptyFields} empty {emptyFields, plural, one {field} other {fields}}. {metaFields} meta {metaFields, plural, one {field} other {fields}}.',
-              values: {
-                availableFields: fieldGroups.AvailableFields.fields.length,
-                // empty fields can be undefined if there is no existence information to be fetched
-                emptyFields: fieldGroups.EmptyFields?.fields.length || 0,
-                metaFields: fieldGroups.MetaFields.fields.length,
-              },
-            })}
-          </div>
-        </EuiScreenReaderOnly>
         <EuiFlexItem>
-          <FieldList
-            exists={checkFieldExists}
-            fieldProps={fieldProps}
+          <FieldListGrouped<IndexPatternField>
             fieldGroups={fieldGroups}
-            hasSyncedExistingFields={!!existingFieldsForIndexPattern}
-            filter={filter}
-            currentIndexPatternId={currentIndexPatternId}
-            existenceFetchFailed={existenceFetchFailed}
-            existenceFetchTimeout={existenceFetchTimeout}
-            existFieldsInIndex={!!allFields.length}
-            dropOntoWorkspace={dropOntoWorkspace}
-            hasSuggestionForField={hasSuggestionForField}
-            editField={editField}
-            removeField={removeField}
-            uiActions={uiActions}
+            fieldsExistenceStatus={fieldsExistenceStatus}
+            fieldsExistInIndex={!!allFields.length}
+            renderFieldItem={renderFieldItem}
+            screenReaderDescriptionForSearchInputId={fieldSearchDescriptionId}
+            data-test-subj="lnsIndexPattern"
           />
         </EuiFlexItem>
       </EuiFlexGroup>

x-pack/plugins/lens/public/datasources/form_based/dimension_panel/dimension_editor.tsx

@@ -606,7 +606,6 @@ export function DimensionEditor(props: DimensionEditorProps) {
     setIsCloseable,
     paramEditorCustomProps,
     ReferenceEditor,
-    existingFields: props.existingFields,
     ...services,
   };
 
@@ -789,7 +788,6 @@ export function DimensionEditor(props: DimensionEditorProps) {
                 }}
                 validation={validation}
                 currentIndexPattern={currentIndexPattern}
-                existingFields={props.existingFields}
                 selectionStyle={selectedOperationDefinition.selectionStyle}
                 dateRange={dateRange}
                 labelAppend={selectedOperationDefinition?.getHelpMessage?.({
@@ -815,7 +813,6 @@ export function DimensionEditor(props: DimensionEditorProps) {
           selectedColumn={selectedColumn as FieldBasedIndexPatternColumn}
           columnId={columnId}
           indexPattern={currentIndexPattern}
-          existingFields={props.existingFields}
           operationSupportMatrix={operationSupportMatrix}
           updateLayer={(newLayer) => {
             if (temporaryQuickFunction) {
@@ -845,7 +842,6 @@ export function DimensionEditor(props: DimensionEditorProps) {
   const customParamEditor = ParamEditor ? (
     <>
       <ParamEditor
-        existingFields={props.existingFields}
         layer={state.layers[layerId]}
         activeData={props.activeData}
         paramEditorUpdater={

x-pack/plugins/lens/public/datasources/form_based/dimension_panel/dimension_panel.test.tsx

@@ -32,6 +32,7 @@ import {
   CoreStart,
 } from '@kbn/core/public';
 import { IStorageWrapper } from '@kbn/kibana-utils-plugin/public';
+import { useExistingFieldsReader } from '@kbn/unified-field-list-plugin/public/hooks/use_existing_fields';
 import { generateId } from '../../../id_generator';
 import { FormBasedPrivateState } from '../types';
 import { LayerTypes } from '@kbn/expression-xy-plugin/public';
@@ -78,6 +79,16 @@ jest.mock('../operations/definitions/formula/editor/formula_editor', () => {
   };
 });
 
+jest.mock('@kbn/unified-field-list-plugin/public/hooks/use_existing_fields', () => ({
+  useExistingFieldsReader: jest.fn(() => {
+    return {
+      hasFieldData: (dataViewId: string, fieldName: string) => {
+        return ['timestamp', 'bytes', 'memory', 'source'].includes(fieldName);
+      },
+    };
+  }),
+}));
+
 const fields = [
   {
     name: 'timestamp',
@@ -197,14 +208,6 @@ describe('FormBasedDimensionEditor', () => {
 
     defaultProps = {
       indexPatterns: expectedIndexPatterns,
-      existingFields: {
-        'my-fake-index-pattern': {
-          timestamp: true,
-          bytes: true,
-          memory: true,
-          source: true,
-        },
-      },
       state,
       setState,
       dateRange: { fromDate: 'now-1d', toDate: 'now' },
@@ -339,16 +342,15 @@ describe('FormBasedDimensionEditor', () => {
   });
 
   it('should hide fields that have no data', () => {
-    const props = {
-      ...defaultProps,
-      existingFields: {
-        'my-fake-index-pattern': {
-          timestamp: true,
-          source: true,
+    (useExistingFieldsReader as jest.Mock).mockImplementationOnce(() => {
+      return {
+        hasFieldData: (dataViewId: string, fieldName: string) => {
+          return ['timestamp', 'source'].includes(fieldName);
         },
-      },
-    };
-    wrapper = mount(<FormBasedDimensionEditorComponent {...props} />);
+      };
+    });
+
+    wrapper = mount(<FormBasedDimensionEditorComponent {...defaultProps} />);
 
     const options = wrapper
       .find(EuiComboBox)

x-pack/plugins/lens/public/datasources/form_based/dimension_panel/field_input.test.tsx

@@ -112,11 +112,7 @@ function getLayer(col1: GenericIndexPatternColumn = getStringBasedOperationColum
     },
   };
 }
-function getDefaultOperationSupportMatrix(
-  layer: FormBasedLayer,
-  columnId: string,
-  existingFields: Record<string, Record<string, boolean>>
-) {
+function getDefaultOperationSupportMatrix(layer: FormBasedLayer, columnId: string) {
   return getOperationSupportMatrix({
     state: {
       layers: { layer1: layer },
@@ -130,29 +126,36 @@ function getDefaultOperationSupportMatrix(
   });
 }
 
-function getExistingFields() {
-  const fields: Record<string, boolean> = {};
-  for (const field of defaultProps.indexPattern.fields) {
-    fields[field.name] = true;
-  }
-  return {
-    [defaultProps.indexPattern.title]: fields,
-  };
-}
+const mockedReader = {
+  hasFieldData: (dataViewId: string, fieldName: string) => {
+    if (defaultProps.indexPattern.id !== dataViewId) {
+      return false;
+    }
+
+    const map: Record<string, boolean> = {};
+    for (const field of defaultProps.indexPattern.fields) {
+      map[field.name] = true;
+    }
+
+    return map[fieldName];
+  },
+};
+
+jest.mock('@kbn/unified-field-list-plugin/public/hooks/use_existing_fields', () => ({
+  useExistingFieldsReader: jest.fn(() => mockedReader),
+}));
 
 describe('FieldInput', () => {
   it('should render a field select box', () => {
     const updateLayerSpy = jest.fn();
     const layer = getLayer();
-    const existingFields = getExistingFields();
-    const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1', existingFields);
+    const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1');
     const instance = mount(
       <FieldInput
         {...defaultProps}
         layer={layer}
         columnId={'col1'}
         updateLayer={updateLayerSpy}
-        existingFields={existingFields}
         operationSupportMatrix={operationSupportMatrix}
       />
     );
@@ -163,15 +166,13 @@ describe('FieldInput', () => {
   it('should render an error message when incomplete operation is on', () => {
     const updateLayerSpy = jest.fn();
     const layer = getLayer();
-    const existingFields = getExistingFields();
-    const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1', existingFields);
+    const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1');
     const instance = mount(
       <FieldInput
         {...defaultProps}
         layer={layer}
         columnId={'col1'}
         updateLayer={updateLayerSpy}
-        existingFields={existingFields}
         operationSupportMatrix={operationSupportMatrix}
         incompleteOperation={'terms'}
         selectedColumn={getStringBasedOperationColumn()}
@@ -195,19 +196,13 @@ describe('FieldInput', () => {
     (_, col: ReferenceBasedIndexPatternColumn) => {
       const updateLayerSpy = jest.fn();
       const layer = getLayer(col);
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix(
-        layer,
-        'col1',
-        existingFields
-      );
+      const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1');
       const instance = mount(
         <FieldInput
           {...defaultProps}
           layer={layer}
           columnId={'col1'}
           updateLayer={updateLayerSpy}
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           incompleteOperation={'terms'}
         />
@@ -234,19 +229,13 @@ describe('FieldInput', () => {
     (_, col: ReferenceBasedIndexPatternColumn) => {
       const updateLayerSpy = jest.fn();
       const layer = getLayer(col);
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix(
-        layer,
-        'col1',
-        existingFields
-      );
+      const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1');
       const instance = mount(
         <FieldInput
           {...defaultProps}
           layer={layer}
           columnId={'col1'}
           updateLayer={updateLayerSpy}
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           selectedColumn={getStringBasedOperationColumn()}
           incompleteOperation={'terms'}
@@ -269,15 +258,13 @@ describe('FieldInput', () => {
   it('should render an error message for invalid fields', () => {
     const updateLayerSpy = jest.fn();
     const layer = getLayer();
-    const existingFields = getExistingFields();
-    const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1', existingFields);
+    const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1');
     const instance = mount(
       <FieldInput
         {...defaultProps}
         layer={layer}
         columnId={'col1'}
         updateLayer={updateLayerSpy}
-        existingFields={existingFields}
         operationSupportMatrix={operationSupportMatrix}
         currentFieldIsInvalid
       />
@@ -295,15 +282,13 @@ describe('FieldInput', () => {
   it('should render a help message when passed and no errors are found', () => {
     const updateLayerSpy = jest.fn();
     const layer = getLayer();
-    const existingFields = getExistingFields();
-    const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1', existingFields);
+    const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1');
     const instance = mount(
       <FieldInput
         {...defaultProps}
         layer={layer}
         columnId={'col1'}
         updateLayer={updateLayerSpy}
-        existingFields={existingFields}
         operationSupportMatrix={operationSupportMatrix}
         helpMessage={'My help message'}
       />
@@ -320,15 +305,13 @@ describe('FieldInput', () => {
   it('should prioritize errors over help messages', () => {
     const updateLayerSpy = jest.fn();
     const layer = getLayer();
-    const existingFields = getExistingFields();
-    const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1', existingFields);
+    const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1');
     const instance = mount(
       <FieldInput
         {...defaultProps}
         layer={layer}
         columnId={'col1'}
         updateLayer={updateLayerSpy}
-        existingFields={existingFields}
         operationSupportMatrix={operationSupportMatrix}
         currentFieldIsInvalid
         helpMessage={'My help message'}
@@ -346,15 +329,13 @@ describe('FieldInput', () => {
   it('should update the layer on field selection', () => {
     const updateLayerSpy = jest.fn();
     const layer = getLayer();
-    const existingFields = getExistingFields();
-    const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1', existingFields);
+    const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1');
     const instance = mount(
       <FieldInput
         {...defaultProps}
         layer={layer}
         columnId={'col1'}
         updateLayer={updateLayerSpy}
-        existingFields={existingFields}
         operationSupportMatrix={operationSupportMatrix}
         selectedColumn={getStringBasedOperationColumn()}
       />
@@ -372,15 +353,13 @@ describe('FieldInput', () => {
   it('should not trigger when the same selected field is selected again', () => {
     const updateLayerSpy = jest.fn();
     const layer = getLayer();
-    const existingFields = getExistingFields();
-    const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1', existingFields);
+    const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1');
     const instance = mount(
       <FieldInput
         {...defaultProps}
         layer={layer}
         columnId={'col1'}
         updateLayer={updateLayerSpy}
-        existingFields={existingFields}
         operationSupportMatrix={operationSupportMatrix}
         selectedColumn={getStringBasedOperationColumn()}
       />
@@ -398,15 +377,13 @@ describe('FieldInput', () => {
   it('should prioritize incomplete fields over selected column field to display', () => {
     const updateLayerSpy = jest.fn();
     const layer = getLayer();
-    const existingFields = getExistingFields();
-    const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1', existingFields);
+    const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1');
     const instance = mount(
       <FieldInput
         {...defaultProps}
         layer={layer}
         columnId={'col1'}
         updateLayer={updateLayerSpy}
-        existingFields={existingFields}
         operationSupportMatrix={operationSupportMatrix}
         incompleteField={'dest'}
         selectedColumn={getStringBasedOperationColumn()}
@@ -425,15 +402,13 @@ describe('FieldInput', () => {
     const updateLayerSpy = jest.fn();
     const onDeleteColumn = jest.fn();
     const layer = getLayer();
-    const existingFields = getExistingFields();
-    const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1', existingFields);
+    const operationSupportMatrix = getDefaultOperationSupportMatrix(layer, 'col1');
     const instance = mount(
       <FieldInput
         {...defaultProps}
         layer={layer}
         columnId={'col1'}
         updateLayer={updateLayerSpy}
-        existingFields={existingFields}
         operationSupportMatrix={operationSupportMatrix}
         onDeleteColumn={onDeleteColumn}
       />

x-pack/plugins/lens/public/datasources/form_based/dimension_panel/field_input.tsx

@@ -22,7 +22,6 @@ export function FieldInput({
   selectedColumn,
   columnId,
   indexPattern,
-  existingFields,
   operationSupportMatrix,
   updateLayer,
   onDeleteColumn,
@@ -62,7 +61,6 @@ export function FieldInput({
       <FieldSelect
         fieldIsInvalid={currentFieldIsInvalid}
         currentIndexPattern={indexPattern}
-        existingFields={existingFields[indexPattern.title]}
         operationByField={operationSupportMatrix.operationByField}
         selectedOperationType={
           // Allows operation to be selected before creating a valid column

x-pack/plugins/lens/public/datasources/form_based/dimension_panel/field_select.tsx

@@ -10,6 +10,7 @@ import { partition } from 'lodash';
 import React, { useMemo } from 'react';
 import { i18n } from '@kbn/i18n';
 import { EuiComboBoxOptionOption, EuiComboBoxProps } from '@elastic/eui';
+import { useExistingFieldsReader } from '@kbn/unified-field-list-plugin/public';
 import type { OperationType } from '../form_based';
 import type { OperationSupportMatrix } from './operation_support';
 import {
@@ -18,7 +19,7 @@ import {
   FieldPicker,
 } from '../../../shared_components/field_picker';
 import { fieldContainsData } from '../../../shared_components';
-import type { ExistingFieldsMap, IndexPattern } from '../../../types';
+import type { IndexPattern } from '../../../types';
 import { getFieldType } from '../pure_utils';
 
 export type FieldChoiceWithOperationType = FieldOptionValue & {
@@ -33,7 +34,6 @@ export interface FieldSelectProps extends EuiComboBoxProps<EuiComboBoxOptionOpti
   operationByField: OperationSupportMatrix['operationByField'];
   onChoose: (choice: FieldChoiceWithOperationType) => void;
   onDeleteColumn?: () => void;
-  existingFields: ExistingFieldsMap[string];
   fieldIsInvalid: boolean;
   markAllFieldsCompatible?: boolean;
   'data-test-subj'?: string;
@@ -47,12 +47,12 @@ export function FieldSelect({
   operationByField,
   onChoose,
   onDeleteColumn,
-  existingFields,
   fieldIsInvalid,
   markAllFieldsCompatible,
   ['data-test-subj']: dataTestSub,
   ...rest
 }: FieldSelectProps) {
+  const { hasFieldData } = useExistingFieldsReader();
   const memoizedFieldOptions = useMemo(() => {
     const fields = Object.keys(operationByField).sort();
 
@@ -67,8 +67,8 @@ export function FieldSelect({
       (field) => currentIndexPattern.getFieldByName(field)?.type === 'document'
     );
 
-    function containsData(field: string) {
-      return fieldContainsData(field, currentIndexPattern, existingFields);
+    function containsData(fieldName: string) {
+      return fieldContainsData(fieldName, currentIndexPattern, hasFieldData);
     }
 
     function fieldNamesToOptions(items: string[]) {
@@ -145,7 +145,7 @@ export function FieldSelect({
     selectedOperationType,
     currentIndexPattern,
     operationByField,
-    existingFields,
+    hasFieldData,
     markAllFieldsCompatible,
   ]);
 

x-pack/plugins/lens/public/datasources/form_based/dimension_panel/reference_editor.test.tsx

@@ -28,6 +28,16 @@ import {
 import { FieldSelect } from './field_select';
 import { FormBasedLayer } from '../types';
 
+jest.mock('@kbn/unified-field-list-plugin/public/hooks/use_existing_fields', () => ({
+  useExistingFieldsReader: jest.fn(() => {
+    return {
+      hasFieldData: (dataViewId: string, fieldName: string) => {
+        return ['timestamp', 'bytes', 'memory', 'source'].includes(fieldName);
+      },
+    };
+  }),
+}));
+
 jest.mock('../operations');
 
 describe('reference editor', () => {
@@ -59,14 +69,6 @@ describe('reference editor', () => {
       paramEditorUpdater,
       selectionStyle: 'full' as const,
       currentIndexPattern: createMockedIndexPattern(),
-      existingFields: {
-        'my-fake-index-pattern': {
-          timestamp: true,
-          bytes: true,
-          memory: true,
-          source: true,
-        },
-      },
       dateRange: { fromDate: 'now-1d', toDate: 'now' },
       storage: {} as IStorageWrapper,
       uiSettings: {} as IUiSettingsClient,

x-pack/plugins/lens/public/datasources/form_based/dimension_panel/reference_editor.tsx

@@ -29,12 +29,7 @@ import {
 import { FieldChoiceWithOperationType, FieldSelect } from './field_select';
 import { hasField } from '../pure_utils';
 import type { FormBasedLayer } from '../types';
-import type {
-  ExistingFieldsMap,
-  IndexPattern,
-  IndexPatternField,
-  ParamEditorCustomProps,
-} from '../../../types';
+import type { IndexPattern, IndexPatternField, ParamEditorCustomProps } from '../../../types';
 import type { FormBasedDimensionEditorProps } from './dimension_panel';
 import { FormRow } from '../operations/definitions/shared_components';
 
@@ -83,7 +78,6 @@ export interface ReferenceEditorProps {
   fieldLabel?: string;
   operationDefinitionMap: Record<string, GenericOperationDefinition>;
   isInline?: boolean;
-  existingFields: ExistingFieldsMap;
   dateRange: DateRange;
   labelAppend?: EuiFormRowProps['labelAppend'];
   isFullscreen: boolean;
@@ -114,7 +108,6 @@ export interface ReferenceEditorProps {
 export const ReferenceEditor = (props: ReferenceEditorProps) => {
   const {
     currentIndexPattern,
-    existingFields,
     validation,
     selectionStyle,
     labelAppend,
@@ -307,7 +300,6 @@ export const ReferenceEditor = (props: ReferenceEditorProps) => {
           <FieldSelect
             fieldIsInvalid={showFieldInvalid || showFieldMissingInvalid}
             currentIndexPattern={currentIndexPattern}
-            existingFields={existingFields[currentIndexPattern.title]}
             operationByField={operationSupportMatrix.operationByField}
             selectedOperationType={
               // Allows operation to be selected before creating a valid column

x-pack/plugins/lens/public/datasources/form_based/field_list.tsx

@@ -1,220 +0,0 @@
-/*
- * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
- * or more contributor license agreements. Licensed under the Elastic License
- * 2.0; you may not use this file except in compliance with the Elastic License
- * 2.0.
- */
-
-import './field_list.scss';
-import { partition, throttle } from 'lodash';
-import React, { useState, Fragment, useCallback, useMemo, useEffect } from 'react';
-import { EuiSpacer } from '@elastic/eui';
-import { UiActionsStart } from '@kbn/ui-actions-plugin/public';
-import { FieldItem } from './field_item';
-import { NoFieldsCallout } from './no_fields_callout';
-import { FieldItemSharedProps, FieldsAccordion } from './fields_accordion';
-import type { DatasourceDataPanelProps, IndexPatternField } from '../../types';
-const PAGINATION_SIZE = 50;
-
-export type FieldGroups = Record<
-  string,
-  {
-    fields: IndexPatternField[];
-    fieldCount: number;
-    showInAccordion: boolean;
-    isInitiallyOpen: boolean;
-    title: string;
-    helpText?: string;
-    isAffectedByGlobalFilter: boolean;
-    isAffectedByTimeFilter: boolean;
-    hideDetails?: boolean;
-    defaultNoFieldsMessage?: string;
-    hideIfEmpty?: boolean;
-  }
->;
-
-function getDisplayedFieldsLength(
-  fieldGroups: FieldGroups,
-  accordionState: Partial<Record<string, boolean>>
-) {
-  return Object.entries(fieldGroups)
-    .filter(([key]) => accordionState[key])
-    .reduce((allFieldCount, [, { fields }]) => allFieldCount + fields.length, 0);
-}
-
-export const FieldList = React.memo(function FieldList({
-  exists,
-  fieldGroups,
-  existenceFetchFailed,
-  existenceFetchTimeout,
-  fieldProps,
-  hasSyncedExistingFields,
-  filter,
-  currentIndexPatternId,
-  existFieldsInIndex,
-  dropOntoWorkspace,
-  hasSuggestionForField,
-  editField,
-  removeField,
-  uiActions,
-}: {
-  exists: (field: IndexPatternField) => boolean;
-  fieldGroups: FieldGroups;
-  fieldProps: FieldItemSharedProps;
-  hasSyncedExistingFields: boolean;
-  existenceFetchFailed?: boolean;
-  existenceFetchTimeout?: boolean;
-  filter: {
-    nameFilter: string;
-    typeFilter: string[];
-  };
-  currentIndexPatternId: string;
-  existFieldsInIndex: boolean;
-  dropOntoWorkspace: DatasourceDataPanelProps['dropOntoWorkspace'];
-  hasSuggestionForField: DatasourceDataPanelProps['hasSuggestionForField'];
-  editField?: (name: string) => void;
-  removeField?: (name: string) => void;
-  uiActions: UiActionsStart;
-}) {
-  const [fieldGroupsToShow, fieldFroupsToCollapse] = partition(
-    Object.entries(fieldGroups),
-    ([, { showInAccordion }]) => showInAccordion
-  );
-  const [pageSize, setPageSize] = useState(PAGINATION_SIZE);
-  const [scrollContainer, setScrollContainer] = useState<Element | undefined>(undefined);
-  const [accordionState, setAccordionState] = useState<Partial<Record<string, boolean>>>(() =>
-    Object.fromEntries(
-      fieldGroupsToShow.map(([key, { isInitiallyOpen }]) => [key, isInitiallyOpen])
-    )
-  );
-
-  useEffect(() => {
-    // Reset the scroll if we have made material changes to the field list
-    if (scrollContainer) {
-      scrollContainer.scrollTop = 0;
-      setPageSize(PAGINATION_SIZE);
-    }
-  }, [filter.nameFilter, filter.typeFilter, currentIndexPatternId, scrollContainer]);
-
-  const lazyScroll = useCallback(() => {
-    if (scrollContainer) {
-      const nearBottom =
-        scrollContainer.scrollTop + scrollContainer.clientHeight >
-        scrollContainer.scrollHeight * 0.9;
-      if (nearBottom) {
-        setPageSize(
-          Math.max(
-            PAGINATION_SIZE,
-            Math.min(
-              pageSize + PAGINATION_SIZE * 0.5,
-              getDisplayedFieldsLength(fieldGroups, accordionState)
-            )
-          )
-        );
-      }
-    }
-  }, [scrollContainer, pageSize, setPageSize, fieldGroups, accordionState]);
-
-  const paginatedFields = useMemo(() => {
-    let remainingItems = pageSize;
-    return Object.fromEntries(
-      fieldGroupsToShow.map(([key, fieldGroup]) => {
-        if (!accordionState[key] || remainingItems <= 0) {
-          return [key, []];
-        }
-        const slicedFieldList = fieldGroup.fields.slice(0, remainingItems);
-        remainingItems = remainingItems - slicedFieldList.length;
-        return [key, slicedFieldList];
-      })
-    );
-  }, [pageSize, fieldGroupsToShow, accordionState]);
-
-  return (
-    <div
-      className="lnsIndexPatternFieldList"
-      ref={(el) => {
-        if (el && !el.dataset.dynamicScroll) {
-          el.dataset.dynamicScroll = 'true';
-          setScrollContainer(el);
-        }
-      }}
-      onScroll={throttle(lazyScroll, 100)}
-    >
-      <div className="lnsIndexPatternFieldList__accordionContainer">
-        <ul>
-          {fieldFroupsToCollapse.flatMap(([, { fields }]) =>
-            fields.map((field, index) => (
-              <FieldItem
-                {...fieldProps}
-                exists={exists(field)}
-                field={field}
-                editField={editField}
-                removeField={removeField}
-                hideDetails={true}
-                key={field.name}
-                itemIndex={index}
-                groupIndex={0}
-                dropOntoWorkspace={dropOntoWorkspace}
-                hasSuggestionForField={hasSuggestionForField}
-                uiActions={uiActions}
-              />
-            ))
-          )}
-        </ul>
-        <EuiSpacer size="s" />
-        {fieldGroupsToShow.map(([key, fieldGroup], index) => {
-          if (Boolean(fieldGroup.hideIfEmpty) && !fieldGroup.fields.length) return null;
-          return (
-            <Fragment key={key}>
-              <FieldsAccordion
-                dropOntoWorkspace={dropOntoWorkspace}
-                hasSuggestionForField={hasSuggestionForField}
-                initialIsOpen={Boolean(accordionState[key])}
-                key={key}
-                id={`lnsIndexPattern${key}`}
-                label={fieldGroup.title}
-                helpTooltip={fieldGroup.helpText}
-                exists={exists}
-                editField={editField}
-                removeField={removeField}
-                hideDetails={fieldGroup.hideDetails}
-                hasLoaded={!!hasSyncedExistingFields}
-                fieldsCount={fieldGroup.fields.length}
-                isFiltered={fieldGroup.fieldCount !== fieldGroup.fields.length}
-                paginatedFields={paginatedFields[key]}
-                fieldProps={fieldProps}
-                groupIndex={index + 1}
-                onToggle={(open) => {
-                  setAccordionState((s) => ({
-                    ...s,
-                    [key]: open,
-                  }));
-                  const displayedFieldLength = getDisplayedFieldsLength(fieldGroups, {
-                    ...accordionState,
-                    [key]: open,
-                  });
-                  setPageSize(
-                    Math.max(PAGINATION_SIZE, Math.min(pageSize * 1.5, displayedFieldLength))
-                  );
-                }}
-                showExistenceFetchError={existenceFetchFailed}
-                showExistenceFetchTimeout={existenceFetchTimeout}
-                renderCallout={
-                  <NoFieldsCallout
-                    isAffectedByGlobalFilter={fieldGroup.isAffectedByGlobalFilter}
-                    isAffectedByTimerange={fieldGroup.isAffectedByTimeFilter}
-                    isAffectedByFieldFilter={fieldGroup.fieldCount !== fieldGroup.fields.length}
-                    existFieldsInIndex={!!existFieldsInIndex}
-                    defaultNoFieldsMessage={fieldGroup.defaultNoFieldsMessage}
-                  />
-                }
-                uiActions={uiActions}
-              />
-              <EuiSpacer size="m" />
-            </Fragment>
-          );
-        })}
-      </div>
-    </div>
-  );
-});

x-pack/plugins/lens/public/datasources/form_based/fields_accordion.test.tsx

@@ -1,110 +0,0 @@
-/*
- * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
- * or more contributor license agreements. Licensed under the Elastic License
- * 2.0; you may not use this file except in compliance with the Elastic License
- * 2.0.
- */
-
-import React from 'react';
-import { EuiLoadingSpinner, EuiNotificationBadge } from '@elastic/eui';
-import { coreMock } from '@kbn/core/public/mocks';
-import { mountWithIntl, shallowWithIntl } from '@kbn/test-jest-helpers';
-import { fieldFormatsServiceMock } from '@kbn/field-formats-plugin/public/mocks';
-import { IndexPattern } from '../../types';
-import { FieldItem } from './field_item';
-import { FieldsAccordion, FieldsAccordionProps, FieldItemSharedProps } from './fields_accordion';
-import { chartPluginMock } from '@kbn/charts-plugin/public/mocks';
-import { uiActionsPluginMock } from '@kbn/ui-actions-plugin/public/mocks';
-
-describe('Fields Accordion', () => {
-  let defaultProps: FieldsAccordionProps;
-  let indexPattern: IndexPattern;
-  let core: ReturnType<typeof coreMock['createStart']>;
-  let fieldProps: FieldItemSharedProps;
-
-  beforeEach(() => {
-    indexPattern = {
-      id: '1',
-      title: 'my-fake-index-pattern',
-      timeFieldName: 'timestamp',
-      fields: [
-        {
-          name: 'timestamp',
-          type: 'date',
-          aggregatable: true,
-          searchable: true,
-        },
-        {
-          name: 'bytes',
-          type: 'number',
-          aggregatable: true,
-          searchable: true,
-        },
-      ],
-    } as IndexPattern;
-    core = coreMock.createStart();
-    core.http.post.mockClear();
-
-    fieldProps = {
-      indexPattern,
-      fieldFormats: fieldFormatsServiceMock.createStartContract(),
-      core,
-      highlight: '',
-      dateRange: {
-        fromDate: 'now-7d',
-        toDate: 'now',
-      },
-      query: { query: '', language: 'lucene' },
-      filters: [],
-      chartsThemeService: chartPluginMock.createSetupContract().theme,
-    };
-
-    defaultProps = {
-      initialIsOpen: true,
-      onToggle: jest.fn(),
-      id: 'id',
-      label: 'label',
-      hasLoaded: true,
-      fieldsCount: 2,
-      isFiltered: false,
-      paginatedFields: indexPattern.fields,
-      fieldProps,
-      renderCallout: <div id="lens-test-callout">Callout</div>,
-      exists: () => true,
-      groupIndex: 0,
-      dropOntoWorkspace: () => {},
-      hasSuggestionForField: () => false,
-      uiActions: uiActionsPluginMock.createStartContract(),
-    };
-  });
-
-  it('renders correct number of Field Items', () => {
-    const wrapper = mountWithIntl(
-      <FieldsAccordion {...defaultProps} exists={(field) => field.name === 'timestamp'} />
-    );
-    expect(wrapper.find(FieldItem).at(0).prop('exists')).toEqual(true);
-    expect(wrapper.find(FieldItem).at(1).prop('exists')).toEqual(false);
-  });
-
-  it('passed correct exists flag to each field', () => {
-    const wrapper = mountWithIntl(<FieldsAccordion {...defaultProps} />);
-    expect(wrapper.find(FieldItem).length).toEqual(2);
-  });
-
-  it('renders callout if no fields', () => {
-    const wrapper = shallowWithIntl(
-      <FieldsAccordion {...defaultProps} fieldsCount={0} paginatedFields={[]} />
-    );
-    expect(wrapper.find('#lens-test-callout').length).toEqual(1);
-  });
-
-  it('renders accented notificationBadge state if isFiltered', () => {
-    const wrapper = mountWithIntl(<FieldsAccordion {...defaultProps} isFiltered={true} />);
-    expect(wrapper.find(EuiNotificationBadge).prop('color')).toEqual('accent');
-  });
-
-  it('renders spinner if has not loaded', () => {
-    const wrapper = mountWithIntl(<FieldsAccordion {...defaultProps} hasLoaded={false} />);
-    expect(wrapper.find(EuiLoadingSpinner).length).toEqual(1);
-  });
-});

x-pack/plugins/lens/public/datasources/form_based/layerpanel.test.tsx

@@ -181,8 +181,6 @@ describe('Layer Data Panel', () => {
           { id: '2', title: 'my-fake-restricted-pattern' },
           { id: '3', title: 'my-compatible-pattern' },
         ],
-        existingFields: {},
-        isFirstExistenceFetch: false,
         indexPatterns: {
           '1': {
             id: '1',

x-pack/plugins/lens/public/datasources/form_based/operations/definitions/date_histogram.test.tsx

@@ -113,14 +113,6 @@ const defaultOptions = {
   isFullscreen: false,
   toggleFullscreen: jest.fn(),
   setIsCloseable: jest.fn(),
-  existingFields: {
-    my_index_pattern: {
-      timestamp: true,
-      bytes: true,
-      memory: true,
-      source: true,
-    },
-  },
 };
 
 describe('date_histogram', () => {

x-pack/plugins/lens/public/datasources/form_based/operations/definitions/filters/filters.test.tsx

@@ -38,14 +38,6 @@ const defaultProps = {
   toggleFullscreen: jest.fn(),
   setIsCloseable: jest.fn(),
   layerId: '1',
-  existingFields: {
-    my_index_pattern: {
-      timestamp: true,
-      bytes: true,
-      memory: true,
-      source: true,
-    },
-  },
 };
 
 // mocking random id generator function

x-pack/plugins/lens/public/datasources/form_based/operations/definitions/index.ts

@@ -198,7 +198,6 @@ export interface ParamEditorProps<
   activeData?: FormBasedDimensionEditorProps['activeData'];
   operationDefinitionMap: Record<string, GenericOperationDefinition>;
   paramEditorCustomProps?: ParamEditorCustomProps;
-  existingFields: Record<string, Record<string, boolean>>;
   isReferenced?: boolean;
 }
 
@@ -215,10 +214,6 @@ export interface FieldInputProps<C> {
   incompleteParams: Omit<IncompleteColumn, 'sourceField' | 'operationType'>;
   dimensionGroups: FormBasedDimensionEditorProps['dimensionGroups'];
   groupId: FormBasedDimensionEditorProps['groupId'];
-  /**
-   * indexPatternId -> fieldName -> boolean
-   */
-  existingFields: Record<string, Record<string, boolean>>;
   operationSupportMatrix: OperationSupportMatrix;
   helpMessage?: React.ReactNode;
   operationDefinitionMap: Record<string, GenericOperationDefinition>;

x-pack/plugins/lens/public/datasources/form_based/operations/definitions/last_value.test.tsx

@@ -44,14 +44,6 @@ const defaultProps = {
   toggleFullscreen: jest.fn(),
   setIsCloseable: jest.fn(),
   layerId: '1',
-  existingFields: {
-    my_index_pattern: {
-      timestamp: true,
-      bytes: true,
-      memory: true,
-      source: true,
-    },
-  },
 };
 
 describe('last_value', () => {

x-pack/plugins/lens/public/datasources/form_based/operations/definitions/percentile.test.tsx

@@ -60,14 +60,6 @@ const defaultProps = {
   toggleFullscreen: jest.fn(),
   setIsCloseable: jest.fn(),
   layerId: '1',
-  existingFields: {
-    my_index_pattern: {
-      timestamp: true,
-      bytes: true,
-      memory: true,
-      source: true,
-    },
-  },
 };
 
 describe('percentile', () => {

x-pack/plugins/lens/public/datasources/form_based/operations/definitions/percentile_ranks.test.tsx

@@ -53,14 +53,6 @@ const defaultProps = {
   toggleFullscreen: jest.fn(),
   setIsCloseable: jest.fn(),
   layerId: '1',
-  existingFields: {
-    my_index_pattern: {
-      timestamp: true,
-      bytes: true,
-      memory: true,
-      source: true,
-    },
-  },
 };
 
 describe('percentile ranks', () => {

x-pack/plugins/lens/public/datasources/form_based/operations/definitions/ranges/ranges.test.tsx

@@ -86,14 +86,6 @@ const defaultOptions = {
   storage: {} as IStorageWrapper,
   uiSettings: uiSettingsMock,
   savedObjectsClient: {} as SavedObjectsClientContract,
-  existingFields: {
-    my_index_pattern: {
-      timestamp: true,
-      bytes: true,
-      memory: true,
-      source: true,
-    },
-  },
   dateRange: {
     fromDate: 'now-1y',
     toDate: 'now',

x-pack/plugins/lens/public/datasources/form_based/operations/definitions/static_value.test.tsx

@@ -52,14 +52,6 @@ const defaultProps = {
   toggleFullscreen: jest.fn(),
   setIsCloseable: jest.fn(),
   layerId: '1',
-  existingFields: {
-    my_index_pattern: {
-      timestamp: true,
-      bytes: true,
-      memory: true,
-      source: true,
-    },
-  },
 };
 
 describe('static_value', () => {

x-pack/plugins/lens/public/datasources/form_based/operations/definitions/terms/field_inputs.tsx

@@ -8,7 +8,7 @@
 import React, { useCallback, useMemo } from 'react';
 import { htmlIdGenerator } from '@elastic/eui';
 import { i18n } from '@kbn/i18n';
-import { ExistingFieldsMap, IndexPattern } from '../../../../../types';
+import { IndexPattern } from '../../../../../types';
 import {
   DragDropBuckets,
   FieldsBucketContainer,
@@ -27,7 +27,6 @@ export const MAX_MULTI_FIELDS_SIZE = 3;
 export interface FieldInputsProps {
   column: TermsIndexPatternColumn;
   indexPattern: IndexPattern;
-  existingFields: ExistingFieldsMap;
   invalidFields?: string[];
   operationSupportMatrix: Pick<OperationSupportMatrix, 'operationByField'>;
   onChange: (newValues: string[]) => void;
@@ -49,7 +48,6 @@ export function FieldInputs({
   column,
   onChange,
   indexPattern,
-  existingFields,
   operationSupportMatrix,
   invalidFields,
 }: FieldInputsProps) {
@@ -153,7 +151,6 @@ export function FieldInputs({
               <FieldSelect
                 fieldIsInvalid={shouldShowError}
                 currentIndexPattern={indexPattern}
-                existingFields={existingFields[indexPattern.title]}
                 operationByField={filteredOperationByField}
                 selectedOperationType={column.operationType}
                 selectedField={value}

x-pack/plugins/lens/public/datasources/form_based/operations/definitions/terms/index.tsx

@@ -426,7 +426,6 @@ export const termsOperation: OperationDefinition<
       selectedColumn,
       columnId,
       indexPattern,
-      existingFields,
       operationSupportMatrix,
       updateLayer,
       dimensionGroups,
@@ -549,7 +548,6 @@ export const termsOperation: OperationDefinition<
         <FieldInputs
           column={selectedColumn}
           indexPattern={indexPattern}
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           onChange={onFieldSelectChange}
           invalidFields={invalidFields}
@@ -568,7 +566,6 @@ The top values of a specified field ranked by the chosen metric.
     currentColumn,
     columnId,
     indexPattern,
-    existingFields,
     operationDefinitionMap,
     ReferenceEditor,
     paramEditorCustomProps,
@@ -808,7 +805,6 @@ The top values of a specified field ranked by the chosen metric.
               }}
               column={currentColumn.params.orderAgg}
               incompleteColumn={incompleteColumn}
-              existingFields={existingFields}
               onDeleteColumn={() => {
                 throw new Error('Should not be called');
               }}

x-pack/plugins/lens/public/datasources/form_based/operations/definitions/terms/terms.test.tsx

@@ -50,6 +50,16 @@ jest.mock('@kbn/unified-field-list-plugin/public/services/field_stats', () => ({
   }),
 }));
 
+jest.mock('@kbn/unified-field-list-plugin/public/hooks/use_existing_fields', () => ({
+  useExistingFieldsReader: jest.fn(() => {
+    return {
+      hasFieldData: (dataViewId: string, fieldName: string) => {
+        return ['timestamp', 'bytes', 'memory', 'source'].includes(fieldName);
+      },
+    };
+  }),
+}));
+
 // mocking random id generator function
 jest.mock('@elastic/eui', () => {
   const original = jest.requireActual('@elastic/eui');
@@ -93,14 +103,6 @@ const defaultProps = {
   setIsCloseable: jest.fn(),
   layerId: '1',
   ReferenceEditor,
-  existingFields: {
-    'my-fake-index-pattern': {
-      timestamp: true,
-      bytes: true,
-      memory: true,
-      source: true,
-    },
-  },
 };
 
 describe('terms', () => {
@@ -1170,20 +1172,7 @@ describe('terms', () => {
       >,
     };
 
-    function getExistingFields() {
-      const fields: Record<string, boolean> = {};
-      for (const field of defaultProps.indexPattern.fields) {
-        fields[field.name] = true;
-      }
-      return {
-        [defaultProps.indexPattern.title]: fields,
-      };
-    }
-
-    function getDefaultOperationSupportMatrix(
-      columnId: string,
-      existingFields: Record<string, Record<string, boolean>>
-    ) {
+    function getDefaultOperationSupportMatrix(columnId: string) {
       return getOperationSupportMatrix({
         state: {
           layers: { layer1: layer },
@@ -1199,15 +1188,13 @@ describe('terms', () => {
 
     it('should render the default field input for no field (incomplete operation)', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
       const instance = mount(
         <InlineFieldInput
           {...defaultFieldInputProps}
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           incompleteOperation="terms"
         />
@@ -1226,8 +1213,7 @@ describe('terms', () => {
 
     it('should show an error message when first field is invalid', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
 
       layer.columns.col1 = {
         label: 'Top value of unsupported',
@@ -1247,7 +1233,6 @@ describe('terms', () => {
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           currentFieldIsInvalid
         />
@@ -1259,8 +1244,7 @@ describe('terms', () => {
 
     it('should show an error message when first field is not supported', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
 
       layer.columns.col1 = {
         label: 'Top value of timestamp',
@@ -1280,7 +1264,6 @@ describe('terms', () => {
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           selectedColumn={layer.columns.col1 as TermsIndexPatternColumn}
           incompleteOperation="terms"
@@ -1293,8 +1276,7 @@ describe('terms', () => {
 
     it('should show an error message when any field but the first is invalid', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
 
       layer.columns.col1 = {
         label: 'Top value of geo.src + 1 other',
@@ -1315,7 +1297,6 @@ describe('terms', () => {
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           selectedColumn={layer.columns.col1 as TermsIndexPatternColumn}
         />
@@ -1327,8 +1308,7 @@ describe('terms', () => {
 
     it('should show an error message when any field but the first is not supported', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
 
       layer.columns.col1 = {
         label: 'Top value of geo.src + 1 other',
@@ -1349,7 +1329,6 @@ describe('terms', () => {
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           selectedColumn={layer.columns.col1 as TermsIndexPatternColumn}
         />
@@ -1361,15 +1340,13 @@ describe('terms', () => {
 
     it('should render the an add button for single layer and disabled the remove button', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
       const instance = mount(
         <InlineFieldInput
           {...defaultFieldInputProps}
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           selectedColumn={layer.columns.col1 as TermsIndexPatternColumn}
         />
@@ -1392,15 +1369,13 @@ describe('terms', () => {
 
     it('should switch to the first supported operation when in single term mode and the picked field is not supported', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
       const instance = mount(
         <InlineFieldInput
           {...defaultFieldInputProps}
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           selectedColumn={layer.columns.col1 as TermsIndexPatternColumn}
         />
@@ -1426,8 +1401,7 @@ describe('terms', () => {
 
     it('should render the multi terms specific UI', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
 
       (layer.columns.col1 as TermsIndexPatternColumn).params.secondaryFields = ['bytes'];
       const instance = mount(
@@ -1436,7 +1410,6 @@ describe('terms', () => {
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           selectedColumn={layer.columns.col1 as TermsIndexPatternColumn}
         />
@@ -1457,8 +1430,7 @@ describe('terms', () => {
 
     it('should return to single value UI when removing second item of two', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
 
       (layer.columns.col1 as TermsIndexPatternColumn).params.secondaryFields = ['memory'];
       const instance = mount(
@@ -1467,7 +1439,6 @@ describe('terms', () => {
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           selectedColumn={layer.columns.col1 as TermsIndexPatternColumn}
         />
@@ -1489,8 +1460,7 @@ describe('terms', () => {
 
     it('should disable remove button and reorder drag when single value and one temporary new field', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
 
       let instance = mount(
         <InlineFieldInput
@@ -1498,7 +1468,6 @@ describe('terms', () => {
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           selectedColumn={layer.columns.col1 as TermsIndexPatternColumn}
         />
@@ -1532,8 +1501,7 @@ describe('terms', () => {
 
     it('should accept scripted fields for single value', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
 
       (layer.columns.col1 as TermsIndexPatternColumn).sourceField = 'scripted';
       const instance = mount(
@@ -1542,7 +1510,6 @@ describe('terms', () => {
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           selectedColumn={layer.columns.col1 as TermsIndexPatternColumn}
         />
@@ -1558,8 +1525,7 @@ describe('terms', () => {
 
     it('should mark scripted fields for multiple values', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
 
       (layer.columns.col1 as TermsIndexPatternColumn).sourceField = 'scripted';
       (layer.columns.col1 as TermsIndexPatternColumn).params.secondaryFields = ['memory'];
@@ -1569,7 +1535,6 @@ describe('terms', () => {
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           selectedColumn={layer.columns.col1 as TermsIndexPatternColumn}
         />
@@ -1588,8 +1553,7 @@ describe('terms', () => {
 
     it('should not filter scripted fields when in single value', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
 
       const instance = mount(
         <InlineFieldInput
@@ -1597,7 +1561,6 @@ describe('terms', () => {
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           selectedColumn={layer.columns.col1 as TermsIndexPatternColumn}
         />
@@ -1618,8 +1581,7 @@ describe('terms', () => {
 
     it('should filter scripted fields when in multi terms mode', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
 
       (layer.columns.col1 as TermsIndexPatternColumn).params.secondaryFields = ['memory'];
       const instance = mount(
@@ -1628,7 +1590,6 @@ describe('terms', () => {
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           selectedColumn={layer.columns.col1 as TermsIndexPatternColumn}
         />
@@ -1650,8 +1611,7 @@ describe('terms', () => {
 
     it('should filter already used fields when displaying fields list', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
 
       (layer.columns.col1 as TermsIndexPatternColumn).params.secondaryFields = ['memory', 'bytes'];
       let instance = mount(
@@ -1660,7 +1620,6 @@ describe('terms', () => {
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           selectedColumn={layer.columns.col1 as TermsIndexPatternColumn}
         />
@@ -1690,8 +1649,7 @@ describe('terms', () => {
 
     it('should filter fields with unsupported types when in multi terms mode', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
 
       (layer.columns.col1 as TermsIndexPatternColumn).params.secondaryFields = ['memory'];
       const instance = mount(
@@ -1700,7 +1658,6 @@ describe('terms', () => {
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           selectedColumn={layer.columns.col1 as TermsIndexPatternColumn}
         />
@@ -1722,8 +1679,7 @@ describe('terms', () => {
 
     it('should limit the number of multiple fields', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
 
       (layer.columns.col1 as TermsIndexPatternColumn).params.secondaryFields = [
         'memory',
@@ -1736,7 +1692,6 @@ describe('terms', () => {
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           selectedColumn={layer.columns.col1 as TermsIndexPatternColumn}
         />
@@ -1757,8 +1712,7 @@ describe('terms', () => {
 
     it('should let the user add new empty field up to the limit', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
 
       let instance = mount(
         <InlineFieldInput
@@ -1766,7 +1720,6 @@ describe('terms', () => {
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           selectedColumn={layer.columns.col1 as TermsIndexPatternColumn}
         />
@@ -1793,8 +1746,7 @@ describe('terms', () => {
 
     it('should update the parentFormatter on transition between single to multi terms', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
 
       let instance = mount(
         <InlineFieldInput
@@ -1802,7 +1754,6 @@ describe('terms', () => {
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           selectedColumn={layer.columns.col1 as TermsIndexPatternColumn}
         />
@@ -1834,8 +1785,7 @@ describe('terms', () => {
 
     it('should preserve custom label when set by the user', () => {
       const updateLayerSpy = jest.fn();
-      const existingFields = getExistingFields();
-      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1', existingFields);
+      const operationSupportMatrix = getDefaultOperationSupportMatrix('col1');
 
       layer.columns.col1 = {
         label: 'MyCustomLabel',
@@ -1857,7 +1807,6 @@ describe('terms', () => {
           layer={layer}
           updateLayer={updateLayerSpy}
           columnId="col1"
-          existingFields={existingFields}
           operationSupportMatrix={operationSupportMatrix}
           selectedColumn={layer.columns.col1 as TermsIndexPatternColumn}
         />

x-pack/plugins/lens/public/datasources/text_based/datapanel.test.tsx

@@ -8,6 +8,7 @@
 import React from 'react';
 import ReactDOM from 'react-dom';
 import { act } from 'react-dom/test-utils';
+import { ReactWrapper } from 'enzyme';
 import type { Query } from '@kbn/es-query';
 
 import { dataPluginMock } from '@kbn/data-plugin/public/mocks';
@@ -30,7 +31,6 @@ import { createIndexPatternServiceMock } from '../../mocks/data_views_service_mo
 import { createMockFramePublicAPI } from '../../mocks';
 import { createMockedDragDropContext } from './mocks';
 import { DataViewsState } from '../../state_management';
-import { ExistingFieldsMap, IndexPattern } from '../../types';
 
 const fieldsFromQuery = [
   {
@@ -101,18 +101,6 @@ const fieldsOne = [
   },
 ];
 
-function getExistingFields(indexPatterns: Record<string, IndexPattern>) {
-  const existingFields: ExistingFieldsMap = {};
-  for (const { title, fields } of Object.values(indexPatterns)) {
-    const fieldsMap: Record<string, boolean> = {};
-    for (const { displayName, name } of fields) {
-      fieldsMap[displayName ?? name] = true;
-    }
-    existingFields[title] = fieldsMap;
-  }
-  return existingFields;
-}
-
 const initialState: TextBasedPrivateState = {
   layers: {
     first: {
@@ -130,8 +118,41 @@ const initialState: TextBasedPrivateState = {
   fieldList: fieldsFromQuery,
 };
 
-function getFrameAPIMock({ indexPatterns, existingFields, ...rest }: Partial<DataViewsState> = {}) {
+function getFrameAPIMock({
+  indexPatterns,
+  ...rest
+}: Partial<DataViewsState> & { indexPatterns: DataViewsState['indexPatterns'] }) {
   const frameAPI = createMockFramePublicAPI();
+  return {
+    ...frameAPI,
+    dataViews: {
+      ...frameAPI.dataViews,
+      indexPatterns,
+      ...rest,
+    },
+  };
+}
+
+// @ts-expect-error Portal mocks are notoriously difficult to type
+ReactDOM.createPortal = jest.fn((element) => element);
+
+async function mountAndWaitForLazyModules(component: React.ReactElement): Promise<ReactWrapper> {
+  let inst: ReactWrapper;
+  await act(async () => {
+    inst = await mountWithIntl(component);
+    // wait for lazy modules
+    await new Promise((resolve) => setTimeout(resolve, 0));
+    inst.update();
+  });
+
+  await inst!.update();
+
+  return inst!;
+}
+
+describe('TextBased Query Languages Data Panel', () => {
+  let core: ReturnType<typeof coreMock['createStart']>;
+  let dataViews: DataViewPublicStart;
   const defaultIndexPatterns = {
     '1': {
       id: '1',
@@ -144,27 +165,10 @@ function getFrameAPIMock({ indexPatterns, existingFields, ...rest }: Partial<Dat
       spec: {},
     },
   };
-  return {
-    ...frameAPI,
-    dataViews: {
-      ...frameAPI.dataViews,
-      indexPatterns: indexPatterns ?? defaultIndexPatterns,
-      existingFields: existingFields ?? getExistingFields(indexPatterns ?? defaultIndexPatterns),
-      isFirstExistenceFetch: false,
-      ...rest,
-    },
-  };
-}
-
-// @ts-expect-error Portal mocks are notoriously difficult to type
-ReactDOM.createPortal = jest.fn((element) => element);
-
-describe('TextBased Query Languages Data Panel', () => {
-  let core: ReturnType<typeof coreMock['createStart']>;
-  let dataViews: DataViewPublicStart;
 
   let defaultProps: TextBasedDataPanelProps;
   const dataViewsMock = dataViewPluginMocks.createStartContract();
+
   beforeEach(() => {
     core = coreMock.createStart();
     dataViews = dataViewPluginMocks.createStartContract();
@@ -194,7 +198,7 @@ describe('TextBased Query Languages Data Panel', () => {
       hasSuggestionForField: jest.fn(() => false),
       uiActions: uiActionsPluginMock.createStartContract(),
       indexPatternService: createIndexPatternServiceMock({ core, dataViews }),
-      frame: getFrameAPIMock(),
+      frame: getFrameAPIMock({ indexPatterns: defaultIndexPatterns }),
       state: initialState,
       setState: jest.fn(),
       onChangeIndexPattern: jest.fn(),
@@ -202,23 +206,33 @@ describe('TextBased Query Languages Data Panel', () => {
   });
 
   it('should render a search box', async () => {
-    const wrapper = mountWithIntl(<TextBasedDataPanel {...defaultProps} />);
-    expect(wrapper.find('[data-test-subj="lnsTextBasedLangugesFieldSearch"]').length).toEqual(1);
+    const wrapper = await mountAndWaitForLazyModules(<TextBasedDataPanel {...defaultProps} />);
+
+    expect(wrapper.find('[data-test-subj="lnsTextBasedLanguagesFieldSearch"]').length).toEqual(1);
   });
 
   it('should list all supported fields in the pattern', async () => {
-    const wrapper = mountWithIntl(<TextBasedDataPanel {...defaultProps} />);
+    const wrapper = await mountAndWaitForLazyModules(<TextBasedDataPanel {...defaultProps} />);
+
     expect(
       wrapper
-        .find('[data-test-subj="lnsTextBasedLanguagesPanelFields"]')
+        .find('[data-test-subj="lnsTextBasedLanguagesAvailableFields"]')
         .find(FieldButton)
         .map((fieldItem) => fieldItem.prop('fieldName'))
-    ).toEqual(['timestamp', 'bytes', 'memory']);
+    ).toEqual(['bytes', 'memory', 'timestamp']);
+
+    expect(wrapper.find('[data-test-subj="lnsTextBasedLanguagesEmptyFields"]').exists()).toBe(
+      false
+    );
+    expect(wrapper.find('[data-test-subj="lnsTextBasedLanguagesMetaFields"]').exists()).toBe(false);
   });
 
   it('should not display the selected fields accordion if there are no fields displayed', async () => {
-    const wrapper = mountWithIntl(<TextBasedDataPanel {...defaultProps} />);
-    expect(wrapper.find('[data-test-subj="lnsSelectedFieldsTextBased"]').length).toEqual(0);
+    const wrapper = await mountAndWaitForLazyModules(<TextBasedDataPanel {...defaultProps} />);
+
+    expect(wrapper.find('[data-test-subj="lnsTextBasedLanguagesSelectedFields"]').length).toEqual(
+      0
+    );
   });
 
   it('should display the selected fields accordion if there are fields displayed', async () => {
@@ -226,13 +240,17 @@ describe('TextBased Query Languages Data Panel', () => {
       ...defaultProps,
       layerFields: ['memory'],
     };
-    const wrapper = mountWithIntl(<TextBasedDataPanel {...props} />);
-    expect(wrapper.find('[data-test-subj="lnsSelectedFieldsTextBased"]').length).not.toEqual(0);
+    const wrapper = await mountAndWaitForLazyModules(<TextBasedDataPanel {...props} />);
+
+    expect(
+      wrapper.find('[data-test-subj="lnsTextBasedLanguagesSelectedFields"]').length
+    ).not.toEqual(0);
   });
 
   it('should list all supported fields in the pattern that match the search input', async () => {
-    const wrapper = mountWithIntl(<TextBasedDataPanel {...defaultProps} />);
-    const searchBox = wrapper.find('[data-test-subj="lnsTextBasedLangugesFieldSearch"]');
+    const wrapper = await mountAndWaitForLazyModules(<TextBasedDataPanel {...defaultProps} />);
+
+    const searchBox = wrapper.find('[data-test-subj="lnsTextBasedLanguagesFieldSearch"]');
 
     act(() => {
       searchBox.prop('onChange')!({
@@ -240,10 +258,10 @@ describe('TextBased Query Languages Data Panel', () => {
       } as React.ChangeEvent<HTMLInputElement>);
     });
 
-    wrapper.update();
+    await wrapper.update();
     expect(
       wrapper
-        .find('[data-test-subj="lnsTextBasedLanguagesPanelFields"]')
+        .find('[data-test-subj="lnsTextBasedLanguagesAvailableFields"]')
         .find(FieldButton)
         .map((fieldItem) => fieldItem.prop('fieldName'))
     ).toEqual(['memory']);

x-pack/plugins/lens/public/datasources/text_based/datapanel.tsx

@@ -5,7 +5,7 @@
  * 2.0.
  */
 
-import React, { useState, useEffect, useMemo } from 'react';
+import React, { useCallback, useEffect, useState } from 'react';
 import { EuiFlexGroup, EuiFlexItem, EuiFormControlLayout, htmlIdGenerator } from '@elastic/eui';
 import { i18n } from '@kbn/i18n';
 import usePrevious from 'react-use/lib/usePrevious';
@@ -14,13 +14,22 @@ import { DataPublicPluginStart } from '@kbn/data-plugin/public';
 import type { DataViewsPublicPluginStart } from '@kbn/data-views-plugin/public';
 
 import { isOfAggregateQueryType } from '@kbn/es-query';
-import { ExpressionsStart } from '@kbn/expressions-plugin/public';
+import { DatatableColumn, ExpressionsStart } from '@kbn/expressions-plugin/public';
 import { KibanaContextProvider } from '@kbn/kibana-react-plugin/public';
+import {
+  ExistenceFetchStatus,
+  FieldListGrouped,
+  FieldListGroupedProps,
+  FieldsGroupNames,
+  useGroupedFields,
+} from '@kbn/unified-field-list-plugin/public';
+import { FieldButton } from '@kbn/react-field';
 import type { DatasourceDataPanelProps } from '../../types';
 import type { TextBasedPrivateState } from './types';
 import { getStateFromAggregateQuery } from './utils';
-import { ChildDragDropProvider } from '../../drag_drop';
-import { FieldsAccordion } from './fields_accordion';
+import { ChildDragDropProvider, DragDrop } from '../../drag_drop';
+import { DataType } from '../../types';
+import { LensFieldIcon } from '../../shared_components';
 
 export type TextBasedDataPanelProps = DatasourceDataPanelProps<TextBasedPrivateState> & {
   data: DataPublicPluginStart;
@@ -67,8 +76,16 @@ export function TextBasedDataPanel({
   }, [data, dataViews, expressions, prevQuery, query, setState, state]);
 
   const { fieldList } = state;
-  const filteredFields = useMemo(() => {
-    return fieldList.filter((field) => {
+
+  const onSelectedFieldFilter = useCallback(
+    (field: DatatableColumn): boolean => {
+      return Boolean(layerFields?.includes(field.name));
+    },
+    [layerFields]
+  );
+
+  const onFilterField = useCallback(
+    (field: DatatableColumn) => {
       if (
         localState.nameFilter &&
         !field.name.toLowerCase().includes(localState.nameFilter.toLowerCase())
@@ -76,9 +93,57 @@ export function TextBasedDataPanel({
         return false;
       }
       return true;
-    });
-  }, [fieldList, localState.nameFilter]);
-  const usedByLayersFields = fieldList.filter((field) => layerFields?.includes(field.name));
+    },
+    [localState]
+  );
+
+  const onOverrideFieldGroupDetails = useCallback((groupName) => {
+    if (groupName === FieldsGroupNames.AvailableFields) {
+      return {
+        helpText: i18n.translate('xpack.lens.indexPattern.allFieldsForTextBasedLabelHelp', {
+          defaultMessage:
+            'Drag and drop available fields to the workspace and create visualizations. To change the available fields, edit your query.',
+        }),
+      };
+    }
+  }, []);
+
+  const { fieldGroups } = useGroupedFields<DatatableColumn>({
+    dataViewId: null,
+    allFields: fieldList,
+    services: {
+      dataViews,
+    },
+    onFilterField,
+    onSelectedFieldFilter,
+    onOverrideFieldGroupDetails,
+  });
+
+  const renderFieldItem: FieldListGroupedProps<DatatableColumn>['renderFieldItem'] = useCallback(
+    ({ field, itemIndex, groupIndex, hideDetails }) => {
+      return (
+        <DragDrop
+          draggable
+          order={[itemIndex]}
+          value={{
+            field: field?.name,
+            id: field.id,
+            humanData: { label: field?.name },
+          }}
+          dataTestSubj={`lnsFieldListPanelField-${field.name}`}
+        >
+          <FieldButton
+            className={`lnsFieldItem lnsFieldItem--${field?.meta?.type}`}
+            isActive={false}
+            onClick={() => {}}
+            fieldIcon={<LensFieldIcon type={field?.meta.type as DataType} />}
+            fieldName={field?.name}
+          />
+        </DragDrop>
+      );
+    },
+    []
+  );
 
   return (
     <KibanaContextProvider
@@ -111,7 +176,7 @@ export function TextBasedDataPanel({
             >
               <input
                 className="euiFieldText euiFieldText--fullWidth lnsInnerIndexPatternDataPanel__textField"
-                data-test-subj="lnsTextBasedLangugesFieldSearch"
+                data-test-subj="lnsTextBasedLanguagesFieldSearch"
                 placeholder={i18n.translate('xpack.lens.indexPatterns.filterByNameLabel', {
                   defaultMessage: 'Search field names',
                   description: 'Search the list of fields in the data view for the provided text',
@@ -129,32 +194,16 @@ export function TextBasedDataPanel({
             </EuiFormControlLayout>
           </EuiFlexItem>
           <EuiFlexItem>
-            <div className="lnsIndexPatternFieldList">
-              <div className="lnsIndexPatternFieldList__accordionContainer">
-                {usedByLayersFields.length > 0 && (
-                  <FieldsAccordion
-                    initialIsOpen={true}
-                    hasLoaded={dataHasLoaded}
-                    isFiltered={Boolean(localState.nameFilter)}
-                    fields={usedByLayersFields}
-                    id="lnsSelectedFieldsTextBased"
-                    label={i18n.translate('xpack.lens.textBased.selectedFieldsLabel', {
-                      defaultMessage: 'Selected fields',
-                    })}
-                  />
-                )}
-                <FieldsAccordion
-                  initialIsOpen={true}
-                  hasLoaded={dataHasLoaded}
-                  isFiltered={Boolean(localState.nameFilter)}
-                  fields={filteredFields}
-                  id="lnsAvailableFieldsTextBased"
-                  label={i18n.translate('xpack.lens.textBased.availableFieldsLabel', {
-                    defaultMessage: 'Available fields',
-                  })}
-                />
-              </div>
-            </div>
+            <FieldListGrouped<DatatableColumn>
+              fieldGroups={fieldGroups}
+              fieldsExistenceStatus={
+                dataHasLoaded ? ExistenceFetchStatus.succeeded : ExistenceFetchStatus.unknown
+              }
+              fieldsExistInIndex={Boolean(fieldList.length)}
+              renderFieldItem={renderFieldItem}
+              screenReaderDescriptionForSearchInputId={fieldSearchDescriptionId}
+              data-test-subj="lnsTextBasedLanguages"
+            />
           </EuiFlexItem>
         </EuiFlexGroup>
       </ChildDragDropProvider>

x-pack/plugins/lens/public/datasources/text_based/fields_accordion.tsx

@@ -1,106 +0,0 @@
-/*
- * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
- * or more contributor license agreements. Licensed under the Elastic License
- * 2.0; you may not use this file except in compliance with the Elastic License
- * 2.0.
- */
-
-import React, { memo, useMemo } from 'react';
-import type { DatatableColumn } from '@kbn/expressions-plugin/public';
-
-import {
-  EuiText,
-  EuiNotificationBadge,
-  EuiAccordion,
-  EuiLoadingSpinner,
-  EuiSpacer,
-} from '@elastic/eui';
-import { FieldButton } from '@kbn/react-field';
-import { DragDrop } from '../../drag_drop';
-import { LensFieldIcon } from '../../shared_components';
-import type { DataType } from '../../types';
-
-export interface FieldsAccordionProps {
-  initialIsOpen: boolean;
-  hasLoaded: boolean;
-  isFiltered: boolean;
-  // forceState: 'open' | 'closed';
-  id: string;
-  label: string;
-  fields: DatatableColumn[];
-}
-
-export const FieldsAccordion = memo(function InnerFieldsAccordion({
-  initialIsOpen,
-  hasLoaded,
-  isFiltered,
-  id,
-  label,
-  fields,
-}: FieldsAccordionProps) {
-  const renderButton = useMemo(() => {
-    return (
-      <EuiText size="xs">
-        <strong>{label}</strong>
-      </EuiText>
-    );
-  }, [label]);
-
-  const extraAction = useMemo(() => {
-    if (hasLoaded) {
-      return (
-        <EuiNotificationBadge
-          size="m"
-          color={isFiltered ? 'accent' : 'subdued'}
-          data-test-subj={`${id}-count`}
-        >
-          {fields.length}
-        </EuiNotificationBadge>
-      );
-    }
-
-    return <EuiLoadingSpinner size="m" />;
-  }, [fields.length, hasLoaded, id, isFiltered]);
-
-  return (
-    <>
-      <EuiAccordion
-        initialIsOpen={initialIsOpen}
-        id={id}
-        buttonContent={renderButton}
-        extraAction={extraAction}
-        data-test-subj={id}
-      >
-        <ul
-          className="lnsInnerIndexPatternDataPanel__fieldItems"
-          data-test-subj="lnsTextBasedLanguagesPanelFields"
-        >
-          {fields.length > 0 &&
-            fields.map((field, index) => (
-              <li key={field?.name}>
-                <DragDrop
-                  draggable
-                  order={[index]}
-                  value={{
-                    field: field?.name,
-                    id: field.id,
-                    humanData: { label: field?.name },
-                  }}
-                  dataTestSubj={`lnsFieldListPanelField-${field.name}`}
-                >
-                  <FieldButton
-                    className={`lnsFieldItem lnsFieldItem--${field?.meta?.type}`}
-                    isActive={false}
-                    onClick={() => {}}
-                    fieldIcon={<LensFieldIcon type={field?.meta.type as DataType} />}
-                    fieldName={field?.name}
-                  />
-                </DragDrop>
-              </li>
-            ))}
-        </ul>
-      </EuiAccordion>
-      <EuiSpacer size="m" />
-    </>
-  );
-});

x-pack/plugins/lens/public/datasources/text_based/layerpanel.test.tsx

@@ -68,8 +68,6 @@ describe('Layer Data Panel', () => {
           { id: '2', title: 'my-fake-restricted-pattern', name: 'my-fake-restricted-pattern' },
           { id: '3', title: 'my-compatible-pattern', name: 'my-compatible-pattern' },
         ],
-        existingFields: {},
-        isFirstExistenceFetch: false,
         indexPatterns: {},
       } as DataViewsState,
     };

x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/layer_panel.tsx

@@ -568,7 +568,6 @@ export function LayerPanel(
                                       invalid: group.invalid,
                                       invalidMessage: group.invalidMessage,
                                       indexPatterns: dataViews.indexPatterns,
-                                      existingFields: dataViews.existingFields,
                                     }}
                                   />
                                 ) : (
@@ -728,7 +727,6 @@ export function LayerPanel(
                   formatSelectorOptions: activeGroup.formatSelectorOptions,
                   layerType: activeVisualization.getLayerType(layerId, visualizationState),
                   indexPatterns: dataViews.indexPatterns,
-                  existingFields: dataViews.existingFields,
                   activeData: layerVisualizationConfigProps.activeData,
                 }}
               />

x-pack/plugins/lens/public/mocks/store_mocks.tsx

@@ -59,8 +59,6 @@ export const defaultState = {
   dataViews: {
     indexPatterns: {},
     indexPatternRefs: [],
-    existingFields: {},
-    isFirstExistenceFetch: false,
   },
 };
 

x-pack/plugins/lens/public/shared_components/dataview_picker/helpers.ts

@@ -5,25 +5,17 @@
  * 2.0.
  */
 
+import { type ExistingFieldsReader } from '@kbn/unified-field-list-plugin/public';
 import { IndexPattern } from '../../types';
 
 /**
  * Checks if the provided field contains data (works for meta field)
  */
 export function fieldContainsData(
-  field: string,
+  fieldName: string,
   indexPattern: IndexPattern,
-  existingFields: Record<string, boolean>
+  hasFieldData: ExistingFieldsReader['hasFieldData']
 ) {
-  return (
-    indexPattern.getFieldByName(field)?.type === 'document' || fieldExists(existingFields, field)
-  );
-}
-
-/**
- * Performs an existence check on the existingFields data structure for the provided field.
- * Does not work for meta fields.
- */
-export function fieldExists(existingFields: Record<string, boolean>, fieldName: string) {
-  return existingFields[fieldName];
+  const field = indexPattern.getFieldByName(fieldName);
+  return field?.type === 'document' || hasFieldData(indexPattern.id, fieldName);
 }

x-pack/plugins/lens/public/shared_components/dataview_picker/index.ts

@@ -6,4 +6,4 @@
  */
 
 export { ChangeIndexPattern } from './dataview_picker';
-export { fieldExists, fieldContainsData } from './helpers';
+export { fieldContainsData } from './helpers';

x-pack/plugins/lens/public/shared_components/index.ts

@@ -11,7 +11,7 @@ export { LegendSettingsPopover } from './legend_settings_popover';
 export { PalettePicker } from './palette_picker';
 export { FieldPicker, LensFieldIcon, TruncatedLabel } from './field_picker';
 export type { FieldOption, FieldOptionValue } from './field_picker';
-export { ChangeIndexPattern, fieldExists, fieldContainsData } from './dataview_picker';
+export { ChangeIndexPattern, fieldContainsData } from './dataview_picker';
 export { QueryInput, isQueryValid, validateQuery } from './query_input';
 export {
   NewBucketButton,

x-pack/plugins/lens/public/state_management/__snapshots__/load_initial.test.tsx.snap

@@ -5,10 +5,8 @@ Object {
   "lens": Object {
     "activeDatasourceId": "testDatasource",
     "dataViews": Object {
-      "existingFields": Object {},
       "indexPatternRefs": Array [],
       "indexPatterns": Object {},
-      "isFirstExistenceFetch": true,
     },
     "datasourceStates": Object {
       "testDatasource": Object {

x-pack/plugins/lens/public/state_management/lens_slice.ts

@@ -50,8 +50,6 @@ export const initialState: LensAppState = {
   dataViews: {
     indexPatternRefs: [],
     indexPatterns: {},
-    existingFields: {},
-    isFirstExistenceFetch: true,
   },
 };
 

x-pack/plugins/lens/public/state_management/types.ts

@@ -30,10 +30,6 @@ export interface VisualizationState {
 export interface DataViewsState {
   indexPatternRefs: IndexPatternRef[];
   indexPatterns: Record<string, IndexPattern>;
-  existingFields: Record<string, Record<string, boolean>>;
-  isFirstExistenceFetch: boolean;
-  existenceFetchFailed?: boolean;
-  existenceFetchTimeout?: boolean;
 }
 
 export type DatasourceStates = Record<string, { isLoading: boolean; state: unknown }>;

x-pack/plugins/lens/public/types.ts

@@ -108,7 +108,6 @@ export interface EditorFrameProps {
 export type VisualizationMap = Record<string, Visualization>;
 export type DatasourceMap = Record<string, Datasource>;
 export type IndexPatternMap = Record<string, IndexPattern>;
-export type ExistingFieldsMap = Record<string, Record<string, boolean>>;
 
 export interface EditorFrameInstance {
   EditorFrameContainer: (props: EditorFrameProps) => React.ReactElement;
@@ -589,7 +588,6 @@ export type DatasourceDimensionProps<T> = SharedDimensionProps & {
   state: T;
   activeData?: Record<string, Datatable>;
   indexPatterns: IndexPatternMap;
-  existingFields: Record<string, Record<string, boolean>>;
   hideTooltip?: boolean;
   invalid?: boolean;
   invalidMessage?: string;

x-pack/plugins/lens/public/utils.ts

@@ -80,8 +80,6 @@ export function getInitialDataViewsObject(
   return {
     indexPatterns,
     indexPatternRefs,
-    existingFields: {},
-    isFirstExistenceFetch: true,
   };
 }
 
@@ -107,9 +105,6 @@ export async function refreshIndexPatternsList({
     onIndexPatternRefresh: () => onRefreshCallbacks.forEach((fn) => fn()),
   });
   const indexPattern = newlyMappedIndexPattern[indexPatternId];
-  // But what about existingFields here?
-  // When the indexPatterns cache object gets updated, the data panel will
-  // notice it and refetch the fields list existence map
   indexPatternService.updateDataViewsState({
     indexPatterns: {
       ...indexPatternsCache,

x-pack/plugins/lens/public/visualizations/xy/xy_config_panel/annotations_config_panel/annotations_panel.tsx

@@ -23,6 +23,7 @@ import {
   QueryPointEventAnnotationConfig,
 } from '@kbn/event-annotation-plugin/common';
 import moment from 'moment';
+import { useExistingFieldsReader } from '@kbn/unified-field-list-plugin/public';
 import {
   FieldOption,
   FieldOptionValue,
@@ -31,7 +32,6 @@ import {
 import { FormatFactory } from '../../../../../common';
 import {
   DimensionEditorSection,
-  fieldExists,
   NameInput,
   useDebouncedValue,
 } from '../../../../shared_components';
@@ -58,6 +58,7 @@ export const AnnotationsPanel = (
 ) => {
   const { state, setState, layerId, accessor, frame } = props;
   const isHorizontal = isHorizontalChart(state.layers);
+  const { hasFieldData } = useExistingFieldsReader();
 
   const { inputValue: localState, handleInputChange: setLocalState } = useDebouncedValue<XYState>({
     value: state,
@@ -248,10 +249,7 @@ export const AnnotationsPanel = (
                           field: field.name,
                           dataType: field.type,
                         },
-                        exists: fieldExists(
-                          frame.dataViews.existingFields[currentIndexPattern.title],
-                          field.name
-                        ),
+                        exists: hasFieldData(currentIndexPattern.id, field.name),
                         compatible: true,
                         'data-test-subj': `lnsXY-annotation-fieldOption-${field.name}`,
                       } as FieldOption<FieldOptionValue>)
@@ -379,7 +377,6 @@ export const AnnotationsPanel = (
               currentConfig={currentAnnotation}
               setConfig={setAnnotations}
               indexPattern={frame.dataViews.indexPatterns[localLayer.indexPatternId]}
-              existingFields={frame.dataViews.existingFields}
             />
           </EuiFormRow>
         </DimensionEditorSection>

x-pack/plugins/lens/public/visualizations/xy/xy_config_panel/annotations_config_panel/query_annotation_panel.tsx

@@ -10,8 +10,8 @@ import type { Query } from '@kbn/data-plugin/common';
 import type { QueryPointEventAnnotationConfig } from '@kbn/event-annotation-plugin/common';
 import { i18n } from '@kbn/i18n';
 import React from 'react';
+import { useExistingFieldsReader } from '@kbn/unified-field-list-plugin/public';
 import {
-  fieldExists,
   FieldOption,
   FieldOptionValue,
   FieldPicker,
@@ -41,7 +41,7 @@ export const ConfigPanelQueryAnnotation = ({
   queryInputShouldOpen?: boolean;
 }) => {
   const currentIndexPattern = frame.dataViews.indexPatterns[layer.indexPatternId];
-  const currentExistingFields = frame.dataViews.existingFields[currentIndexPattern.title];
+  const { hasFieldData } = useExistingFieldsReader();
   // list only date fields
   const options = currentIndexPattern.fields
     .filter((field) => field.type === 'date' && field.displayName)
@@ -53,7 +53,7 @@ export const ConfigPanelQueryAnnotation = ({
           field: field.name,
           dataType: field.type,
         },
-        exists: fieldExists(currentExistingFields, field.name),
+        exists: hasFieldData(currentIndexPattern.id, field.name),
         compatible: true,
         'data-test-subj': `lns-fieldOption-${field.name}`,
       } as FieldOption<FieldOptionValue>;

x-pack/plugins/lens/public/visualizations/xy/xy_config_panel/annotations_config_panel/tooltip_annotation_panel.tsx

@@ -9,9 +9,9 @@ import { htmlIdGenerator, EuiFlexItem, EuiPanel, EuiText } from '@elastic/eui';
 import { i18n } from '@kbn/i18n';
 import React, { useCallback, useMemo } from 'react';
 import { QueryPointEventAnnotationConfig } from '@kbn/event-annotation-plugin/common';
-import type { ExistingFieldsMap, IndexPattern } from '../../../../types';
+import { useExistingFieldsReader } from '@kbn/unified-field-list-plugin/public';
+import type { IndexPattern } from '../../../../types';
 import {
-  fieldExists,
   FieldOption,
   FieldOptionValue,
   FieldPicker,
@@ -31,7 +31,6 @@ export interface FieldInputsProps {
   currentConfig: QueryPointEventAnnotationConfig;
   setConfig: (config: QueryPointEventAnnotationConfig) => void;
   indexPattern: IndexPattern;
-  existingFields: ExistingFieldsMap;
   invalidFields?: string[];
 }
 
@@ -51,9 +50,9 @@ export function TooltipSection({
   currentConfig,
   setConfig,
   indexPattern,
-  existingFields,
   invalidFields,
 }: FieldInputsProps) {
+  const { hasFieldData } = useExistingFieldsReader();
   const onChangeWrapped = useCallback(
     (values: WrappedValue[]) => {
       setConfig({
@@ -124,7 +123,6 @@ export function TooltipSection({
       </>
     );
   }
-  const currentExistingField = existingFields[indexPattern.title];
 
   const options = indexPattern.fields
     .filter(
@@ -140,7 +138,7 @@ export function TooltipSection({
             field: field.name,
             dataType: field.type,
           },
-          exists: fieldExists(currentExistingField, field.name),
+          exists: hasFieldData(indexPattern.id, field.name),
           compatible: true,
           'data-test-subj': `lnsXY-annotation-tooltip-fieldOption-${field.name}`,
         } as FieldOption<FieldOptionValue>)

x-pack/plugins/translations/translations/fr-FR.json

@@ -17250,7 +17250,6 @@
     "xpack.lens.indexPattern.timeShiftSmallWarning": "{label} utilise un décalage temporel de {columnTimeShift} qui est inférieur à l'intervalle de l'histogramme des dates de {interval}. Pour éviter une non-correspondance des données, utilisez un multiple de {interval} comme décalage.",
     "xpack.lens.indexPattern.uniqueLabel": "{label} [{num}]",
     "xpack.lens.indexPattern.valueCountOf": "Nombre de {name}",
-    "xpack.lens.indexPatterns.fieldSearchLiveRegion": "{availableFields} {availableFields, plural, one {champ} other {champs}} disponible(s). {emptyFields} {emptyFields, plural, one {champ} other {champs}} vide(s). {metaFields} {metaFields, plural, one {champ} other {champs}} méta.",
     "xpack.lens.indexPatternSuggestion.removeLayerLabel": "Afficher uniquement {indexPatternTitle}",
     "xpack.lens.indexPatternSuggestion.removeLayerPositionLabel": "Afficher uniquement le calque {layerNumber}",
     "xpack.lens.modalTitle.title.clear": "Effacer le calque {layerType} ?",
@@ -17558,7 +17557,6 @@
     "xpack.lens.heatmapVisualization.missingXAccessorLongMessage": "La configuration de l'axe horizontal est manquante.",
     "xpack.lens.heatmapVisualization.missingXAccessorShortMessage": "Axe horizontal manquant.",
     "xpack.lens.indexPattern.advancedSettings": "Avancé",
-    "xpack.lens.indexPattern.allFieldsLabel": "Tous les champs",
     "xpack.lens.indexPattern.allFieldsLabelHelp": "Glissez-déposez les champs disponibles dans l’espace de travail et créez des visualisations. Pour modifier les champs disponibles, sélectionnez une vue de données différente, modifiez vos requêtes ou utilisez une plage temporelle différente. Certains types de champ ne peuvent pas être visualisés dans Lens, y compris les champ de texte intégral et champs géographiques.",
     "xpack.lens.indexPattern.allFieldsSamplingLabelHelp": "Les champs disponibles contiennent les données des 500 premiers documents correspondant aux filtres. Pour afficher tous les filtres, développez les champs vides. Vous ne pouvez pas créer de visualisations avec des champs de texte intégral, géographiques, lissés et d’objet.",
     "xpack.lens.indexPattern.ascendingCountPrecisionErrorWarning.link": "veuillez consulter la documentation",
@@ -17605,12 +17603,7 @@
     "xpack.lens.indexPattern.differences.signature": "indicateur : nombre",
     "xpack.lens.indexPattern.emptyDimensionButton": "Dimension vide",
     "xpack.lens.indexPattern.emptyFieldsLabel": "Champs vides",
-    "xpack.lens.indexPattern.emptyFieldsLabelHelp": "Les champs vides ne contenaient aucune valeur dans les 500 premiers documents basés sur vos filtres.",
     "xpack.lens.indexPattern.enableAccuracyMode": "Activer le mode de précision",
-    "xpack.lens.indexPattern.existenceErrorAriaLabel": "La récupération de l'existence a échoué",
-    "xpack.lens.indexPattern.existenceErrorLabel": "Impossible de charger les informations de champ",
-    "xpack.lens.indexPattern.existenceTimeoutAriaLabel": "La récupération de l'existence a expiré",
-    "xpack.lens.indexPattern.existenceTimeoutLabel": "Les informations de champ ont pris trop de temps",
     "xpack.lens.indexPattern.fieldItemTooltip": "Effectuez un glisser-déposer pour visualiser.",
     "xpack.lens.indexPattern.fieldPlaceholder": "Champ",
     "xpack.lens.indexPattern.fieldStatsButtonEmptyLabel": "Ce champ ne comporte aucune donnée mais vous pouvez toujours effectuer un glisser-déposer pour visualiser.",
@@ -17788,16 +17781,6 @@
     "xpack.lens.indexPattern.useAsTopLevelAgg": "Regrouper d'abord en fonction de ce champ",
     "xpack.lens.indexPatterns.clearFiltersLabel": "Effacer le nom et saisissez les filtres",
     "xpack.lens.indexPatterns.filterByNameLabel": "Rechercher les noms de champs",
-    "xpack.lens.indexPatterns.noAvailableDataLabel": "Aucun champ disponible ne contient de données.",
-    "xpack.lens.indexPatterns.noDataLabel": "Aucun champ.",
-    "xpack.lens.indexPatterns.noEmptyDataLabel": "Aucun champ vide.",
-    "xpack.lens.indexPatterns.noFields.extendTimeBullet": "Extension de la plage temporelle",
-    "xpack.lens.indexPatterns.noFields.fieldTypeFilterBullet": "Utilisation de différents filtres de champ",
-    "xpack.lens.indexPatterns.noFields.globalFiltersBullet": "Modification des filtres globaux",
-    "xpack.lens.indexPatterns.noFields.tryText": "Essayer :",
-    "xpack.lens.indexPatterns.noFieldsLabel": "Aucun champ n'existe dans cette vue de données.",
-    "xpack.lens.indexPatterns.noFilteredFieldsLabel": "Aucun champ ne correspond aux filtres sélectionnés.",
-    "xpack.lens.indexPatterns.noMetaDataLabel": "Aucun champ méta.",
     "xpack.lens.label.gauge.labelMajor.header": "Titre",
     "xpack.lens.label.gauge.labelMinor.header": "Sous-titre",
     "xpack.lens.label.header": "Étiquette",

x-pack/plugins/translations/translations/ja-JP.json

@@ -17231,7 +17231,6 @@
     "xpack.lens.indexPattern.timeShiftSmallWarning": "{label}は{columnTimeShift}の時間シフトを使用しています。これは{interval}の日付ヒストグラム間隔よりも小さいです。不一致のデータを防止するには、時間シフトとして{interval}を使用します。",
     "xpack.lens.indexPattern.uniqueLabel": "{label} [{num}]",
     "xpack.lens.indexPattern.valueCountOf": "{name}のカウント",
-    "xpack.lens.indexPatterns.fieldSearchLiveRegion": "{availableFields}使用可能な{availableFields, plural, other  {フィールド}}。{emptyFields}空の{emptyFields, plural, other  {フィールド}}。 {metaFields}メタ{metaFields, plural, other  {フィールド}}。",
     "xpack.lens.indexPatternSuggestion.removeLayerLabel": "{indexPatternTitle}のみを表示",
     "xpack.lens.indexPatternSuggestion.removeLayerPositionLabel": "レイヤー{layerNumber}のみを表示",
     "xpack.lens.modalTitle.title.clear": "{layerType}レイヤーをクリアしますか？",
@@ -17541,7 +17540,6 @@
     "xpack.lens.heatmapVisualization.missingXAccessorLongMessage": "横軸の構成がありません。",
     "xpack.lens.heatmapVisualization.missingXAccessorShortMessage": "横軸がありません。",
     "xpack.lens.indexPattern.advancedSettings": "高度な設定",
-    "xpack.lens.indexPattern.allFieldsLabel": "すべてのフィールド",
     "xpack.lens.indexPattern.allFieldsLabelHelp": "使用可能なフィールドをワークスペースまでドラッグし、ビジュアライゼーションを作成します。使用可能なフィールドを変更するには、別のデータビューを選択するか、クエリを編集するか、別の時間範囲を使用します。一部のフィールドタイプは、完全なテキストおよびグラフィックフィールドを含む Lens では、ビジュアライゼーションできません。",
     "xpack.lens.indexPattern.allFieldsSamplingLabelHelp": "使用可能なフィールドには、フィルターと一致する最初の 500 件のドキュメントのデータがあります。すべてのフィールドを表示するには、空のフィールドを展開します。全文、地理、フラット化、オブジェクトフィールドでビジュアライゼーションを作成できません。",
     "xpack.lens.indexPattern.ascendingCountPrecisionErrorWarning.link": "ドキュメントをご覧ください",
@@ -17588,12 +17586,7 @@
     "xpack.lens.indexPattern.differences.signature": "メトリック：数値",
     "xpack.lens.indexPattern.emptyDimensionButton": "空のディメンション",
     "xpack.lens.indexPattern.emptyFieldsLabel": "空のフィールド",
-    "xpack.lens.indexPattern.emptyFieldsLabelHelp": "空のフィールドには、フィルターに基づく最初の 500 件のドキュメントの値が含まれていませんでした。",
     "xpack.lens.indexPattern.enableAccuracyMode": "精度モードを有効にする",
-    "xpack.lens.indexPattern.existenceErrorAriaLabel": "存在の取り込みに失敗しました",
-    "xpack.lens.indexPattern.existenceErrorLabel": "フィールド情報を読み込めません",
-    "xpack.lens.indexPattern.existenceTimeoutAriaLabel": "存在の取り込みがタイムアウトしました",
-    "xpack.lens.indexPattern.existenceTimeoutLabel": "フィールド情報に時間がかかりすぎました",
     "xpack.lens.indexPattern.fieldItemTooltip": "可視化するには、ドラッグアンドドロップします。",
     "xpack.lens.indexPattern.fieldPlaceholder": "フィールド",
     "xpack.lens.indexPattern.fieldStatsButtonEmptyLabel": "このフィールドにはデータがありませんが、ドラッグアンドドロップで可視化できます。",
@@ -17771,16 +17764,6 @@
     "xpack.lens.indexPattern.useAsTopLevelAgg": "最初にこのフィールドでグループ化",
     "xpack.lens.indexPatterns.clearFiltersLabel": "名前とタイプフィルターを消去",
     "xpack.lens.indexPatterns.filterByNameLabel": "検索フィールド名",
-    "xpack.lens.indexPatterns.noAvailableDataLabel": "データを含むフィールドはありません。",
-    "xpack.lens.indexPatterns.noDataLabel": "フィールドがありません。",
-    "xpack.lens.indexPatterns.noEmptyDataLabel": "空のフィールドがありません。",
-    "xpack.lens.indexPatterns.noFields.extendTimeBullet": "時間範囲を拡張中",
-    "xpack.lens.indexPatterns.noFields.fieldTypeFilterBullet": "別のフィールドフィルターを使用",
-    "xpack.lens.indexPatterns.noFields.globalFiltersBullet": "グローバルフィルターを変更",
-    "xpack.lens.indexPatterns.noFields.tryText": "試行対象：",
-    "xpack.lens.indexPatterns.noFieldsLabel": "このデータビューにはフィールドがありません。",
-    "xpack.lens.indexPatterns.noFilteredFieldsLabel": "選択したフィルターと一致するフィールドはありません。",
-    "xpack.lens.indexPatterns.noMetaDataLabel": "メタフィールドがありません。",
     "xpack.lens.label.gauge.labelMajor.header": "タイトル",
     "xpack.lens.label.gauge.labelMinor.header": "サブタイトル",
     "xpack.lens.label.header": "ラベル",

x-pack/plugins/translations/translations/zh-CN.json

@@ -17256,7 +17256,6 @@
     "xpack.lens.indexPattern.timeShiftSmallWarning": "{label} 使用的时间偏移 {columnTimeShift} 小于 Date Histogram 时间间隔 {interval} 。要防止数据不匹配，请使用 {interval} 的倍数作为时间偏移。",
     "xpack.lens.indexPattern.uniqueLabel": "{label} [{num}]",
     "xpack.lens.indexPattern.valueCountOf": "{name} 的计数",
-    "xpack.lens.indexPatterns.fieldSearchLiveRegion": "{availableFields} 个可用{availableFields, plural, other {字段}}。{emptyFields} 个空{emptyFields, plural, other {字段}}。{metaFields} 个元{metaFields, plural,other {字段}}。",
     "xpack.lens.indexPatternSuggestion.removeLayerLabel": "仅显示 {indexPatternTitle}",
     "xpack.lens.indexPatternSuggestion.removeLayerPositionLabel": "仅显示图层 {layerNumber}",
     "xpack.lens.modalTitle.title.clear": "清除 {layerType} 图层？",
@@ -17566,7 +17565,6 @@
     "xpack.lens.heatmapVisualization.missingXAccessorLongMessage": "水平轴配置缺失。",
     "xpack.lens.heatmapVisualization.missingXAccessorShortMessage": "缺失水平轴。",
     "xpack.lens.indexPattern.advancedSettings": "高级",
-    "xpack.lens.indexPattern.allFieldsLabel": "所有字段",
     "xpack.lens.indexPattern.allFieldsLabelHelp": "将可用字段拖放到工作区并创建可视化。要更改可用字段，请选择不同数据视图，编辑您的查询或使用不同时间范围。一些字段类型无法在 Lens 中可视化，包括全文本字段和地理字段。",
     "xpack.lens.indexPattern.allFieldsSamplingLabelHelp": "可用字段包含与您的筛选匹配的前 500 个文档中的数据。要查看所有字段，请展开空字段。无法使用全文本、地理、扁平和对象字段创建可视化。",
     "xpack.lens.indexPattern.ascendingCountPrecisionErrorWarning.link": "访问文档",
@@ -17613,12 +17611,7 @@
     "xpack.lens.indexPattern.differences.signature": "指标：数字",
     "xpack.lens.indexPattern.emptyDimensionButton": "空维度",
     "xpack.lens.indexPattern.emptyFieldsLabel": "空字段",
-    "xpack.lens.indexPattern.emptyFieldsLabelHelp": "空字段在基于您的筛选的前 500 个文档中不包含任何值。",
     "xpack.lens.indexPattern.enableAccuracyMode": "启用准确性模式",
-    "xpack.lens.indexPattern.existenceErrorAriaLabel": "现有内容提取失败",
-    "xpack.lens.indexPattern.existenceErrorLabel": "无法加载字段信息",
-    "xpack.lens.indexPattern.existenceTimeoutAriaLabel": "现有内容提取超时",
-    "xpack.lens.indexPattern.existenceTimeoutLabel": "字段信息花费时间过久",
     "xpack.lens.indexPattern.fieldItemTooltip": "拖放以可视化。",
     "xpack.lens.indexPattern.fieldPlaceholder": "字段",
     "xpack.lens.indexPattern.fieldStatsButtonEmptyLabel": "此字段不包含任何数据，但您仍然可以拖放以进行可视化。",
@@ -17796,16 +17789,6 @@
     "xpack.lens.indexPattern.useAsTopLevelAgg": "先按此字段分组",
     "xpack.lens.indexPatterns.clearFiltersLabel": "清除名称和类型筛选",
     "xpack.lens.indexPatterns.filterByNameLabel": "搜索字段名称",
-    "xpack.lens.indexPatterns.noAvailableDataLabel": "没有包含数据的可用字段。",
-    "xpack.lens.indexPatterns.noDataLabel": "无字段。",
-    "xpack.lens.indexPatterns.noEmptyDataLabel": "无空字段。",
-    "xpack.lens.indexPatterns.noFields.extendTimeBullet": "延伸时间范围",
-    "xpack.lens.indexPatterns.noFields.fieldTypeFilterBullet": "使用不同的字段筛选",
-    "xpack.lens.indexPatterns.noFields.globalFiltersBullet": "更改全局筛选",
-    "xpack.lens.indexPatterns.noFields.tryText": "尝试：",
-    "xpack.lens.indexPatterns.noFieldsLabel": "在此数据视图中不存在任何字段。",
-    "xpack.lens.indexPatterns.noFilteredFieldsLabel": "没有字段匹配选定筛选。",
-    "xpack.lens.indexPatterns.noMetaDataLabel": "无元字段。",
     "xpack.lens.label.gauge.labelMajor.header": "标题",
     "xpack.lens.label.gauge.labelMinor.header": "子标题",
     "xpack.lens.label.header": "标签",