[Lens] Formula editor (#99297)

* 💄 Hack to fix suggestion box

* 🐛 Fix validation messages

* 🐛 Relax operations check for managedReferences

* Change completion params

* 🏷️ Fix missing arg issue

* ✨ Add more tinymath fns

* 🐛 Improved validation around math operations + multiple named arguments

* 🐛 Use new onError feature in math expression

* ♻️ Refactor namedArguments validation

* 🐛 Fix circular dependency issue in tests + minor fixes

* Move formula into a tab

* 🔥 Leftovers from previous merge

* ✨ Move over namedArgs from previous function

* ✅ Add tests for transferable scenarios

* ✅ Fixed broken test

* ✨ Use custom label for axis

* Allow switching back and forth to formula tab

* Add a section for the function reference

* Add modal editor and markdown docs

* Change the way math nodes are validated

* Use custom portal to fix monaco positioning

* Fix model sharing issues

* Provide signature help

* 🐛 Fix small test issue

* 🐛 Mark pow arguments as required

* 🐛 validate on first render only if a formula is present

* 🔥 Remove log10 fn for now

* ✨ Improved math validation + add tests for math functions

* Fix mount/unmount issues with Monaco

* [Lens] Fully unmount React when flyout closes

* Fix bug with editor frame unmounting

* Fix type

* Add tests for monaco providers, add hover provider

* Add test for last_value

* Usability improvements

* Add KQL and Lucene named parameters

* Add kql, lucene completion and validation

* Fix autocomplete on weird characters and properly connect KQL

* Highlight functions that have additional requirements after validating

* Fix type error and move help text to popover

* Fix escape characters inside KQL

* 🐛 Fix dataType issue when moving over to Formula

* Automatically insert single quotes on every named param

* Only insert single quotes when typing kql= or lucene=

* Reorganize help popover

* Fix merge issues

* Update grammar for formulas

* Fix bad merge

* Rough fullscreen mode

* Type updates

* Pass through fullscreen state

* Remove more chrome from full screen mode

* Fix minor bugs in formula typing

* 🐛 Decouple column order of references and output

* 🔧 Fix tests and types

* ✅ Add first functional test

* Fix copying formulas and empty formula

* Trigger suggestion prompt when hitting enter on function or typing kql=

* 🐛 Prevent flyout from closing while interacting with monaco

* refactoring

* move main column generation into parse module

* fix tests

* refactor small formula styles and markup

* documentation

* adjustments in formula footer

* Formula refactoring (#12)

* refactoring

* move main column generation into parse module

* fix tests

* more style and markup tweak for custom formula

* Fix tests

* [Expressions] Use table column ID instead of name when set

* [Lens] Create managedReference type for formulas

* Fix test failures

* Fix i18n types

* fix fullscreen flex issues

* Delete managedReference when replacing

* refactor css and markup; add button placeholders

* [Lens] Formulas

* Tests for formula

Co-authored-by: Marco Liberati <marco.liberati@elastic.co>

* added error count placeholder

* Add tooltips

* Refactoring from code review

* Fix some editor issues

* Update ID matching to match by name sometimes

* Improve performance of Monaco, fix formulas with 0, update labels

* Improve performance of full screen toggle

* Fix formula tests

* fix stuff

* Add an extra case to prevent insertion of duplicate column

* Simplify logic and add test for output ID

* add telemetry for Lens formula (#15)

* Respond to review comments

* ✨ Improve the signatures with better documentation and examples

* adjust border styles to account for docs collapse

* refactor docs markup; restructure docs obj; styles

* Fix formula auto reordering (#18)

* fix formula auto reordering

* add unit test

* Fix and improve suggestion experience in Formula (#19)

* ✨ Revisit documentation and suggestions

* 👌 Integrated feedback

* ✨ Add query validation for quotes

* Usability updates & type fixes

* add search to formula

* fix form styles to match designs

* fix text styles; revert to Markdown for control

* 👌 Integrated more feedback

* improve search

* improve suggestions

* improve suggestions even more

* 🐛 Fix i18n issues (#22)

* Persist formula on leave, fix fullscreen and popovers

* Fix documentation tests

* 🏷️ fix type issue

* 🐛 Remove hidden operations from valid functions list

* 🐛 Fix empty string query edge case

* 🐛 Enable more suggestions + extends validation

* Fix tests that depended on setState being called without function

* Error state and text wrapping updates

* ✨ Add new module to CodeEditor for brackets matching (#25)

* Fix type

* show warning

* keep current quick function

* ✨ Improve suggestions within kql query

* 📷 Fix snapshot editor test

* 🐛 Improved suggestion for single quote and refactored debounce

* Fix lodash usage

* Fix tests

* Revert "keep current quick function"

This reverts commit ed477054c5.

* Improve performance of dispatch by using timeout

* Improve memoization of datapanel

* Fix escape characters

* fix reduced suggestions

* fix responsiveness

* fix unit test

* Fix autocomplete on nested math

* Show errors and warnings on first render

* fix transposing column crash

* Update comment

* 🐛 Fix field error message

* fix test types

* 📝 Fix i18n name

* 💄 Manage wordwrap via react component

* Fix selector for palettes that interferes with quick functions

* Use word wrapping by default

* Errors for managed references are handled at the top level

* 🐛 Move the cursor just next to new inserted text

* ⚗️ First pass for performance

* 🐛 Fix unwanted change

* ⚡ Memoize as many combobox props as possible

* ⚡ More memoization

* Show errors in hover

* Use temporary invalid state when moving away from formula

* Remove setActiveDimension and shouldClose, fixed by async setters

* Fix test dependency

* do not show quick functions tab

* increase documentation popover width

* fix functional test

* Call setActiveDimension when updating visualization

* Simplify handling of flyout with incomplete columns

* Fix test issues

* add description to formula telemetry

* fix schema

* Update from design feedback

* More review comments

* Hide callout border from v7 theme

Co-authored-by: dej611 <dej611@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>
Co-authored-by: Michael Marcialis <michael.marcialis@elastic.co>
Co-authored-by: Joe Reuter <email@johannes-reuter.de>
Co-authored-by: Marco Liberati <marco.liberati@elastic.co>
Co-authored-by: Marco Liberati <dej611@users.noreply.github.com>

packages/kbn-monaco/src/monaco_imports.ts

@@ -21,5 +21,6 @@ import 'monaco-editor/esm/vs/editor/contrib/folding/folding.js'; // Needed for f
 import 'monaco-editor/esm/vs/editor/contrib/suggest/suggestController.js'; // Needed for suggestions
 import 'monaco-editor/esm/vs/editor/contrib/hover/hover.js'; // Needed for hover
 import 'monaco-editor/esm/vs/editor/contrib/parameterHints/parameterHints.js'; // Needed for signature
+import 'monaco-editor/esm/vs/editor/contrib/bracketMatching/bracketMatching.js'; // Needed for brackets matching highlight
 
 export { monaco };

packages/kbn-tinymath/grammar/grammar.peggy

@@ -1,16 +1,16 @@
 // tinymath parsing grammar
 
 {
-   function simpleLocation (location) {
-  // Returns an object representing the position of the function within the expression,
-  // demarcated by the position of its first character and last character. We calculate these values
-  // using the offset because the expression could span multiple lines, and we don't want to deal
-  // with column and line values.
-  return {
-   min: location.start.offset,
-   max: location.end.offset
+  function simpleLocation (location) {
+    // Returns an object representing the position of the function within the expression,
+    // demarcated by the position of its first character and last character. We calculate these values
+    // using the offset because the expression could span multiple lines, and we don't want to deal
+    // with column and line values.
+    return {
+      min: location.start.offset,
+      max: location.end.offset
+    }
   }
- }
 }
 
 start
@@ -74,26 +74,34 @@ Expression
   = AddSubtract
 
 AddSubtract
-  = _ left:MultiplyDivide rest:(('+' / '-') MultiplyDivide)* _ {
-    return rest.reduce((acc, curr) => ({
+  = _ left:MultiplyDivide rest:(('+' / '-') MultiplyDivide)+ _ {
+    const topLevel = rest.reduce((acc, curr) => ({
       type: 'function',
       name: curr[0] === '+' ? 'add' : 'subtract',
       args: [acc, curr[1]],
-      location: simpleLocation(location()),
-      text: text()
-    }), left)
+    }), left);
+    if (typeof topLevel === 'object') {
+      topLevel.location = simpleLocation(location());
+      topLevel.text = text();
+    }
+    return topLevel;
   }
+  / MultiplyDivide
 
 MultiplyDivide
   = _ left:Factor rest:(('*' / '/') Factor)* _ {
-    return rest.reduce((acc, curr) => ({
+    const topLevel = rest.reduce((acc, curr) => ({
       type: 'function',
       name: curr[0] === '*' ? 'multiply' : 'divide',
       args: [acc, curr[1]],
-      location: simpleLocation(location()),
-      text: text()
-    }), left)
+    }), left);
+    if (typeof topLevel === 'object') {
+      topLevel.location = simpleLocation(location());
+      topLevel.text = text();
+    }
+    return topLevel;
   }
+  / Factor
 
 Factor
   = Group

packages/kbn-tinymath/index.d.ts

@@ -24,9 +24,11 @@ export interface TinymathLocation {
 export interface TinymathFunction {
   type: 'function';
   name: string;
-  text: string;
   args: TinymathAST[];
-  location: TinymathLocation;
+  // Location is not guaranteed because PEG grammars are not left-recursive
+  location?: TinymathLocation;
+  // Text is not guaranteed because PEG grammars are not left-recursive
+  text?: string;
 }
 
 export interface TinymathVariable {

packages/kbn-tinymath/test/library.test.js

@@ -41,6 +41,35 @@ describe('Parser', () => {
     });
   });
 
+  describe('Math', () => {
+    it('converts basic symbols into left-to-right pairs', () => {
+      expect(parse('a + b + c - d')).toEqual({
+        args: [
+          {
+            name: 'add',
+            type: 'function',
+            args: [
+              {
+                name: 'add',
+                type: 'function',
+                args: [
+                  expect.objectContaining({ location: { min: 0, max: 2 } }),
+                  expect.objectContaining({ location: { min: 3, max: 6 } }),
+                ],
+              },
+              expect.objectContaining({ location: { min: 7, max: 10 } }),
+            ],
+          },
+          expect.objectContaining({ location: { min: 11, max: 13 } }),
+        ],
+        name: 'subtract',
+        type: 'function',
+        text: 'a + b + c - d',
+        location: { min: 0, max: 13 },
+      });
+    });
+  });
+
   describe('Variables', () => {
     it('strings', () => {
       expect(parse('f')).toEqual(variableEqual('f'));
@@ -263,6 +292,8 @@ describe('Evaluate', () => {
     expect(evaluate('5/20')).toEqual(0.25);
     expect(evaluate('1 + 1 + 2 + 3 + 12')).toEqual(19);
     expect(evaluate('100 / 10 / 10')).toEqual(1);
+    expect(evaluate('0 * 1 - 100 / 10 / 10')).toEqual(-1);
+    expect(evaluate('100 / (10 / 10)')).toEqual(100);
   });
 
   it('equations with functions', () => {

src/plugins/dashboard/server/usage/dashboard_telemetry.test.ts

@@ -72,6 +72,22 @@ const lensXYSeriesB = ({
         visualization: {
           preferredSeriesType: 'seriesB',
         },
+        datasourceStates: {
+          indexpattern: {
+            layers: {
+              first: {
+                columns: {
+                  first: {
+                    operationType: 'terms',
+                  },
+                  second: {
+                    operationType: 'formula',
+                  },
+                },
+              },
+            },
+          },
+        },
       },
     },
   },
@@ -144,6 +160,7 @@ describe('dashboard telemetry', () => {
       expect(collectorData.lensByValue.a).toBe(3);
       expect(collectorData.lensByValue.seriesA).toBe(2);
       expect(collectorData.lensByValue.seriesB).toBe(1);
+      expect(collectorData.lensByValue.formula).toBe(1);
     });
 
     it('handles misshapen lens panels', () => {

src/plugins/dashboard/server/usage/dashboard_telemetry.ts

@@ -27,6 +27,16 @@ interface LensPanel extends SavedDashboardPanel730ToLatest {
         visualization?: {
           preferredSeriesType?: string;
         };
+        datasourceStates?: {
+          indexpattern?: {
+            layers: Record<
+              string,
+              {
+                columns: Record<string, { operationType: string }>;
+              }
+            >;
+          };
+        };
       };
     };
   };
@@ -109,6 +119,19 @@ export const collectByValueLensInfo: DashboardCollectorFunction = (panels, colle
       }
 
       collectorData.lensByValue[type] = collectorData.lensByValue[type] + 1;
+
+      const hasFormula = Object.values(
+        lensPanel.embeddableConfig.attributes.state?.datasourceStates?.indexpattern?.layers || {}
+      ).some((layer) =>
+        Object.values(layer.columns).some((column) => column.operationType === 'formula')
+      );
+
+      if (hasFormula && !collectorData.lensByValue.formula) {
+        collectorData.lensByValue.formula = 0;
+      }
+      if (hasFormula) {
+        collectorData.lensByValue.formula++;
+      }
     }
   }
 };

src/plugins/expressions/common/expression_functions/specs/map_column.ts

@@ -10,7 +10,7 @@ import { Observable } from 'rxjs';
 import { take } from 'rxjs/operators';
 import { i18n } from '@kbn/i18n';
 import { ExpressionFunctionDefinition } from '../types';
-import { Datatable, getType } from '../../expression_types';
+import { Datatable, DatatableColumn, getType } from '../../expression_types';
 
 export interface MapColumnArguments {
   id?: string | null;
@@ -110,10 +110,10 @@ export const mapColumn: ExpressionFunctionDefinition<
 
     return Promise.all(rowPromises).then((rows) => {
       const type = rows.length ? getType(rows[0][columnId]) : 'null';
-      const newColumn = {
+      const newColumn: DatatableColumn = {
         id: columnId,
         name: args.name,
-        meta: { type },
+        meta: { type, params: { id: type } },
       };
       if (args.copyMetaFrom) {
         const metaSourceFrom = columns.find(({ id }) => id === args.copyMetaFrom);

src/plugins/expressions/common/expression_functions/specs/tests/map_column.test.ts

@@ -29,7 +29,11 @@ describe('mapColumn', () => {
     expect(result.type).toBe('datatable');
     expect(result.columns).toEqual([
       ...testTable.columns,
-      { id: 'pricePlusTwo', name: 'pricePlusTwo', meta: { type: 'number' } },
+      {
+        id: 'pricePlusTwo',
+        name: 'pricePlusTwo',
+        meta: { type: 'number', params: { id: 'number' } },
+      },
     ]);
     expect(result.columns[result.columns.length - 1]).toHaveProperty('name', 'pricePlusTwo');
     expect(result.rows[arbitraryRowIndex]).toHaveProperty('pricePlusTwo');

src/plugins/kibana_react/public/code_editor/__snapshots__/code_editor.test.tsx.snap

@@ -11,6 +11,7 @@ exports[`is rendered 1`] = `
     onChange={[Function]}
     options={
       Object {
+        "matchBrackets": "never",
         "minimap": Object {
           "enabled": false,
         },
@@ -39,6 +40,7 @@ exports[`is rendered 1`] = `
     nodeType="div"
     onResize={[Function]}
     querySelector={null}
+    refreshMode="debounce"
     refreshRate={1000}
     skipOnMount={false}
     targetDomEl={null}

src/plugins/kibana_react/public/code_editor/code_editor.tsx

@@ -187,10 +187,16 @@ export class CodeEditor extends React.Component<Props, {}> {
             wordBasedSuggestions: false,
             wordWrap: 'on',
             wrappingIndent: 'indent',
+            matchBrackets: 'never',
             ...options,
           }}
         />
-        <ReactResizeDetector handleWidth handleHeight onResize={this._updateDimensions} />
+        <ReactResizeDetector
+          handleWidth
+          handleHeight
+          onResize={this._updateDimensions}
+          refreshMode="debounce"
+        />
       </>
     );
   }

x-pack/plugins/canvas/shareable_runtime/webpack.config.js

@@ -38,7 +38,7 @@ module.exports = {
         'src/plugins/data/public/expressions/interpreter'
       ),
       'kbn/interpreter': path.resolve(KIBANA_ROOT, 'packages/kbn-interpreter/target/common'),
-      tinymath: path.resolve(KIBANA_ROOT, 'node_modules/tinymath/lib/tinymath.es5.js'),
+      tinymath: path.resolve(KIBANA_ROOT, 'node_modules/tinymath/lib/tinymath.min.js'),
       core_app_image_assets: path.resolve(KIBANA_ROOT, 'src/core/public/core_app/images'),
     },
     extensions: ['.js', '.json', '.ts', '.tsx', '.scss'],

x-pack/plugins/lens/public/app_plugin/lens_top_nav.tsx

@@ -7,7 +7,7 @@
 
 import { isEqual } from 'lodash';
 import { i18n } from '@kbn/i18n';
-import React from 'react';
+import React, { useCallback, useMemo } from 'react';
 import { TopNavMenuData } from '../../../../../src/plugins/navigation/public';
 import { LensAppServices, LensTopNavActions, LensTopNavMenuProps } from './types';
 import { downloadMultipleAs } from '../../../../../src/plugins/share/public';
@@ -164,79 +164,152 @@ export const LensTopNavMenu = ({
   const unsavedTitle = i18n.translate('xpack.lens.app.unsavedFilename', {
     defaultMessage: 'unsaved',
   });
-  const topNavConfig = getLensTopNavConfig({
-    showSaveAndReturn: Boolean(
-      isLinkedToOriginatingApp &&
-        // Temporarily required until the 'by value' paradigm is default.
-        (dashboardFeatureFlag.allowByValueEmbeddables || Boolean(initialInput))
-    ),
-    enableExportToCSV: Boolean(isSaveable && activeData && Object.keys(activeData).length),
-    isByValueMode: getIsByValueMode(),
-    allowByValue: dashboardFeatureFlag.allowByValueEmbeddables,
-    showCancel: Boolean(isLinkedToOriginatingApp),
-    savingToLibraryPermitted,
-    savingToDashboardPermitted,
-    actions: {
-      exportToCSV: () => {
-        if (!activeData) {
-          return;
-        }
-        const datatables = Object.values(activeData);
-        const content = datatables.reduce<Record<string, { content: string; type: string }>>(
-          (memo, datatable, i) => {
-            // skip empty datatables
-            if (datatable) {
-              const postFix = datatables.length > 1 ? `-${i + 1}` : '';
+  const topNavConfig = useMemo(
+    () =>
+      getLensTopNavConfig({
+        showSaveAndReturn: Boolean(
+          isLinkedToOriginatingApp &&
+            // Temporarily required until the 'by value' paradigm is default.
+            (dashboardFeatureFlag.allowByValueEmbeddables || Boolean(initialInput))
+        ),
+        enableExportToCSV: Boolean(isSaveable && activeData && Object.keys(activeData).length),
+        isByValueMode: getIsByValueMode(),
+        allowByValue: dashboardFeatureFlag.allowByValueEmbeddables,
+        showCancel: Boolean(isLinkedToOriginatingApp),
+        savingToLibraryPermitted,
+        savingToDashboardPermitted,
+        actions: {
+          exportToCSV: () => {
+            if (!activeData) {
+              return;
+            }
+            const datatables = Object.values(activeData);
+            const content = datatables.reduce<Record<string, { content: string; type: string }>>(
+              (memo, datatable, i) => {
+                // skip empty datatables
+                if (datatable) {
+                  const postFix = datatables.length > 1 ? `-${i + 1}` : '';
 
-              memo[`${lastKnownDoc?.title || unsavedTitle}${postFix}.csv`] = {
-                content: exporters.datatableToCSV(datatable, {
-                  csvSeparator: uiSettings.get('csv:separator', ','),
-                  quoteValues: uiSettings.get('csv:quoteValues', true),
-                  formatFactory: data.fieldFormats.deserialize,
-                }),
-                type: exporters.CSV_MIME_TYPE,
-              };
+                  memo[`${lastKnownDoc?.title || unsavedTitle}${postFix}.csv`] = {
+                    content: exporters.datatableToCSV(datatable, {
+                      csvSeparator: uiSettings.get('csv:separator', ','),
+                      quoteValues: uiSettings.get('csv:quoteValues', true),
+                      formatFactory: data.fieldFormats.deserialize,
+                    }),
+                    type: exporters.CSV_MIME_TYPE,
+                  };
+                }
+                return memo;
+              },
+              {}
+            );
+            if (content) {
+              downloadMultipleAs(content);
             }
-            return memo;
           },
-          {}
-        );
-        if (content) {
-          downloadMultipleAs(content);
-        }
-      },
-      saveAndReturn: () => {
-        if (savingToDashboardPermitted && lastKnownDoc) {
-          // disabling the validation on app leave because the document has been saved.
-          onAppLeave((actions) => {
-            return actions.default();
-          });
-          runSave(
-            {
-              newTitle: lastKnownDoc.title,
-              newCopyOnSave: false,
-              isTitleDuplicateConfirmed: false,
-              returnToOrigin: true,
-            },
-            {
-              saveToLibrary:
-                (initialInput && attributeService.inputIsRefType(initialInput)) ?? false,
+          saveAndReturn: () => {
+            if (savingToDashboardPermitted && lastKnownDoc) {
+              // disabling the validation on app leave because the document has been saved.
+              onAppLeave((actions) => {
+                return actions.default();
+              });
+              runSave(
+                {
+                  newTitle: lastKnownDoc.title,
+                  newCopyOnSave: false,
+                  isTitleDuplicateConfirmed: false,
+                  returnToOrigin: true,
+                },
+                {
+                  saveToLibrary:
+                    (initialInput && attributeService.inputIsRefType(initialInput)) ?? false,
+                }
+              );
             }
-          );
+          },
+          showSaveModal: () => {
+            if (savingToDashboardPermitted || savingToLibraryPermitted) {
+              setIsSaveModalVisible(true);
+            }
+          },
+          cancel: () => {
+            if (redirectToOrigin) {
+              redirectToOrigin();
+            }
+          },
+        },
+      }),
+    [
+      activeData,
+      attributeService,
+      dashboardFeatureFlag.allowByValueEmbeddables,
+      data.fieldFormats.deserialize,
+      getIsByValueMode,
+      initialInput,
+      isLinkedToOriginatingApp,
+      isSaveable,
+      lastKnownDoc,
+      onAppLeave,
+      redirectToOrigin,
+      runSave,
+      savingToDashboardPermitted,
+      savingToLibraryPermitted,
+      setIsSaveModalVisible,
+      uiSettings,
+      unsavedTitle,
+    ]
+  );
+
+  const onQuerySubmitWrapped = useCallback(
+    (payload) => {
+      const { dateRange, query: newQuery } = payload;
+      const currentRange = data.query.timefilter.timefilter.getTime();
+      if (dateRange.from !== currentRange.from || dateRange.to !== currentRange.to) {
+        data.query.timefilter.timefilter.setTime(dateRange);
+        trackUiEvent('app_date_change');
+      } else {
+        // Query has changed, renew the session id.
+        // Time change will be picked up by the time subscription
+        dispatchSetState({ searchSessionId: data.search.session.start() });
+        trackUiEvent('app_query_change');
+      }
+      if (newQuery) {
+        if (!isEqual(newQuery, query)) {
+          dispatchSetState({ query: newQuery });
         }
-      },
-      showSaveModal: () => {
-        if (savingToDashboardPermitted || savingToLibraryPermitted) {
-          setIsSaveModalVisible(true);
-        }
-      },
-      cancel: () => {
-        if (redirectToOrigin) {
-          redirectToOrigin();
-        }
-      },
+      }
     },
-  });
+    [data.query.timefilter.timefilter, data.search.session, dispatchSetState, query]
+  );
+
+  const onSavedWrapped = useCallback(
+    (newSavedQuery) => {
+      dispatchSetState({ savedQuery: newSavedQuery });
+    },
+    [dispatchSetState]
+  );
+
+  const onSavedQueryUpdatedWrapped = useCallback(
+    (newSavedQuery) => {
+      const savedQueryFilters = newSavedQuery.attributes.filters || [];
+      const globalFilters = data.query.filterManager.getGlobalFilters();
+      data.query.filterManager.setFilters([...globalFilters, ...savedQueryFilters]);
+      dispatchSetState({
+        query: newSavedQuery.attributes.query,
+        savedQuery: { ...newSavedQuery },
+      }); // Shallow query for reference issues
+    },
+    [data.query.filterManager, dispatchSetState]
+  );
+
+  const onClearSavedQueryWrapped = useCallback(() => {
+    data.query.filterManager.setFilters(data.query.filterManager.getGlobalFilters());
+    dispatchSetState({
+      filters: data.query.filterManager.getGlobalFilters(),
+      query: data.query.queryString.getDefaultQuery(),
+      savedQuery: undefined,
+    });
+  }, [data.query.filterManager, data.query.queryString, dispatchSetState]);
 
   return (
     <TopNavMenu
@@ -244,44 +317,10 @@ export const LensTopNavMenu = ({
       config={topNavConfig}
       showSaveQuery={Boolean(application.capabilities.visualize.saveQuery)}
       savedQuery={savedQuery}
-      onQuerySubmit={(payload) => {
-        const { dateRange, query: newQuery } = payload;
-        const currentRange = data.query.timefilter.timefilter.getTime();
-        if (dateRange.from !== currentRange.from || dateRange.to !== currentRange.to) {
-          data.query.timefilter.timefilter.setTime(dateRange);
-          trackUiEvent('app_date_change');
-        } else {
-          // Query has changed, renew the session id.
-          // Time change will be picked up by the time subscription
-          dispatchSetState({ searchSessionId: data.search.session.start() });
-          trackUiEvent('app_query_change');
-        }
-        if (newQuery) {
-          if (!isEqual(newQuery, query)) {
-            dispatchSetState({ query: newQuery });
-          }
-        }
-      }}
-      onSaved={(newSavedQuery) => {
-        dispatchSetState({ savedQuery: newSavedQuery });
-      }}
-      onSavedQueryUpdated={(newSavedQuery) => {
-        const savedQueryFilters = newSavedQuery.attributes.filters || [];
-        const globalFilters = data.query.filterManager.getGlobalFilters();
-        data.query.filterManager.setFilters([...globalFilters, ...savedQueryFilters]);
-        dispatchSetState({
-          query: newSavedQuery.attributes.query,
-          savedQuery: { ...newSavedQuery },
-        }); // Shallow query for reference issues
-      }}
-      onClearSavedQuery={() => {
-        data.query.filterManager.setFilters(data.query.filterManager.getGlobalFilters());
-        dispatchSetState({
-          filters: data.query.filterManager.getGlobalFilters(),
-          query: data.query.queryString.getDefaultQuery(),
-          savedQuery: undefined,
-        });
-      }}
+      onQuerySubmit={onQuerySubmitWrapped}
+      onSaved={onSavedWrapped}
+      onSavedQueryUpdated={onSavedQueryUpdatedWrapped}
+      onClearSavedQuery={onClearSavedQueryWrapped}
       indexPatterns={indexPatternsForTopNav}
       query={query}
       dateRangeFrom={from}

x-pack/plugins/lens/public/datatable_visualization/components/columns.tsx

@@ -51,7 +51,7 @@ export const createGridColumns = (
     columnId,
   }: Pick<EuiDataGridColumnCellActionProps, 'rowIndex' | 'columnId'>) => {
     const rowValue = table.rows[rowIndex][columnId];
-    const column = columnsReverseLookup[columnId];
+    const column = columnsReverseLookup?.[columnId];
     const contentsIsDefined = rowValue != null;
 
     const cellContent = formatFactory(column?.meta?.params).convert(rowValue);

x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/config_panel.test.tsx

@@ -76,6 +76,8 @@ describe('ConfigPanel', () => {
       framePublicAPI: frame,
       dispatch: jest.fn(),
       core: coreMock.createStart(),
+      isFullscreen: false,
+      toggleFullscreen: jest.fn(),
     };
   }
 
@@ -119,19 +121,23 @@ describe('ConfigPanel', () => {
     expect(component.find(LayerPanel).exists()).toBe(false);
   });
 
-  it('allow datasources and visualizations to use setters', () => {
+  it('allow datasources and visualizations to use setters', async () => {
     const props = getDefaultProps();
     const component = mountWithIntl(<LayerPanels {...props} />);
     const { updateDatasource, updateAll } = component.find(LayerPanel).props();
 
     const updater = () => 'updated';
     updateDatasource('ds1', updater);
+    // wait for one tick so async updater has a chance to trigger
+    await new Promise((r) => setTimeout(r, 0));
     expect(props.dispatch).toHaveBeenCalledTimes(1);
     expect(props.dispatch.mock.calls[0][0].updater(props.datasourceStates.ds1.state)).toEqual(
       'updated'
     );
 
     updateAll('ds1', updater, props.visualizationState);
+    // wait for one tick so async updater has a chance to trigger
+    await new Promise((r) => setTimeout(r, 0));
     expect(props.dispatch).toHaveBeenCalledTimes(2);
     expect(props.dispatch.mock.calls[0][0].updater(props.datasourceStates.ds1.state)).toEqual(
       'updated'

x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/config_panel.tsx

@@ -71,32 +71,54 @@ export function LayerPanels(
     },
     [dispatch]
   );
+  const updateDatasourceAsync = useMemo(
+    () => (datasourceId: string, newState: unknown) => {
+      // React will synchronously update if this is triggered from a third party component,
+      // which we don't want. The timeout lets user interaction have priority, then React updates.
+      setTimeout(() => {
+        updateDatasource(datasourceId, newState);
+      }, 0);
+    },
+    [updateDatasource]
+  );
   const updateAll = useMemo(
     () => (datasourceId: string, newDatasourceState: unknown, newVisualizationState: unknown) => {
-      dispatch({
-        type: 'UPDATE_STATE',
-        subType: 'UPDATE_ALL_STATES',
-        updater: (prevState) => {
-          const updatedDatasourceState =
-            typeof newDatasourceState === 'function'
-              ? newDatasourceState(prevState.datasourceStates[datasourceId].state)
-              : newDatasourceState;
-          return {
-            ...prevState,
-            datasourceStates: {
-              ...prevState.datasourceStates,
-              [datasourceId]: {
-                state: updatedDatasourceState,
-                isLoading: false,
+      // React will synchronously update if this is triggered from a third party component,
+      // which we don't want. The timeout lets user interaction have priority, then React updates.
+      setTimeout(() => {
+        dispatch({
+          type: 'UPDATE_STATE',
+          subType: 'UPDATE_ALL_STATES',
+          updater: (prevState) => {
+            const updatedDatasourceState =
+              typeof newDatasourceState === 'function'
+                ? newDatasourceState(prevState.datasourceStates[datasourceId].state)
+                : newDatasourceState;
+            return {
+              ...prevState,
+              datasourceStates: {
+                ...prevState.datasourceStates,
+                [datasourceId]: {
+                  state: updatedDatasourceState,
+                  isLoading: false,
+                },
               },
-            },
-            visualization: {
-              ...prevState.visualization,
-              state: newVisualizationState,
-            },
-            stagedPreview: undefined,
-          };
-        },
+              visualization: {
+                ...prevState.visualization,
+                state: newVisualizationState,
+              },
+              stagedPreview: undefined,
+            };
+          },
+        });
+      }, 0);
+    },
+    [dispatch]
+  );
+  const toggleFullscreen = useMemo(
+    () => () => {
+      dispatch({
+        type: 'TOGGLE_FULLSCREEN',
       });
     },
     [dispatch]
@@ -118,6 +140,7 @@ export function LayerPanels(
             visualizationState={visualizationState}
             updateVisualization={setVisualizationState}
             updateDatasource={updateDatasource}
+            updateDatasourceAsync={updateDatasourceAsync}
             updateAll={updateAll}
             isOnlyLayer={layerIds.length === 1}
             onRemoveLayer={() => {
@@ -135,6 +158,7 @@ export function LayerPanels(
               });
               removeLayerRef(layerId);
             }}
+            toggleFullscreen={toggleFullscreen}
           />
         ) : null
       )}

x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/dimension_container.scss

@@ -8,21 +8,36 @@
   position: absolute;
   left: 0;
   animation: euiFlyout $euiAnimSpeedNormal $euiAnimSlightResistance;
+
   @include euiBreakpoint('l', 'xl') {
     top: 0 !important;
     height: 100% !important;
   }
+
   @include euiBreakpoint('xs', 's', 'm') {
     @include euiFlyout;
   }
+
+  .lnsFrameLayout__sidebar-isFullscreen & {
+    border-left: $euiBorderThin; // Force border regardless of theme in fullscreen
+    box-shadow: none;
+  }
 }
 
 .lnsDimensionContainer__footer {
   padding: $euiSizeS;
+
+  .lnsFrameLayout__sidebar-isFullscreen & {
+    display: none;
+  }
 }
 
 .lnsDimensionContainer__header {
   padding: $euiSizeS $euiSizeXS;
+
+  .lnsFrameLayout__sidebar-isFullscreen & {
+    display: none;
+  }
 }
 
 .lnsDimensionContainer__headerTitle {

x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/dimension_container.tsx

@@ -29,26 +29,33 @@ export function DimensionContainer({
   groupLabel,
   handleClose,
   panel,
+  isFullscreen,
   panelRef,
 }: {
   isOpen: boolean;
-  handleClose: () => void;
-  panel: React.ReactElement;
+  handleClose: () => boolean;
+  panel: React.ReactElement | null;
   groupLabel: string;
+  isFullscreen: boolean;
   panelRef: (el: HTMLDivElement) => void;
 }) {
   const [focusTrapIsEnabled, setFocusTrapIsEnabled] = useState(false);
 
   const closeFlyout = useCallback(() => {
-    handleClose();
-    setFocusTrapIsEnabled(false);
+    const canClose = handleClose();
+    if (canClose) {
+      setFocusTrapIsEnabled(false);
+    }
+    return canClose;
   }, [handleClose]);
 
   const closeOnEscape = useCallback(
     (event: KeyboardEvent) => {
       if (event.key === keys.ESCAPE) {
-        event.preventDefault();
-        closeFlyout();
+        const canClose = closeFlyout();
+        if (canClose) {
+          event.preventDefault();
+        }
       }
     },
     [closeFlyout]
@@ -69,7 +76,15 @@ export function DimensionContainer({
     <div ref={panelRef}>
       <EuiFocusTrap disabled={!focusTrapIsEnabled} clickOutsideDisables={true}>
         <EuiWindowEvent event="keydown" handler={closeOnEscape} />
-        <EuiOutsideClickDetector onOutsideClick={closeFlyout} isDisabled={!isOpen}>
+        <EuiOutsideClickDetector
+          onOutsideClick={() => {
+            if (isFullscreen) {
+              return;
+            }
+            closeFlyout();
+          }}
+          isDisabled={!isOpen}
+        >
           <div
             role="dialog"
             aria-labelledby="lnsDimensionContainerTitle"

x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/layer_panel.test.tsx

@@ -78,6 +78,7 @@ describe('LayerPanel', () => {
       visualizationState: 'state',
       updateVisualization: jest.fn(),
       updateDatasource: jest.fn(),
+      updateDatasourceAsync: jest.fn(),
       updateAll: jest.fn(),
       framePublicAPI: frame,
       isOnlyLayer: true,
@@ -86,6 +87,8 @@ describe('LayerPanel', () => {
       core: coreMock.createStart(),
       layerIndex: 0,
       registerNewLayerRef: jest.fn(),
+      isFullscreen: false,
+      toggleFullscreen: jest.fn(),
     };
   }
 
@@ -255,7 +258,7 @@ describe('LayerPanel', () => {
     it('should not update the visualization if the datasource is incomplete', () => {
       (generateId as jest.Mock).mockReturnValue(`newid`);
       const updateAll = jest.fn();
-      const updateDatasource = jest.fn();
+      const updateDatasourceAsync = jest.fn();
 
       mockVisualization.getConfiguration.mockReturnValue({
         groups: [
@@ -273,7 +276,7 @@ describe('LayerPanel', () => {
       const component = mountWithIntl(
         <LayerPanel
           {...getDefaultProps()}
-          updateDatasource={updateDatasource}
+          updateDatasourceAsync={updateDatasourceAsync}
           updateAll={updateAll}
         />
       );
@@ -292,15 +295,88 @@ describe('LayerPanel', () => {
           mockDatasource.renderDimensionEditor.mock.calls.length - 1
         ][1].setState;
 
+      act(() => {
+        stateFn(
+          {
+            indexPatternId: '1',
+            columns: {},
+            columnOrder: [],
+            incompleteColumns: { newId: { operationType: 'count' } },
+          },
+          { isDimensionComplete: false }
+        );
+      });
+      expect(updateAll).not.toHaveBeenCalled();
+      expect(updateDatasourceAsync).toHaveBeenCalled();
+
       act(() => {
         stateFn({
           indexPatternId: '1',
           columns: {},
           columnOrder: [],
-          incompleteColumns: { newId: { operationType: 'count' } },
         });
       });
-      expect(updateAll).not.toHaveBeenCalled();
+      expect(updateAll).toHaveBeenCalled();
+    });
+
+    it('should remove the dimension when the datasource marks it as removed', () => {
+      const updateAll = jest.fn();
+      const updateDatasource = jest.fn();
+
+      mockVisualization.getConfiguration.mockReturnValue({
+        groups: [
+          {
+            groupLabel: 'A',
+            groupId: 'a',
+            accessors: [{ columnId: 'y' }],
+            filterOperations: () => true,
+            supportsMoreColumns: true,
+            dataTestSubj: 'lnsGroup',
+          },
+        ],
+      });
+
+      const component = mountWithIntl(
+        <LayerPanel
+          {...getDefaultProps()}
+          datasourceStates={{
+            ds1: {
+              isLoading: false,
+              state: {
+                layers: [
+                  {
+                    indexPatternId: '1',
+                    columns: {
+                      y: {
+                        operationType: 'moving_average',
+                        references: ['ref'],
+                      },
+                    },
+                    columnOrder: ['y'],
+                    incompleteColumns: {},
+                  },
+                ],
+              },
+            },
+          }}
+          updateDatasource={updateDatasource}
+          updateAll={updateAll}
+        />
+      );
+
+      act(() => {
+        component.find('[data-test-subj="lnsLayerPanel-dimensionLink"]').first().simulate('click');
+      });
+      component.update();
+
+      expect(mockDatasource.renderDimensionEditor).toHaveBeenCalledWith(
+        expect.any(Element),
+        expect.objectContaining({ columnId: 'y' })
+      );
+      const stateFn =
+        mockDatasource.renderDimensionEditor.mock.calls[
+          mockDatasource.renderDimensionEditor.mock.calls.length - 1
+        ][1].setState;
 
       act(() => {
         stateFn(
@@ -308,11 +384,19 @@ describe('LayerPanel', () => {
             indexPatternId: '1',
             columns: {},
             columnOrder: [],
+            incompleteColumns: { y: { operationType: 'average' } },
           },
-          { shouldReplaceDimension: true }
+          {
+            isDimensionComplete: false,
+          }
         );
       });
       expect(updateAll).toHaveBeenCalled();
+      expect(mockVisualization.removeDimension).toHaveBeenCalledWith(
+        expect.objectContaining({
+          columnId: 'y',
+        })
+      );
     });
 
     it('should keep the DimensionContainer open when configuring a new dimension', () => {
@@ -331,6 +415,7 @@ describe('LayerPanel', () => {
             accessors: [],
             filterOperations: () => true,
             supportsMoreColumns: true,
+            enableDimensionEditor: true,
             dataTestSubj: 'lnsGroup',
           },
         ],
@@ -345,6 +430,7 @@ describe('LayerPanel', () => {
             accessors: [{ columnId: 'newid' }],
             filterOperations: () => true,
             supportsMoreColumns: false,
+            enableDimensionEditor: true,
             dataTestSubj: 'lnsGroup',
           },
         ],
@@ -357,6 +443,20 @@ describe('LayerPanel', () => {
       component.update();
 
       expect(component.find('EuiFlyoutHeader').exists()).toBe(true);
+
+      const lastArgs =
+        mockDatasource.renderDimensionEditor.mock.calls[
+          mockDatasource.renderDimensionEditor.mock.calls.length - 1
+        ][1];
+
+      // Simulate what is called by the dimension editor
+      act(() => {
+        lastArgs.setState(lastArgs.state, {
+          isDimensionComplete: true,
+        });
+      });
+
+      expect(mockVisualization.renderDimensionEditor).toHaveBeenCalled();
     });
 
     it('should close the DimensionContainer when the active visualization changes', () => {

x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/layer_panel.tsx

@@ -42,6 +42,7 @@ export function LayerPanel(
     isOnlyLayer: boolean;
     updateVisualization: StateSetter<unknown>;
     updateDatasource: (datasourceId: string, newState: unknown) => void;
+    updateDatasourceAsync: (datasourceId: string, newState: unknown) => void;
     updateAll: (
       datasourceId: string,
       newDatasourcestate: unknown,
@@ -49,6 +50,8 @@ export function LayerPanel(
     ) => void;
     onRemoveLayer: () => void;
     registerNewLayerRef: (layerId: string, instance: HTMLDivElement | null) => void;
+    toggleFullscreen: () => void;
+    isFullscreen: boolean;
   }
 ) {
   const [activeDimension, setActiveDimension] = useState<ActiveDimensionState>(
@@ -65,6 +68,8 @@ export function LayerPanel(
     activeVisualization,
     updateVisualization,
     updateDatasource,
+    toggleFullscreen,
+    isFullscreen,
   } = props;
   const datasourcePublicAPI = framePublicAPI.datasourceLayers[layerId];
 
@@ -197,9 +202,16 @@ export function LayerPanel(
     setNextFocusedButtonId,
   ]);
 
+  const isDimensionPanelOpen = Boolean(activeId);
+
   return (
     <>
-      <section tabIndex={-1} ref={registerLayerRef} className="lnsLayerPanel">
+      <section
+        tabIndex={-1}
+        ref={registerLayerRef}
+        className="lnsLayerPanel"
+        style={{ visibility: isDimensionPanelOpen ? 'hidden' : 'visible' }}
+      >
         <EuiPanel data-test-subj={`lns-layerPanel-${layerIndex}`} paddingSize="s">
           <EuiFlexGroup gutterSize="s" alignItems="flexStart" responsive={false}>
             <EuiFlexItem grow={false} className="lnsLayerPanel__settingsFlexItem">
@@ -407,9 +419,16 @@ export function LayerPanel(
 
       <DimensionContainer
         panelRef={(el) => (panelRef.current = el)}
-        isOpen={!!activeId}
+        isOpen={isDimensionPanelOpen}
+        isFullscreen={isFullscreen}
         groupLabel={activeGroup?.groupLabel || ''}
         handleClose={() => {
+          if (
+            layerDatasource.canCloseDimensionEditor &&
+            !layerDatasource.canCloseDimensionEditor(layerDatasourceState)
+          ) {
+            return false;
+          }
           if (layerDatasource.updateStateOnCloseDimension) {
             const newState = layerDatasource.updateStateOnCloseDimension({
               state: layerDatasourceState,
@@ -421,9 +440,13 @@ export function LayerPanel(
             }
           }
           setActiveDimension(initialActiveDimensionState);
+          if (isFullscreen) {
+            toggleFullscreen();
+          }
+          return true;
         }}
         panel={
-          <>
+          <div>
             {activeGroup && activeId && (
               <NativeRenderer
                 render={layerDatasource.renderDimensionEditor}
@@ -435,46 +458,51 @@ export function LayerPanel(
                   hideGrouping: activeGroup.hideGrouping,
                   filterOperations: activeGroup.filterOperations,
                   dimensionGroups: groups,
+                  toggleFullscreen,
+                  isFullscreen,
                   setState: (
                     newState: unknown,
-                    {
-                      shouldReplaceDimension,
-                      shouldRemoveDimension,
-                    }: {
-                      shouldReplaceDimension?: boolean;
-                      shouldRemoveDimension?: boolean;
-                    } = {}
+                    { isDimensionComplete = true }: { isDimensionComplete?: boolean } = {}
                   ) => {
-                    if (shouldReplaceDimension || shouldRemoveDimension) {
+                    if (allAccessors.includes(activeId)) {
+                      if (isDimensionComplete) {
+                        props.updateDatasourceAsync(datasourceId, newState);
+                      } else {
+                        // The datasource can indicate that the previously-valid column is no longer
+                        // complete, which clears the visualization. This keeps the flyout open and reuses
+                        // the previous columnId
+                        props.updateAll(
+                          datasourceId,
+                          newState,
+                          activeVisualization.removeDimension({
+                            layerId,
+                            columnId: activeId,
+                            prevState: props.visualizationState,
+                          })
+                        );
+                      }
+                    } else if (isDimensionComplete) {
                       props.updateAll(
                         datasourceId,
                         newState,
-                        shouldRemoveDimension
-                          ? activeVisualization.removeDimension({
-                              layerId,
-                              columnId: activeId,
-                              prevState: props.visualizationState,
-                            })
-                          : activeVisualization.setDimension({
-                              layerId,
-                              groupId: activeGroup.groupId,
-                              columnId: activeId,
-                              prevState: props.visualizationState,
-                            })
+                        activeVisualization.setDimension({
+                          layerId,
+                          groupId: activeGroup.groupId,
+                          columnId: activeId,
+                          prevState: props.visualizationState,
+                        })
                       );
+                      setActiveDimension({ ...activeDimension, isNew: false });
                     } else {
-                      props.updateDatasource(datasourceId, newState);
+                      props.updateDatasourceAsync(datasourceId, newState);
                     }
-                    setActiveDimension({
-                      ...activeDimension,
-                      isNew: false,
-                    });
                   },
                 }}
               />
             )}
             {activeGroup &&
               activeId &&
+              !isFullscreen &&
               !activeDimension.isNew &&
               activeVisualization.renderDimensionEditor &&
               activeGroup?.enableDimensionEditor && (
@@ -491,7 +519,7 @@ export function LayerPanel(
                   />
                 </div>
               )}
-          </>
+          </div>
         }
       />
     </>

x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/types.ts

@@ -29,6 +29,7 @@ export interface ConfigPanelWrapperProps {
     }
   >;
   core: DatasourceDimensionEditorProps['core'];
+  isFullscreen: boolean;
 }
 
 export interface LayerPanelProps {
@@ -46,6 +47,7 @@ export interface LayerPanelProps {
     }
   >;
   core: DatasourceDimensionEditorProps['core'];
+  isFullscreen: boolean;
 }
 
 export interface LayerDatasourceDropProps {

x-pack/plugins/lens/public/editor_frame_service/editor_frame/editor_frame.tsx

@@ -5,7 +5,7 @@
  * 2.0.
  */
 
-import React, { useEffect, useReducer, useState, useCallback } from 'react';
+import React, { useEffect, useReducer, useState, useCallback, useRef } from 'react';
 import { CoreStart } from 'kibana/public';
 import { isEqual } from 'lodash';
 import { PaletteRegistry } from 'src/plugins/charts/public';
@@ -30,6 +30,7 @@ import {
   applyVisualizeFieldSuggestions,
   getTopSuggestionForField,
   switchToSuggestion,
+  Suggestion,
 } from './suggestion_helpers';
 import { trackUiEvent } from '../../lens_ui_telemetry';
 import {
@@ -327,45 +328,37 @@ export function EditorFrame(props: EditorFrameProps) {
     ]
   );
 
-  const getSuggestionForField = React.useCallback(
-    (field: DragDropIdentifier) => {
-      const { activeDatasourceId, datasourceStates } = state;
-      const activeVisualizationId = state.visualization.activeId;
-      const visualizationState = state.visualization.state;
-      const { visualizationMap, datasourceMap } = props;
+  // Using a ref to prevent rerenders in the child components while keeping the latest state
+  const getSuggestionForField = useRef<(field: DragDropIdentifier) => Suggestion | undefined>();
+  getSuggestionForField.current = (field: DragDropIdentifier) => {
+    const { activeDatasourceId, datasourceStates } = state;
+    const activeVisualizationId = state.visualization.activeId;
+    const visualizationState = state.visualization.state;
+    const { visualizationMap, datasourceMap } = props;
 
-      if (!field || !activeDatasourceId) {
-        return;
-      }
+    if (!field || !activeDatasourceId) {
+      return;
+    }
 
-      return getTopSuggestionForField(
-        datasourceLayers,
-        activeVisualizationId,
-        visualizationMap,
-        visualizationState,
-        datasourceMap[activeDatasourceId],
-        datasourceStates,
-        field
-      );
-    },
-    // eslint-disable-next-line react-hooks/exhaustive-deps
-    [
-      state.visualization.state,
-      props.datasourceMap,
-      props.visualizationMap,
-      state.activeDatasourceId,
-      state.datasourceStates,
-    ]
-  );
+    return getTopSuggestionForField(
+      datasourceLayers,
+      activeVisualizationId,
+      visualizationMap,
+      visualizationState,
+      datasourceMap[activeDatasourceId],
+      datasourceStates,
+      field
+    );
+  };
 
   const hasSuggestionForField = useCallback(
-    (field: DragDropIdentifier) => getSuggestionForField(field) !== undefined,
+    (field: DragDropIdentifier) => getSuggestionForField.current!(field) !== undefined,
     [getSuggestionForField]
   );
 
   const dropOntoWorkspace = useCallback(
     (field) => {
-      const suggestion = getSuggestionForField(field);
+      const suggestion = getSuggestionForField.current!(field);
       if (suggestion) {
         trackUiEvent('drop_onto_workspace');
         switchToSuggestion(dispatch, suggestion, 'SWITCH_VISUALIZATION');
@@ -377,6 +370,7 @@ export function EditorFrame(props: EditorFrameProps) {
   return (
     <RootDragDropProvider>
       <FrameLayout
+        isFullscreen={Boolean(state.isFullscreenDatasource)}
         dataPanel={
           <DataPanelWrapper
             datasourceMap={props.datasourceMap}
@@ -414,6 +408,7 @@ export function EditorFrame(props: EditorFrameProps) {
               visualizationState={state.visualization.state}
               framePublicAPI={framePublicAPI}
               core={props.core}
+              isFullscreen={Boolean(state.isFullscreenDatasource)}
             />
           )
         }
@@ -429,11 +424,12 @@ export function EditorFrame(props: EditorFrameProps) {
               visualizationState={state.visualization.state}
               visualizationMap={props.visualizationMap}
               dispatch={dispatch}
+              isFullscreen={Boolean(state.isFullscreenDatasource)}
               ExpressionRenderer={props.ExpressionRenderer}
               core={props.core}
               plugins={props.plugins}
               visualizeTriggerFieldContext={visualizeTriggerFieldContext}
-              getSuggestionForField={getSuggestionForField}
+              getSuggestionForField={getSuggestionForField.current}
             />
           )
         }

x-pack/plugins/lens/public/editor_frame_service/editor_frame/frame_layout.scss

@@ -67,9 +67,16 @@ a tilemap in an iframe: https://github.com/elastic/kibana/issues/16457 */
   padding: $euiSize $euiSize 0;
   position: relative;
   z-index: $lnsZLevel1;
+
   &:first-child {
     padding-left: $euiSize;
   }
+
+  &.lnsFrameLayout__pageBody-isFullscreen {
+    background: $euiColorEmptyShade;
+    flex: 1;
+    padding: 0;
+  }
 }
 
 .lnsFrameLayout__sidebar {
@@ -81,6 +88,13 @@ a tilemap in an iframe: https://github.com/elastic/kibana/issues/16457 */
   position: relative;
 }
 
+.lnsFrameLayout-isFullscreen .lnsFrameLayout__sidebar--left,
+.lnsFrameLayout-isFullscreen .lnsFrameLayout__suggestionPanel {
+  // Hide the datapanel and suggestions in fullscreen mode. Using display: none does trigger
+  // a rerender when the container becomes visible again, maybe pushing offscreen is better
+  display: none;
+}
+
 .lnsFrameLayout__sidebar--right {
   flex-basis: 25%;
   background-color: lightOrDarkTheme($euiColorLightestShade, $euiColorInk);
@@ -106,3 +120,8 @@ a tilemap in an iframe: https://github.com/elastic/kibana/issues/16457 */
     }
   }
 }
+
+.lnsFrameLayout__sidebar-isFullscreen {
+  flex: 1;
+  max-width: none;
+}

x-pack/plugins/lens/public/editor_frame_service/editor_frame/frame_layout.tsx

@@ -10,23 +10,32 @@ import './frame_layout.scss';
 import React from 'react';
 import { EuiPage, EuiPageBody, EuiScreenReaderOnly } from '@elastic/eui';
 import { i18n } from '@kbn/i18n';
+import classNames from 'classnames';
 
 export interface FrameLayoutProps {
   dataPanel: React.ReactNode;
   configPanel?: React.ReactNode;
   suggestionsPanel?: React.ReactNode;
   workspacePanel?: React.ReactNode;
+  isFullscreen?: boolean;
 }
 
 export function FrameLayout(props: FrameLayoutProps) {
   return (
-    <EuiPage className="lnsFrameLayout">
+    <EuiPage
+      className={classNames('lnsFrameLayout', {
+        'lnsFrameLayout-isFullscreen': props.isFullscreen,
+      })}
+    >
       <EuiPageBody
         restrictWidth={false}
         className="lnsFrameLayout__pageContent"
         aria-labelledby="lns_ChartTitle"
       >
-        <section className="lnsFrameLayout__sidebar" aria-labelledby="dataPanelId">
+        <section
+          className={classNames('lnsFrameLayout__sidebar lnsFrameLayout__sidebar--left', {})}
+          aria-labelledby="dataPanelId"
+        >
           <EuiScreenReaderOnly>
             <h2 id="dataPanelId">
               {i18n.translate('xpack.lens.section.dataPanelLabel', {
@@ -36,7 +45,13 @@ export function FrameLayout(props: FrameLayoutProps) {
           </EuiScreenReaderOnly>
           {props.dataPanel}
         </section>
-        <section className="lnsFrameLayout__pageBody" aria-labelledby="workspaceId">
+        <section
+          className={classNames('lnsFrameLayout__pageBody', {
+            // eslint-disable-next-line @typescript-eslint/naming-convention
+            'lnsFrameLayout__pageBody-isFullscreen': props.isFullscreen,
+          })}
+          aria-labelledby="workspaceId"
+        >
           <EuiScreenReaderOnly>
             <h2 id="workspaceId">
               {i18n.translate('xpack.lens.section.workspaceLabel', {
@@ -45,10 +60,13 @@ export function FrameLayout(props: FrameLayoutProps) {
             </h2>
           </EuiScreenReaderOnly>
           {props.workspacePanel}
-          {props.suggestionsPanel}
+          <div className="lnsFrameLayout__suggestionPanel">{props.suggestionsPanel}</div>
         </section>
         <section
-          className="lnsFrameLayout__sidebar lnsFrameLayout__sidebar--right"
+          className={classNames('lnsFrameLayout__sidebar lnsFrameLayout__sidebar--right', {
+            // eslint-disable-next-line @typescript-eslint/naming-convention
+            'lnsFrameLayout__sidebar-isFullscreen': props.isFullscreen,
+          })}
           aria-labelledby="configPanel"
         >
           <EuiScreenReaderOnly>

x-pack/plugins/lens/public/editor_frame_service/editor_frame/state_management.ts

@@ -22,6 +22,7 @@ export interface EditorFrameState extends PreviewState {
   description?: string;
   stagedPreview?: PreviewState;
   activeDatasourceId: string | null;
+  isFullscreenDatasource?: boolean;
 }
 
 export type Action =
@@ -90,6 +91,9 @@ export type Action =
   | {
       type: 'SWITCH_DATASOURCE';
       newDatasourceId: string;
+    }
+  | {
+      type: 'TOGGLE_FULLSCREEN';
     };
 
 export function getActiveDatasourceIdFromDoc(doc?: Document) {
@@ -281,6 +285,8 @@ export const reducer = (state: EditorFrameState, action: Action): EditorFrameSta
         },
         stagedPreview: action.clearStagedPreview ? undefined : state.stagedPreview,
       };
+    case 'TOGGLE_FULLSCREEN':
+      return { ...state, isFullscreenDatasource: !state.isFullscreenDatasource };
     default:
       return state;
   }

x-pack/plugins/lens/public/editor_frame_service/editor_frame/suggestion_panel.tsx

@@ -200,15 +200,16 @@ export function SuggestionPanel({
             visualizationState: currentVisualizationState,
             activeData: frame.activeData,
           })
-            .filter((suggestion) => !suggestion.hide)
             .filter(
               ({
+                hide,
                 visualizationId,
                 visualizationState: suggestionVisualizationState,
                 datasourceState: suggestionDatasourceState,
                 datasourceId: suggetionDatasourceId,
               }) => {
                 return (
+                  !hide &&
                   validateDatasourceAndVisualization(
                     suggetionDatasourceId ? datasourceMap[suggetionDatasourceId] : null,
                     suggestionDatasourceState,

x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel.test.tsx

@@ -64,6 +64,8 @@ const defaultProps = {
     data: mockDataPlugin(),
   },
   getSuggestionForField: () => undefined,
+  isFullscreen: false,
+  toggleFullscreen: jest.fn(),
 };
 
 describe('workspace_panel', () => {

x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel.tsx

@@ -79,6 +79,7 @@ export interface WorkspacePanelProps {
   title?: string;
   visualizeTriggerFieldContext?: VisualizeFieldContext;
   getSuggestionForField: (field: DragDropIdentifier) => Suggestion | undefined;
+  isFullscreen: boolean;
 }
 
 interface WorkspaceState {
@@ -134,6 +135,7 @@ export const InnerWorkspacePanel = React.memo(function InnerWorkspacePanel({
   title,
   visualizeTriggerFieldContext,
   suggestionForDraggedField,
+  isFullscreen,
 }: Omit<WorkspacePanelProps, 'getSuggestionForField'> & {
   suggestionForDraggedField: Suggestion | undefined;
 }) {
@@ -346,6 +348,8 @@ export const InnerWorkspacePanel = React.memo(function InnerWorkspacePanel({
     );
   };
 
+  const element = expression !== null ? renderVisualization() : renderEmptyWorkspace();
+
   const dragDropContext = useContext(DragContext);
 
   const renderDragDrop = () => {
@@ -363,7 +367,10 @@ export const InnerWorkspacePanel = React.memo(function InnerWorkspacePanel({
       customWorkspaceRenderer()
     ) : (
       <DragDrop
-        className="lnsWorkspacePanel__dragDrop"
+        className={classNames('lnsWorkspacePanel__dragDrop', {
+          // eslint-disable-next-line @typescript-eslint/naming-convention
+          'lnsWorkspacePanel__dragDrop--fullscreen': isFullscreen,
+        })}
         dataTestSubj="lnsWorkspace"
         draggable={false}
         dropTypes={suggestionForDraggedField ? ['field_add'] : undefined}
@@ -372,8 +379,7 @@ export const InnerWorkspacePanel = React.memo(function InnerWorkspacePanel({
         order={dropProps.order}
       >
         <EuiPageContentBody className="lnsWorkspacePanelWrapper__pageContentBody">
-          {renderVisualization()}
-          {Boolean(suggestionForDraggedField) && expression !== null && renderEmptyWorkspace()}
+          {element}
         </EuiPageContentBody>
       </DragDrop>
     );
@@ -389,6 +395,7 @@ export const InnerWorkspacePanel = React.memo(function InnerWorkspacePanel({
       datasourceStates={datasourceStates}
       datasourceMap={datasourceMap}
       visualizationMap={visualizationMap}
+      isFullscreen={isFullscreen}
     >
       {renderDragDrop()}
     </WorkspacePanelWrapper>

x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel_wrapper.scss

@@ -31,6 +31,10 @@
       overflow: hidden;
     }
   }
+
+  &.lnsWorkspacePanelWrapper--fullscreen {
+    margin-bottom: 0;
+  }
 }
 
 .lnsWorkspacePanel__dragDrop {
@@ -62,6 +66,10 @@
       animation: lnsWorkspacePanel__illustrationPulseContinuous 1.5s ease-in-out 0s infinite normal forwards;
     }
   }
+
+  &.lnsWorkspacePanel__dragDrop--fullscreen {
+    border: none;
+  }
 }
 
 .lnsWorkspacePanel__emptyContent {

x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel_wrapper.test.tsx

@@ -37,6 +37,7 @@ describe('workspace_panel_wrapper', () => {
         visualizationMap={{ myVis: mockVisualization }}
         datasourceMap={{}}
         datasourceStates={{}}
+        isFullscreen={false}
       >
         <MyChild />
       </WorkspacePanelWrapper>
@@ -58,6 +59,7 @@ describe('workspace_panel_wrapper', () => {
         visualizationMap={{ myVis: { ...mockVisualization, renderToolbar: renderToolbarMock } }}
         datasourceMap={{}}
         datasourceStates={{}}
+        isFullscreen={false}
       />
     );
 

x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel_wrapper.tsx

@@ -10,6 +10,7 @@ import './workspace_panel_wrapper.scss';
 import React, { useCallback } from 'react';
 import { i18n } from '@kbn/i18n';
 import { EuiPageContent, EuiFlexGroup, EuiFlexItem, EuiScreenReaderOnly } from '@elastic/eui';
+import classNames from 'classnames';
 import { Datasource, FramePublicAPI, Visualization } from '../../../types';
 import { NativeRenderer } from '../../../native_renderer';
 import { Action } from '../state_management';
@@ -32,6 +33,7 @@ export interface WorkspacePanelWrapperProps {
       state: unknown;
     }
   >;
+  isFullscreen: boolean;
 }
 
 export function WorkspacePanelWrapper({
@@ -44,6 +46,7 @@ export function WorkspacePanelWrapper({
   visualizationMap,
   datasourceMap,
   datasourceStates,
+  isFullscreen,
 }: WorkspacePanelWrapperProps) {
   const activeVisualization = visualizationId ? visualizationMap[visualizationId] : null;
   const setVisualizationState = useCallback(
@@ -85,40 +88,42 @@ export function WorkspacePanelWrapper({
           wrap={true}
           justifyContent="spaceBetween"
         >
-          <EuiFlexItem grow={false}>
-            <EuiFlexGroup
-              gutterSize="m"
-              direction="row"
-              responsive={false}
-              wrap={true}
-              className="lnsWorkspacePanelWrapper__toolbar"
-            >
-              <EuiFlexItem grow={false}>
-                <ChartSwitch
-                  data-test-subj="lnsChartSwitcher"
-                  visualizationMap={visualizationMap}
-                  visualizationId={visualizationId}
-                  visualizationState={visualizationState}
-                  datasourceMap={datasourceMap}
-                  datasourceStates={datasourceStates}
-                  dispatch={dispatch}
-                  framePublicAPI={framePublicAPI}
-                />
-              </EuiFlexItem>
-              {activeVisualization && activeVisualization.renderToolbar && (
+          {!isFullscreen ? (
+            <EuiFlexItem grow={false}>
+              <EuiFlexGroup
+                gutterSize="m"
+                direction="row"
+                responsive={false}
+                wrap={true}
+                className="lnsWorkspacePanelWrapper__toolbar"
+              >
                 <EuiFlexItem grow={false}>
-                  <NativeRenderer
-                    render={activeVisualization.renderToolbar}
-                    nativeProps={{
-                      frame: framePublicAPI,
-                      state: visualizationState,
-                      setState: setVisualizationState,
-                    }}
+                  <ChartSwitch
+                    data-test-subj="lnsChartSwitcher"
+                    visualizationMap={visualizationMap}
+                    visualizationId={visualizationId}
+                    visualizationState={visualizationState}
+                    datasourceMap={datasourceMap}
+                    datasourceStates={datasourceStates}
+                    dispatch={dispatch}
+                    framePublicAPI={framePublicAPI}
                   />
                 </EuiFlexItem>
-              )}
-            </EuiFlexGroup>
-          </EuiFlexItem>
+                {activeVisualization && activeVisualization.renderToolbar && (
+                  <EuiFlexItem grow={false}>
+                    <NativeRenderer
+                      render={activeVisualization.renderToolbar}
+                      nativeProps={{
+                        frame: framePublicAPI,
+                        state: visualizationState,
+                        setState: setVisualizationState,
+                      }}
+                    />
+                  </EuiFlexItem>
+                )}
+              </EuiFlexGroup>
+            </EuiFlexItem>
+          ) : null}
           <EuiFlexItem grow={false}>
             {warningMessages && warningMessages.length ? (
               <WarningsPopover>{warningMessages}</WarningsPopover>
@@ -126,7 +131,11 @@ export function WorkspacePanelWrapper({
           </EuiFlexItem>
         </EuiFlexGroup>
       </div>
-      <EuiPageContent className="lnsWorkspacePanelWrapper">
+      <EuiPageContent
+        className={classNames('lnsWorkspacePanelWrapper', {
+          'lnsWorkspacePanelWrapper--fullscreen': isFullscreen,
+        })}
+      >
         <EuiScreenReaderOnly>
           <h1 id="lns_ChartTitle" data-test-subj="lns_ChartTitle">
             {title ||

x-pack/plugins/lens/public/editor_frame_service/mocks.tsx

@@ -57,6 +57,7 @@ export function createMockVisualization(): jest.Mocked<Visualization> {
     setDimension: jest.fn(),
     removeDimension: jest.fn(),
     getErrorMessages: jest.fn((_state) => undefined),
+    renderDimensionEditor: jest.fn(),
   };
 }
 

x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/dimension_editor.scss

@@ -2,7 +2,27 @@
   height: 100%;
 }
 
-.lnsIndexPatternDimensionEditor__section {
+.lnsIndexPatternDimensionEditor__header {
+  position: sticky;
+  top: 0;
+  background: $euiColorEmptyShade;
+  // Raise it above the elements that are after it in DOM order
+  z-index: $euiZLevel1;
+}
+
+.lnsIndexPatternDimensionEditor-isFullscreen {
+  position: absolute;
+  left: 0;
+  right: 0;
+  top: 0;
+  bottom: 0;
+
+  .lnsIndexPatternDimensionEditor__section {
+    height: 100%;
+  }
+}
+
+.lnsIndexPatternDimensionEditor__section--padded {
   padding: $euiSizeS;
 }
 
@@ -10,6 +30,14 @@
   background-color: $euiColorLightestShade;
 }
 
+.lnsIndexPatternDimensionEditor__section--top {
+  border-bottom: $euiBorderThin;
+}
+
+.lnsIndexPatternDimensionEditor__section--bottom {
+  border-top: $euiBorderThin;
+}
+
 .lnsIndexPatternDimensionEditor__columns {
   column-count: 2;
   column-gap: $euiSizeXL;
@@ -29,3 +57,9 @@
   padding-top: 0;
   padding-bottom: 0;
 }
+
+.lnsIndexPatternDimensionEditor__warning {
+  @include kbnThemeStyle('v7') {
+    border: none;
+  }
+}

x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/dimension_editor.tsx

@@ -6,7 +6,7 @@
  */
 
 import './dimension_editor.scss';
-import React, { useState, useMemo } from 'react';
+import React, { useState, useMemo, useCallback } from 'react';
 import { i18n } from '@kbn/i18n';
 import {
   EuiListGroup,
@@ -17,6 +17,9 @@ import {
   EuiFormLabel,
   EuiToolTip,
   EuiText,
+  EuiTabs,
+  EuiTab,
+  EuiCallOut,
 } from '@elastic/eui';
 import { IndexPatternDimensionEditorProps } from './dimension_panel';
 import { OperationSupportMatrix } from './operation_support';
@@ -91,6 +94,8 @@ export function DimensionEditor(props: DimensionEditorProps) {
     hideGrouping,
     dateRange,
     dimensionGroups,
+    toggleFullscreen,
+    isFullscreen,
   } = props;
   const services = {
     data: props.data,
@@ -101,30 +106,34 @@ export function DimensionEditor(props: DimensionEditorProps) {
   };
   const { fieldByOperation, operationWithoutField } = operationSupportMatrix;
 
+  const selectedOperationDefinition =
+    selectedColumn && operationDefinitionMap[selectedColumn.operationType];
+
   const setStateWrapper = (
     setter: IndexPatternLayer | ((prevLayer: IndexPatternLayer) => IndexPatternLayer)
   ) => {
+    const prevOperationType =
+      operationDefinitionMap[state.layers[layerId].columns[columnId]?.operationType]?.input;
+
     const hypotheticalLayer = typeof setter === 'function' ? setter(state.layers[layerId]) : setter;
     const hasIncompleteColumns = Boolean(hypotheticalLayer.incompleteColumns?.[columnId]);
-    const prevOperationType =
-      operationDefinitionMap[hypotheticalLayer.columns[columnId]?.operationType]?.input;
     setState(
       (prevState) => {
         const layer = typeof setter === 'function' ? setter(prevState.layers[layerId]) : setter;
         return mergeLayer({ state: prevState, layerId, newLayer: layer });
       },
       {
-        shouldReplaceDimension: Boolean(hypotheticalLayer.columns[columnId]),
-        // clear the dimension if there's an incomplete column pending && previous operation was a fullReference operation
-        shouldRemoveDimension: Boolean(
-          hasIncompleteColumns && prevOperationType === 'fullReference'
-        ),
+        isDimensionComplete:
+          prevOperationType === 'fullReference'
+            ? !hasIncompleteColumns
+            : Boolean(hypotheticalLayer.columns[columnId]),
       }
     );
   };
 
-  const selectedOperationDefinition =
-    selectedColumn && operationDefinitionMap[selectedColumn.operationType];
+  const setIsCloseable = (isCloseable: boolean) => {
+    setState((prevState) => ({ ...prevState, isDimensionClosePrevented: !isCloseable }));
+  };
 
   const incompleteInfo = (state.layers[layerId].incompleteColumns ?? {})[columnId];
   const incompleteOperation = incompleteInfo?.operationType;
@@ -132,14 +141,16 @@ export function DimensionEditor(props: DimensionEditorProps) {
 
   const ParamEditor = selectedOperationDefinition?.paramEditor;
 
+  const [temporaryQuickFunction, setQuickFunction] = useState(false);
+
   const possibleOperations = useMemo(() => {
     return Object.values(operationDefinitionMap)
       .filter(({ hidden }) => !hidden)
+      .filter(({ type }) => fieldByOperation[type]?.size || operationWithoutField.has(type))
       .sort((op1, op2) => {
         return op1.displayName.localeCompare(op2.displayName);
       })
-      .map((def) => def.type)
-      .filter((type) => fieldByOperation[type]?.size || operationWithoutField.has(type));
+      .map((def) => def.type);
   }, [fieldByOperation, operationWithoutField]);
 
   const [filterByOpenInitially, setFilterByOpenInitally] = useState(false);
@@ -245,37 +256,44 @@ export function DimensionEditor(props: DimensionEditorProps) {
               visualizationGroups: dimensionGroups,
               targetGroup: props.groupId,
             });
+            if (temporaryQuickFunction && newLayer.columns[columnId].operationType !== 'formula') {
+              // Only switch the tab once the formula is fully removed
+              setQuickFunction(false);
+            }
             setStateWrapper(newLayer);
             trackUiEvent(`indexpattern_dimension_operation_${operationType}`);
             return;
           } else if (!selectedColumn || !compatibleWithCurrentField) {
             const possibleFields = fieldByOperation[operationType] || new Set();
 
+            let newLayer: IndexPatternLayer;
             if (possibleFields.size === 1) {
-              setStateWrapper(
-                insertOrReplaceColumn({
-                  layer: props.state.layers[props.layerId],
-                  indexPattern: currentIndexPattern,
-                  columnId,
-                  op: operationType,
-                  field: currentIndexPattern.getFieldByName(possibleFields.values().next().value),
-                  visualizationGroups: dimensionGroups,
-                  targetGroup: props.groupId,
-                })
-              );
+              newLayer = insertOrReplaceColumn({
+                layer: props.state.layers[props.layerId],
+                indexPattern: currentIndexPattern,
+                columnId,
+                op: operationType,
+                field: currentIndexPattern.getFieldByName(possibleFields.values().next().value),
+                visualizationGroups: dimensionGroups,
+                targetGroup: props.groupId,
+              });
             } else {
-              setStateWrapper(
-                insertOrReplaceColumn({
-                  layer: props.state.layers[props.layerId],
-                  indexPattern: currentIndexPattern,
-                  columnId,
-                  op: operationType,
-                  field: undefined,
-                  visualizationGroups: dimensionGroups,
-                  targetGroup: props.groupId,
-                })
-              );
+              newLayer = insertOrReplaceColumn({
+                layer: props.state.layers[props.layerId],
+                indexPattern: currentIndexPattern,
+                columnId,
+                op: operationType,
+                field: undefined,
+                visualizationGroups: dimensionGroups,
+                targetGroup: props.groupId,
+              });
+              // );
             }
+            if (temporaryQuickFunction && newLayer.columns[columnId].operationType !== 'formula') {
+              // Only switch the tab once the formula is fully removed
+              setQuickFunction(false);
+            }
+            setStateWrapper(newLayer);
             trackUiEvent(`indexpattern_dimension_operation_${operationType}`);
             return;
           }
@@ -287,6 +305,9 @@ export function DimensionEditor(props: DimensionEditorProps) {
             return;
           }
 
+          if (temporaryQuickFunction) {
+            setQuickFunction(false);
+          }
           const newLayer = replaceColumn({
             layer: props.state.layers[props.layerId],
             indexPattern: currentIndexPattern,
@@ -315,9 +336,34 @@ export function DimensionEditor(props: DimensionEditorProps) {
       currentFieldIsInvalid
     );
 
-  return (
-    <div id={columnId}>
-      <div className="lnsIndexPatternDimensionEditor__section lnsIndexPatternDimensionEditor__section--shaded">
+  const shouldDisplayExtraOptions =
+    !currentFieldIsInvalid &&
+    !incompleteInfo &&
+    selectedColumn &&
+    selectedColumn.operationType !== 'formula';
+
+  const quickFunctions = (
+    <>
+      {temporaryQuickFunction && selectedColumn?.operationType === 'formula' && (
+        <>
+          <EuiCallOut
+            className="lnsIndexPatternDimensionEditor__warning"
+            size="s"
+            title={i18n.translate('xpack.lens.indexPattern.formulaWarning', {
+              defaultMessage: 'Formula currently applied',
+            })}
+            iconType="alert"
+            color="warning"
+          >
+            <p>
+              {i18n.translate('xpack.lens.indexPattern.formulaWarningText', {
+                defaultMessage: 'To overwrite your formula, select a quick function',
+              })}
+            </p>
+          </EuiCallOut>
+        </>
+      )}
+      <div className="lnsIndexPatternDimensionEditor__section lnsIndexPatternDimensionEditor__section--padded lnsIndexPatternDimensionEditor__section--shaded">
         <EuiFormLabel>
           {i18n.translate('xpack.lens.indexPattern.functionsLabel', {
             defaultMessage: 'Select a function',
@@ -336,7 +382,7 @@ export function DimensionEditor(props: DimensionEditorProps) {
         />
       </div>
       <EuiSpacer size="s" />
-      <div className="lnsIndexPatternDimensionEditor__section lnsIndexPatternDimensionEditor__section--shaded">
+      <div className="lnsIndexPatternDimensionEditor__section lnsIndexPatternDimensionEditor__section--padded lnsIndexPatternDimensionEditor__section--shaded">
         {!incompleteInfo &&
         selectedColumn &&
         'references' in selectedColumn &&
@@ -375,6 +421,9 @@ export function DimensionEditor(props: DimensionEditorProps) {
                     currentColumn: state.layers[layerId].columns[columnId],
                   })}
                   dimensionGroups={dimensionGroups}
+                  isFullscreen={isFullscreen}
+                  toggleFullscreen={toggleFullscreen}
+                  setIsCloseable={setIsCloseable}
                   {...services}
                 />
               );
@@ -385,7 +434,8 @@ export function DimensionEditor(props: DimensionEditorProps) {
 
         {!selectedColumn ||
         selectedOperationDefinition?.input === 'field' ||
-        (incompleteOperation && operationDefinitionMap[incompleteOperation].input === 'field') ? (
+        (incompleteOperation && operationDefinitionMap[incompleteOperation].input === 'field') ||
+        temporaryQuickFunction ? (
           <EuiFormRow
             data-test-subj="indexPattern-field-selection-row"
             label={i18n.translate('xpack.lens.indexPattern.chooseField', {
@@ -436,19 +486,20 @@ export function DimensionEditor(props: DimensionEditorProps) {
           </EuiFormRow>
         ) : null}
 
-        {!currentFieldIsInvalid && !incompleteInfo && selectedColumn && ParamEditor && (
-          <>
-            <ParamEditor
-              layer={state.layers[layerId]}
-              updateLayer={setStateWrapper}
-              columnId={columnId}
-              currentColumn={state.layers[layerId].columns[columnId]}
-              dateRange={dateRange}
-              indexPattern={currentIndexPattern}
-              operationDefinitionMap={operationDefinitionMap}
-              {...services}
-            />
-          </>
+        {shouldDisplayExtraOptions && ParamEditor && (
+          <ParamEditor
+            layer={state.layers[layerId]}
+            updateLayer={setStateWrapper}
+            columnId={columnId}
+            currentColumn={state.layers[layerId].columns[columnId]}
+            dateRange={dateRange}
+            indexPattern={currentIndexPattern}
+            operationDefinitionMap={operationDefinitionMap}
+            toggleFullscreen={toggleFullscreen}
+            isFullscreen={isFullscreen}
+            setIsCloseable={setIsCloseable}
+            {...services}
+          />
         )}
 
         {!currentFieldIsInvalid && !incompleteInfo && selectedColumn && (
@@ -546,9 +597,96 @@ export function DimensionEditor(props: DimensionEditorProps) {
       </div>
 
       <EuiSpacer size="s" />
+    </>
+  );
 
-      {!currentFieldIsInvalid && (
-        <div className="lnsIndexPatternDimensionEditor__section">
+  const formulaTab = ParamEditor ? (
+    <ParamEditor
+      layer={state.layers[layerId]}
+      updateLayer={setStateWrapper}
+      columnId={columnId}
+      currentColumn={state.layers[layerId].columns[columnId]}
+      dateRange={dateRange}
+      indexPattern={currentIndexPattern}
+      operationDefinitionMap={operationDefinitionMap}
+      toggleFullscreen={toggleFullscreen}
+      isFullscreen={isFullscreen}
+      setIsCloseable={setIsCloseable}
+      {...services}
+    />
+  ) : null;
+
+  const onFormatChange = useCallback(
+    (newFormat) => {
+      setState(
+        mergeLayer({
+          state,
+          layerId,
+          newLayer: updateColumnParam({
+            layer: state.layers[layerId],
+            columnId,
+            paramName: 'format',
+            value: newFormat,
+          }),
+        })
+      );
+    },
+    [columnId, layerId, setState, state]
+  );
+
+  return (
+    <div id={columnId}>
+      {!isFullscreen && operationSupportMatrix.operationWithoutField.has('formula') ? (
+        <EuiTabs size="s" className="lnsIndexPatternDimensionEditor__header">
+          <EuiTab
+            isSelected={temporaryQuickFunction || selectedColumn?.operationType !== 'formula'}
+            data-test-subj="lens-dimensionTabs-quickFunctions"
+            onClick={() => {
+              if (selectedColumn?.operationType === 'formula') {
+                setQuickFunction(true);
+              }
+            }}
+          >
+            {i18n.translate('xpack.lens.indexPattern.quickFunctionsLabel', {
+              defaultMessage: 'Quick functions',
+            })}
+          </EuiTab>
+          <EuiTab
+            isSelected={!temporaryQuickFunction && selectedColumn?.operationType === 'formula'}
+            data-test-subj="lens-dimensionTabs-formula"
+            onClick={() => {
+              if (selectedColumn?.operationType !== 'formula') {
+                setQuickFunction(false);
+                const newLayer = insertOrReplaceColumn({
+                  layer: props.state.layers[props.layerId],
+                  indexPattern: currentIndexPattern,
+                  columnId,
+                  op: 'formula',
+                  visualizationGroups: dimensionGroups,
+                });
+                setStateWrapper(newLayer);
+                trackUiEvent(`indexpattern_dimension_operation_formula`);
+                return;
+              } else {
+                setQuickFunction(false);
+              }
+            }}
+          >
+            {i18n.translate('xpack.lens.indexPattern.formulaLabel', {
+              defaultMessage: 'Formula',
+            })}
+          </EuiTab>
+        </EuiTabs>
+      ) : null}
+
+      {isFullscreen
+        ? formulaTab
+        : selectedOperationDefinition?.type === 'formula' && !temporaryQuickFunction
+        ? formulaTab
+        : quickFunctions}
+
+      {!isFullscreen && !currentFieldIsInvalid && !temporaryQuickFunction && (
+        <div className="lnsIndexPatternDimensionEditor__section lnsIndexPatternDimensionEditor__section--padded">
           {!incompleteInfo && selectedColumn && (
             <LabelInput
               value={selectedColumn.label}
@@ -578,7 +716,7 @@ export function DimensionEditor(props: DimensionEditorProps) {
             />
           )}
 
-          {!incompleteInfo && !hideGrouping && (
+          {!isFullscreen && !incompleteInfo && !hideGrouping && (
             <BucketNestingEditor
               layer={state.layers[props.layerId]}
               columnId={props.columnId}
@@ -589,31 +727,17 @@ export function DimensionEditor(props: DimensionEditorProps) {
             />
           )}
 
-          {selectedColumn &&
+          {!isFullscreen &&
+          selectedColumn &&
           (selectedColumn.dataType === 'number' || selectedColumn.operationType === 'range') ? (
-            <FormatSelector
-              selectedColumn={selectedColumn}
-              onChange={(newFormat) => {
-                setState(
-                  mergeLayer({
-                    state,
-                    layerId,
-                    newLayer: updateColumnParam({
-                      layer: state.layers[layerId],
-                      columnId,
-                      paramName: 'format',
-                      value: newFormat,
-                    }),
-                  })
-                );
-              }}
-            />
+            <FormatSelector selectedColumn={selectedColumn} onChange={onFormatChange} />
           ) : null}
         </div>
       )}
     </div>
   );
 }
+
 function getErrorMessage(
   selectedColumn: IndexPatternColumn | undefined,
   incompleteOperation: boolean,

x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/dimension_panel.test.tsx

@@ -208,6 +208,8 @@ describe('IndexPatternDimensionEditorPanel', () => {
       core: {} as CoreSetup,
       dimensionGroups: [],
       groupId: 'a',
+      isFullscreen: false,
+      toggleFullscreen: jest.fn(),
     };
 
     jest.clearAllMocks();
@@ -500,10 +502,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
       comboBox.prop('onChange')!([option]);
     });
 
-    expect(setState.mock.calls[0]).toEqual([
-      expect.any(Function),
-      { shouldRemoveDimension: false, shouldReplaceDimension: true },
-    ]);
+    expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
     expect(setState.mock.calls[0][0](defaultProps.state)).toEqual({
       ...initialState,
       layers: {
@@ -535,10 +534,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
       comboBox.prop('onChange')!([option]);
     });
 
-    expect(setState.mock.calls[0]).toEqual([
-      expect.any(Function),
-      { shouldRemoveDimension: false, shouldReplaceDimension: true },
-    ]);
+    expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
     expect(setState.mock.calls[0][0](defaultProps.state)).toEqual({
       ...state,
       layers: {
@@ -569,10 +565,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
       wrapper.find('button[data-test-subj="lns-indexPatternDimension-min"]').simulate('click');
     });
 
-    expect(setState.mock.calls[0]).toEqual([
-      expect.any(Function),
-      { shouldRemoveDimension: false, shouldReplaceDimension: true },
-    ]);
+    expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
     expect(setState.mock.calls[0][0](state)).toEqual({
       ...state,
       layers: {
@@ -643,10 +636,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
       wrapper.find('button[data-test-subj="lns-indexPatternDimension-min"]').simulate('click');
     });
 
-    expect(setState.mock.calls[0]).toEqual([
-      expect.any(Function),
-      { shouldRemoveDimension: false, shouldReplaceDimension: true },
-    ]);
+    expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
     expect(setState.mock.calls[0][0](state)).toEqual({
       ...state,
       layers: {
@@ -681,10 +671,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
       wrapper.find('button[data-test-subj="lns-indexPatternDimension-min"]').simulate('click');
     });
 
-    expect(setState.mock.calls[0]).toEqual([
-      expect.any(Function),
-      { shouldRemoveDimension: false, shouldReplaceDimension: true },
-    ]);
+    expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
     expect(setState.mock.calls[0][0](state)).toEqual({
       ...state,
       layers: {
@@ -750,10 +737,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
           .simulate('click');
       });
 
-      expect(setState.mock.calls[0]).toEqual([
-        expect.any(Function),
-        { shouldRemoveDimension: false, shouldReplaceDimension: true },
-      ]);
+      expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
       expect(setState.mock.calls[0][0](state)).toEqual({
         ...state,
         layers: {
@@ -879,7 +863,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
 
       expect(setState.mock.calls[0]).toEqual([
         expect.any(Function),
-        { shouldRemoveDimension: false, shouldReplaceDimension: false },
+        { isDimensionComplete: false },
       ]);
       expect(setState.mock.calls[0][0](state)).toEqual({
         ...state,
@@ -948,7 +932,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
       // Now check that the dimension gets cleaned up on state update
       expect(setState.mock.calls[0]).toEqual([
         expect.any(Function),
-        { shouldRemoveDimension: false, shouldReplaceDimension: false },
+        { isDimensionComplete: false },
       ]);
       expect(setState.mock.calls[0][0](state)).toEqual({
         ...state,
@@ -1042,10 +1026,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
       });
 
       expect(setState.mock.calls.length).toEqual(2);
-      expect(setState.mock.calls[1]).toEqual([
-        expect.any(Function),
-        { shouldRemoveDimension: false, shouldReplaceDimension: true },
-      ]);
+      expect(setState.mock.calls[1]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
       expect(setState.mock.calls[1][0](state)).toEqual({
         ...state,
         layers: {
@@ -1143,10 +1124,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
         .find('[data-test-subj="indexPattern-time-scaling-enable"]')
         .hostNodes()
         .simulate('click');
-      expect(setState.mock.calls[0]).toEqual([
-        expect.any(Function),
-        { shouldRemoveDimension: false, shouldReplaceDimension: true },
-      ]);
+      expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
       expect(setState.mock.calls[0][0](props.state)).toEqual({
         ...props.state,
         layers: {
@@ -1175,10 +1153,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
       wrapper
         .find('button[data-test-subj="lns-indexPatternDimension-count incompatible"]')
         .simulate('click');
-      expect(setState.mock.calls[0]).toEqual([
-        expect.any(Function),
-        { shouldRemoveDimension: false, shouldReplaceDimension: true },
-      ]);
+      expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
       expect(setState.mock.calls[0][0](props.state)).toEqual({
         ...props.state,
         layers: {
@@ -1205,10 +1180,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
       });
       wrapper = mount(<IndexPatternDimensionEditorComponent {...props} />);
       wrapper.find('button[data-test-subj="lns-indexPatternDimension-average"]').simulate('click');
-      expect(setState.mock.calls[0]).toEqual([
-        expect.any(Function),
-        { shouldRemoveDimension: false, shouldReplaceDimension: true },
-      ]);
+      expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
       expect(setState.mock.calls[0][0](props.state)).toEqual({
         ...props.state,
         layers: {
@@ -1239,10 +1211,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
         .prop('onChange')!(({
         target: { value: 'h' },
       } as unknown) as ChangeEvent<HTMLSelectElement>);
-      expect(setState.mock.calls[0]).toEqual([
-        expect.any(Function),
-        { shouldRemoveDimension: false, shouldReplaceDimension: true },
-      ]);
+      expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
       expect(setState.mock.calls[0][0](props.state)).toEqual({
         ...props.state,
         layers: {
@@ -1269,10 +1238,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
         .prop('onChange')!(({
         target: { value: 'h' },
       } as unknown) as ChangeEvent<HTMLSelectElement>);
-      expect(setState.mock.calls[0]).toEqual([
-        expect.any(Function),
-        { shouldRemoveDimension: false, shouldReplaceDimension: true },
-      ]);
+      expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
       expect(setState.mock.calls[0][0](props.state)).toEqual({
         ...props.state,
         layers: {
@@ -1300,10 +1266,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
         // eslint-disable-next-line @typescript-eslint/no-explicit-any
         {} as any
       );
-      expect(setState.mock.calls[0]).toEqual([
-        expect.any(Function),
-        { shouldRemoveDimension: false, shouldReplaceDimension: true },
-      ]);
+      expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
       expect(setState.mock.calls[0][0](props.state)).toEqual({
         ...props.state,
         layers: {
@@ -1593,10 +1556,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
         .find('[data-test-subj="indexPattern-filter-by-enable"]')
         .hostNodes()
         .simulate('click');
-      expect(setState.mock.calls[0]).toEqual([
-        expect.any(Function),
-        { shouldRemoveDimension: false, shouldReplaceDimension: true },
-      ]);
+      expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
       expect(setState.mock.calls[0][0](props.state)).toEqual({
         ...props.state,
         layers: {
@@ -1627,10 +1587,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
       wrapper
         .find('button[data-test-subj="lns-indexPatternDimension-count incompatible"]')
         .simulate('click');
-      expect(setState.mock.calls[0]).toEqual([
-        expect.any(Function),
-        { shouldRemoveDimension: false, shouldReplaceDimension: true },
-      ]);
+      expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
       expect(setState.mock.calls[0][0](props.state)).toEqual({
         ...props.state,
         layers: {
@@ -1656,10 +1613,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
         language: 'kuery',
         query: 'c: d',
       });
-      expect(setState.mock.calls[0]).toEqual([
-        expect.any(Function),
-        { shouldRemoveDimension: false, shouldReplaceDimension: true },
-      ]);
+      expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
       expect(setState.mock.calls[0][0](props.state)).toEqual({
         ...props.state,
         layers: {
@@ -1688,10 +1642,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
         // eslint-disable-next-line @typescript-eslint/no-explicit-any
         {} as any
       );
-      expect(setState.mock.calls[0]).toEqual([
-        expect.any(Function),
-        { shouldRemoveDimension: false, shouldReplaceDimension: true },
-      ]);
+      expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
       expect(setState.mock.calls[0][0](props.state)).toEqual({
         ...props.state,
         layers: {
@@ -1743,10 +1694,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
 
     wrapper.find('button[data-test-subj="lns-indexPatternDimension-average"]').simulate('click');
 
-    expect(setState.mock.calls[0]).toEqual([
-      expect.any(Function),
-      { shouldRemoveDimension: false, shouldReplaceDimension: false },
-    ]);
+    expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: false }]);
     expect(setState.mock.calls[0][0](defaultProps.state)).toEqual({
       ...state,
       layers: {
@@ -1810,10 +1758,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
 
     wrapper.find('button[data-test-subj="lns-indexPatternDimension-average"]').simulate('click');
 
-    expect(setState.mock.calls[0]).toEqual([
-      expect.any(Function),
-      { shouldRemoveDimension: false, shouldReplaceDimension: true },
-    ]);
+    expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
     expect(setState.mock.calls[0][0](initialState)).toEqual({
       ...initialState,
       layers: {
@@ -1838,10 +1783,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
 
     wrapper.find('button[data-test-subj="lns-indexPatternDimension-count"]').simulate('click');
 
-    expect(setState.mock.calls[0]).toEqual([
-      expect.any(Function),
-      { shouldRemoveDimension: false, shouldReplaceDimension: true },
-    ]);
+    expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
     expect(setState.mock.calls[0][0](state)).toEqual({
       ...state,
       layers: {
@@ -1975,10 +1917,7 @@ describe('IndexPatternDimensionEditorPanel', () => {
       comboBox.prop('onChange')!([option]);
     });
 
-    expect(setState.mock.calls[0]).toEqual([
-      expect.any(Function),
-      { shouldRemoveDimension: false, shouldReplaceDimension: true },
-    ]);
+    expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]);
     expect(setState.mock.calls[0][0](defaultProps.state)).toEqual({
       ...state,
       layers: {

x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/droppable/droppable.test.ts

@@ -284,6 +284,8 @@ describe('IndexPatternDimensionEditorPanel', () => {
       } as unknown) as DataPublicPluginStart,
       core: {} as CoreSetup,
       dimensionGroups: [],
+      isFullscreen: false,
+      toggleFullscreen: () => {},
     };
 
     jest.clearAllMocks();

x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/format_selector.tsx

@@ -5,7 +5,7 @@
  * 2.0.
  */
 
-import React, { useState } from 'react';
+import React, { useCallback, useMemo, useState } from 'react';
 import { i18n } from '@kbn/i18n';
 import { EuiFormRow, EuiComboBox, EuiSpacer, EuiRange } from '@elastic/eui';
 import { IndexPatternColumn } from '../indexpattern';
@@ -28,6 +28,13 @@ const supportedFormats: Record<string, { title: string }> = {
   },
 };
 
+const defaultOption = {
+  value: '',
+  label: i18n.translate('xpack.lens.indexPattern.defaultFormatLabel', {
+    defaultMessage: 'Default',
+  }),
+};
+
 interface FormatSelectorProps {
   selectedColumn: IndexPatternColumn;
   onChange: (newFormat?: { id: string; params?: Record<string, unknown> }) => void;
@@ -37,6 +44,8 @@ interface State {
   decimalPlaces: number;
 }
 
+const singleSelectionOption = { asPlainText: true };
+
 export function FormatSelector(props: FormatSelectorProps) {
   const { selectedColumn, onChange } = props;
 
@@ -51,13 +60,6 @@ export function FormatSelector(props: FormatSelectorProps) {
 
   const selectedFormat = currentFormat?.id ? supportedFormats[currentFormat.id] : undefined;
 
-  const defaultOption = {
-    value: '',
-    label: i18n.translate('xpack.lens.indexPattern.defaultFormatLabel', {
-      defaultMessage: 'Default',
-    }),
-  };
-
   const label = i18n.translate('xpack.lens.indexPattern.columnFormatLabel', {
     defaultMessage: 'Value format',
   });
@@ -66,6 +68,48 @@ export function FormatSelector(props: FormatSelectorProps) {
     defaultMessage: 'Decimals',
   });
 
+  const stableOptions = useMemo(
+    () => [
+      defaultOption,
+      ...Object.entries(supportedFormats).map(([id, format]) => ({
+        value: id,
+        label: format.title ?? id,
+      })),
+    ],
+    []
+  );
+
+  const onChangeWrapped = useCallback(
+    (choices) => {
+      if (choices.length === 0) {
+        return;
+      }
+
+      if (!choices[0].value) {
+        onChange();
+        return;
+      }
+      onChange({
+        id: choices[0].value,
+        params: { decimals: state.decimalPlaces },
+      });
+    },
+    [onChange, state.decimalPlaces]
+  );
+
+  const currentOption = useMemo(
+    () =>
+      currentFormat
+        ? [
+            {
+              value: currentFormat.id,
+              label: selectedFormat?.title ?? currentFormat.id,
+            },
+          ]
+        : [defaultOption],
+    [currentFormat, selectedFormat?.title]
+  );
+
   return (
     <>
       <EuiFormRow label={label} display="columnCompressed" fullWidth>
@@ -76,38 +120,10 @@ export function FormatSelector(props: FormatSelectorProps) {
             isClearable={false}
             data-test-subj="indexPattern-dimension-format"
             aria-label={label}
-            singleSelection={{ asPlainText: true }}
-            options={[
-              defaultOption,
-              ...Object.entries(supportedFormats).map(([id, format]) => ({
-                value: id,
-                label: format.title ?? id,
-              })),
-            ]}
-            selectedOptions={
-              currentFormat
-                ? [
-                    {
-                      value: currentFormat.id,
-                      label: selectedFormat?.title ?? currentFormat.id,
-                    },
-                  ]
-                : [defaultOption]
-            }
-            onChange={(choices) => {
-              if (choices.length === 0) {
-                return;
-              }
-
-              if (!choices[0].value) {
-                onChange();
-                return;
-              }
-              onChange({
-                id: choices[0].value,
-                params: { decimals: state.decimalPlaces },
-              });
-            }}
+            singleSelection={singleSelectionOption}
+            options={stableOptions}
+            selectedOptions={currentOption}
+            onChange={onChangeWrapped}
           />
           {currentFormat ? (
             <>

x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/reference_editor.test.tsx

@@ -51,6 +51,9 @@ describe('reference editor', () => {
       http: {} as HttpSetup,
       data: {} as DataPublicPluginStart,
       dimensionGroups: [],
+      isFullscreen: false,
+      toggleFullscreen: jest.fn(),
+      setIsCloseable: jest.fn(),
     };
   }
 

x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/reference_editor.tsx

@@ -47,10 +47,14 @@ export interface ReferenceEditorProps {
     setter: IndexPatternLayer | ((prevLayer: IndexPatternLayer) => IndexPatternLayer)
   ) => void;
   currentIndexPattern: IndexPattern;
+
   existingFields: IndexPatternPrivateState['existingFields'];
   dateRange: DateRange;
   labelAppend?: EuiFormRowProps['labelAppend'];
   dimensionGroups: VisualizationDimensionGroupConfig[];
+  isFullscreen: boolean;
+  toggleFullscreen: () => void;
+  setIsCloseable: (isCloseable: boolean) => void;
 
   // Services
   uiSettings: IUiSettingsClient;
@@ -72,6 +76,9 @@ export function ReferenceEditor(props: ReferenceEditorProps) {
     dateRange,
     labelAppend,
     dimensionGroups,
+    isFullscreen,
+    toggleFullscreen,
+    setIsCloseable,
     ...services
   } = props;
 
@@ -347,6 +354,9 @@ export function ReferenceEditor(props: ReferenceEditorProps) {
               indexPattern={currentIndexPattern}
               dateRange={dateRange}
               operationDefinitionMap={operationDefinitionMap}
+              isFullscreen={isFullscreen}
+              toggleFullscreen={toggleFullscreen}
+              setIsCloseable={setIsCloseable}
               {...services}
             />
           </>

x-pack/plugins/lens/public/indexpattern_datasource/indexpattern.tsx

@@ -323,6 +323,11 @@ export function getIndexPatternDatasource({
         domElement
       );
     },
+
+    canCloseDimensionEditor: (state) => {
+      return !state.isDimensionClosePrevented;
+    },
+
     getDropProps,
     onDrop,
 

x-pack/plugins/lens/public/indexpattern_datasource/indexpattern_suggestions.test.tsx

@@ -16,6 +16,7 @@ import {
 } from './indexpattern_suggestions';
 import { documentField } from './document_field';
 import { getFieldByNameFactory } from './pure_helpers';
+import { isEqual } from 'lodash';
 
 jest.mock('./loader');
 jest.mock('../id_generator');
@@ -867,10 +868,7 @@ describe('IndexPattern Data Source suggestions', () => {
           searchable: true,
         });
 
-        expect(suggestions).toHaveLength(1);
-        // Check that the suggestion is a single metric
-        expect(suggestions[0].table.columns).toHaveLength(1);
-        expect(suggestions[0].table.columns[0].operation.isBucketed).toBeFalsy();
+        expect(suggestions).toHaveLength(0);
       });
 
       it('appends a terms column with default size on string field', () => {
@@ -1025,6 +1023,24 @@ describe('IndexPattern Data Source suggestions', () => {
         expect(suggestions).not.toContain(expect.objectContaining({ changeType: 'extended' }));
       });
 
+      it('skips metric only suggestion when the field is already in use', () => {
+        const initialState = stateWithNonEmptyTables();
+        const suggestions = getDatasourceSuggestionsForField(initialState, '1', {
+          name: 'bytes',
+          displayName: 'bytes',
+          type: 'number',
+          aggregatable: true,
+          searchable: true,
+        });
+
+        expect(
+          suggestions.some(
+            (suggestion) =>
+              suggestion.table.changeType === 'initial' && suggestion.table.columns.length === 1
+          )
+        ).toBeFalsy();
+      });
+
       it('skips duplicates when the document-specific field is already in use', () => {
         const initialState = stateWithNonEmptyTables();
         const modifiedState: IndexPatternPrivateState = {
@@ -2344,7 +2360,7 @@ describe('IndexPattern Data Source suggestions', () => {
         );
       });
 
-      it('will skip a reduced suggestion when handling multiple references', () => {
+      it('will create reduced suggestions with all referenced children when handling references', () => {
         const initialState = testInitialState();
         const state: IndexPatternPrivateState = {
           ...initialState,
@@ -2352,7 +2368,17 @@ describe('IndexPattern Data Source suggestions', () => {
             ...initialState.layers,
             first: {
               ...initialState.layers.first,
-              columnOrder: ['date', 'metric', 'metric2', 'ref', 'ref2'],
+              columnOrder: [
+                'date',
+                'metric',
+                'metric2',
+                'ref',
+                'ref2',
+                'ref3',
+                'ref4',
+                'metric3',
+                'metric4',
+              ],
 
               columns: {
                 date: {
@@ -2384,6 +2410,20 @@ describe('IndexPattern Data Source suggestions', () => {
                   operationType: 'count',
                   sourceField: 'Records',
                 },
+                metric3: {
+                  label: '',
+                  dataType: 'number',
+                  isBucketed: false,
+                  operationType: 'count',
+                  sourceField: 'Records',
+                },
+                metric4: {
+                  label: '',
+                  dataType: 'number',
+                  isBucketed: false,
+                  operationType: 'count',
+                  sourceField: 'Records',
+                },
                 ref2: {
                   label: '',
                   dataType: 'number',
@@ -2391,22 +2431,163 @@ describe('IndexPattern Data Source suggestions', () => {
                   operationType: 'cumulative_sum',
                   references: ['metric2'],
                 },
+                ref3: {
+                  label: '',
+                  dataType: 'number',
+                  isBucketed: false,
+                  operationType: 'math',
+                  references: ['ref4', 'metric3'],
+                  params: {
+                    tinymathAst: '',
+                  },
+                },
+                ref4: {
+                  label: '',
+                  dataType: 'number',
+                  isBucketed: false,
+                  operationType: 'math',
+                  references: ['metric4'],
+                  params: {
+                    tinymathAst: '',
+                  },
+                },
               },
             },
           },
         };
 
-        const result = getSuggestionSubset(getDatasourceSuggestionsFromCurrentState(state));
+        const result = getDatasourceSuggestionsFromCurrentState(state);
 
-        expect(result).not.toContainEqual(
-          expect.objectContaining({
-            table: expect.objectContaining({
-              changeType: 'reduced',
-            }),
-          })
-        );
+        // only generate suggestions for top level metrics
+        expect(
+          result.filter((suggestion) => suggestion.table.changeType === 'reduced').length
+        ).toEqual(3);
+
+        // top level "ref" column
+        expect(
+          result.some(
+            (suggestion) =>
+              suggestion.table.changeType === 'reduced' &&
+              isEqual(suggestion.state.layers.first.columnOrder, ['ref', 'metric'])
+          )
+        ).toBeTruthy();
+
+        // top level "ref2" column
+        expect(
+          result.some(
+            (suggestion) =>
+              suggestion.table.changeType === 'reduced' &&
+              isEqual(suggestion.state.layers.first.columnOrder, ['ref2', 'metric2'])
+          )
+        ).toBeTruthy();
+
+        // top level "ref3" column
+        expect(
+          result.some(
+            (suggestion) =>
+              suggestion.table.changeType === 'reduced' &&
+              isEqual(suggestion.state.layers.first.columnOrder, [
+                'ref3',
+                'ref4',
+                'metric3',
+                'metric4',
+              ])
+          )
+        ).toBeTruthy();
       });
     });
+
+    it('will leave dangling references in place', () => {
+      const initialState = testInitialState();
+      const state: IndexPatternPrivateState = {
+        ...initialState,
+        layers: {
+          ...initialState.layers,
+          first: {
+            ...initialState.layers.first,
+            columnOrder: ['date', 'ref'],
+
+            columns: {
+              date: {
+                label: '',
+                dataType: 'date',
+                isBucketed: true,
+                operationType: 'date_histogram',
+                sourceField: 'timestamp',
+                params: { interval: 'auto' },
+              },
+              ref: {
+                label: '',
+                dataType: 'number',
+                isBucketed: false,
+                operationType: 'cumulative_sum',
+                references: ['non_existing_metric'],
+              },
+            },
+          },
+        },
+      };
+
+      const result = getDatasourceSuggestionsFromCurrentState(state);
+
+      // only generate suggestions for top level metrics
+      expect(
+        result.filter((suggestion) => suggestion.table.changeType === 'reduced').length
+      ).toEqual(1);
+
+      // top level "ref" column
+      expect(
+        result.some(
+          (suggestion) =>
+            suggestion.table.changeType === 'reduced' &&
+            isEqual(suggestion.state.layers.first.columnOrder, ['ref'])
+        )
+      ).toBeTruthy();
+    });
+
+    it('will not suggest reduced tables if there is just a referenced top level metric', () => {
+      const initialState = testInitialState();
+      const state: IndexPatternPrivateState = {
+        ...initialState,
+        layers: {
+          ...initialState.layers,
+          first: {
+            ...initialState.layers.first,
+            columnOrder: ['ref', 'metric'],
+
+            columns: {
+              ref: {
+                label: '',
+                dataType: 'number',
+                isBucketed: false,
+                operationType: 'math',
+                params: {
+                  tinymathAst: '',
+                },
+                references: ['metric'],
+              },
+              metric: {
+                label: '',
+                dataType: 'number',
+                isBucketed: false,
+                operationType: 'count',
+                sourceField: 'Records',
+              },
+            },
+          },
+        },
+      };
+
+      const result = getDatasourceSuggestionsFromCurrentState(state);
+
+      expect(
+        result.filter((suggestion) => suggestion.table.changeType === 'unchanged').length
+      ).toEqual(1);
+
+      expect(
+        result.filter((suggestion) => suggestion.table.changeType === 'reduced').length
+      ).toEqual(0);
+    });
   });
 });
 

x-pack/plugins/lens/public/indexpattern_datasource/indexpattern_suggestions.ts

@@ -5,7 +5,7 @@
  * 2.0.
  */
 
-import { flatten, minBy, pick, mapValues } from 'lodash';
+import { flatten, minBy, pick, mapValues, partition } from 'lodash';
 import { i18n } from '@kbn/i18n';
 import { generateId } from '../id_generator';
 import { DatasourceSuggestion, TableChangeType } from '../types';
@@ -20,6 +20,7 @@ import {
   OperationType,
   getExistingColumnGroups,
   isReferenced,
+  getReferencedColumnIds,
 } from './operations';
 import { hasField } from './utils';
 import {
@@ -254,9 +255,11 @@ function getExistingLayerSuggestionsForField(
     }
   }
 
-  const metricSuggestion = createMetricSuggestion(indexPattern, layerId, state, field);
-  if (metricSuggestion) {
-    suggestions.push(metricSuggestion);
+  if (!fieldInUse) {
+    const metricSuggestion = createMetricSuggestion(indexPattern, layerId, state, field);
+    if (metricSuggestion) {
+      suggestions.push(metricSuggestion);
+    }
   }
 
   return suggestions;
@@ -514,8 +517,11 @@ function createAlternativeMetricSuggestions(
 ) {
   const layer = state.layers[layerId];
   const suggestions: Array<DatasourceSuggestion<IndexPatternPrivateState>> = [];
+  const topLevelMetricColumns = layer.columnOrder.filter(
+    (columnId) => !isReferenced(layer, columnId)
+  );
 
-  layer.columnOrder.forEach((columnId) => {
+  topLevelMetricColumns.forEach((columnId) => {
     const column = layer.columns[columnId];
     if (!hasField(column)) {
       return;
@@ -580,11 +586,13 @@ function createSuggestionWithDefaultDateHistogram(
 function createSimplifiedTableSuggestions(state: IndexPatternPrivateState, layerId: string) {
   const layer = state.layers[layerId];
 
-  const [
-    availableBucketedColumns,
-    availableMetricColumns,
-    availableReferenceColumns,
-  ] = getExistingColumnGroups(layer);
+  const [availableBucketedColumns, availableMetricColumns] = partition(
+    layer.columnOrder,
+    (colId) => layer.columns[colId].isBucketed
+  );
+  const topLevelMetricColumns = availableMetricColumns.filter(
+    (columnId) => !isReferenced(layer, columnId)
+  );
 
   return flatten(
     availableBucketedColumns.map((_col, index) => {
@@ -593,46 +601,60 @@ function createSimplifiedTableSuggestions(state: IndexPatternPrivateState, layer
       const allMetricsSuggestion = {
         ...layer,
         columnOrder: [...bucketedColumns, ...availableMetricColumns],
+        noBuckets: false,
       };
 
-      if (availableBucketedColumns.length <= 1 || availableReferenceColumns.length) {
-        // Don't simplify when dealing with single-bucket table. Also don't break
-        // reference-based columns by removing buckets.
+      if (availableBucketedColumns.length <= 1) {
+        // Don't simplify when dealing with single-bucket table.
         return [];
-      } else if (availableMetricColumns.length > 1) {
-        return [{ ...layer, columnOrder: [...bucketedColumns, availableMetricColumns[0]] }];
+      } else if (topLevelMetricColumns.length > 1) {
+        return [
+          {
+            ...layer,
+            columnOrder: [
+              ...bucketedColumns,
+              topLevelMetricColumns[0],
+              ...getReferencedColumnIds(layer, topLevelMetricColumns[0]),
+            ],
+            noBuckets: false,
+          },
+        ];
       } else {
         return allMetricsSuggestion;
       }
     })
   )
     .concat(
-      availableReferenceColumns.length
-        ? []
-        : availableMetricColumns.map((columnId) => {
-            return { ...layer, columnOrder: [columnId] };
+      // if there is just a single top level metric, the unchanged suggestion will take care of this case - only split up if there are multiple metrics or at least one bucket
+      availableBucketedColumns.length > 0 || topLevelMetricColumns.length > 1
+        ? topLevelMetricColumns.map((columnId) => {
+            return {
+              ...layer,
+              columnOrder: [columnId, ...getReferencedColumnIds(layer, columnId)],
+              noBuckets: true,
+            };
           })
+        : []
     )
-    .map((updatedLayer) => {
+    .map(({ noBuckets, ...updatedLayer }) => {
       return buildSuggestion({
         state,
         layerId,
         updatedLayer,
         changeType: 'reduced',
-        label:
-          updatedLayer.columnOrder.length === 1
-            ? getMetricSuggestionTitle(updatedLayer, availableMetricColumns.length === 1)
-            : undefined,
+        label: noBuckets
+          ? getMetricSuggestionTitle(updatedLayer, availableMetricColumns.length === 1)
+          : undefined,
       });
     });
 }
 
-function getMetricSuggestionTitle(layer: IndexPatternLayer, onlyMetric: boolean) {
+function getMetricSuggestionTitle(layer: IndexPatternLayer, onlySimpleMetric: boolean) {
   const { operationType, label } = layer.columns[layer.columnOrder[0]];
   return i18n.translate('xpack.lens.indexpattern.suggestions.overallLabel', {
     defaultMessage: '{operation} overall',
     values: {
-      operation: onlyMetric ? operationDefinitionMap[operationType].displayName : label,
+      operation: onlySimpleMetric ? operationDefinitionMap[operationType].displayName : label,
     },
     description:
       'Title of a suggested chart containing only a single numerical metric calculated over all available data',

x-pack/plugins/lens/public/indexpattern_datasource/operations/__mocks__/index.ts

@@ -17,6 +17,7 @@ jest.spyOn(actualHelpers, 'insertOrReplaceColumn');
 jest.spyOn(actualHelpers, 'insertNewColumn');
 jest.spyOn(actualHelpers, 'replaceColumn');
 jest.spyOn(actualHelpers, 'getErrorMessages');
+jest.spyOn(actualHelpers, 'getColumnOrder');
 
 export const {
   getAvailableOperationsByMetadata,
@@ -48,6 +49,8 @@ export const {
   resetIncomplete,
   isOperationAllowedAsReference,
   canTransition,
+  isColumnValidAsReference,
+  getManagedColumnsFrom,
 } = actualHelpers;
 
 export const { adjustTimeScaleLabelSuffix, DEFAULT_TIME_SCALE } = actualTimeScaleUtils;

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/counter_rate.tsx

@@ -121,5 +121,23 @@ export const counterRateOperation: OperationDefinition<
   },
   timeScalingMode: 'mandatory',
   filterable: true,
+  documentation: {
+    section: 'calculation',
+    signature: i18n.translate('xpack.lens.indexPattern.counterRate.signature', {
+      defaultMessage: 'metric: number',
+    }),
+    description: i18n.translate('xpack.lens.indexPattern.counterRate.documentation', {
+      defaultMessage: `
+Calculates the rate of an ever increasing counter. This function will only yield helpful results on counter metric fields which contain a measurement of some kind monotonically growing over time.
+If the value does get smaller, it will interpret this as a counter reset. To get most precise results, \`counter_rate\` should be calculated on the \`max\` of a field.
+
+This calculation will be done separately for separate series defined by filters or top values dimensions.
+It uses the current interval when used in Formula.
+
+Example: Visualize the rate of bytes received over time by a memcached server:
+\`counter_rate(max(memcached.stats.read.bytes))\`
+      `,
+    }),
+  },
   shiftable: true,
 };

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/cumulative_sum.tsx

@@ -117,5 +117,21 @@ export const cumulativeSumOperation: OperationDefinition<
     )?.join(', ');
   },
   filterable: true,
+  documentation: {
+    section: 'calculation',
+    signature: i18n.translate('xpack.lens.indexPattern.cumulative_sum.signature', {
+      defaultMessage: 'metric: number',
+    }),
+    description: i18n.translate('xpack.lens.indexPattern.cumulativeSum.documentation', {
+      defaultMessage: `
+Calculates the cumulative sum of a metric over time, adding all previous values of a series to each value. To use this function, you need to configure a date histogram dimension as well.
+
+This calculation will be done separately for separate series defined by filters or top values dimensions.
+
+Example: Visualize the received bytes accumulated over time:
+\`cumulative_sum(sum(bytes))\`
+      `,
+    }),
+  },
   shiftable: true,
 };

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/differences.tsx

@@ -109,5 +109,22 @@ export const derivativeOperation: OperationDefinition<
   },
   timeScalingMode: 'optional',
   filterable: true,
+  documentation: {
+    section: 'calculation',
+    signature: i18n.translate('xpack.lens.indexPattern.differences.signature', {
+      defaultMessage: 'metric: number',
+    }),
+    description: i18n.translate('xpack.lens.indexPattern.differences.documentation', {
+      defaultMessage: `
+Calculates the difference to the last value of a metric over time. To use this function, you need to configure a date histogram dimension as well.
+Differences requires the data to be sequential. If your data is empty when using differences, try increasing the date histogram interval.
+
+This calculation will be done separately for separate series defined by filters or top values dimensions.
+
+Example: Visualize the change in bytes received over time:
+\`differences(sum(bytes))\`
+      `,
+    }),
+  },
   shiftable: true,
 };

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/moving_average.tsx

@@ -65,7 +65,9 @@ export const movingAverageOperation: OperationDefinition<
       validateMetadata: (meta) => meta.dataType === 'number' && !meta.isBucketed,
     },
   ],
-  operationParams: [{ name: 'window', type: 'number', required: true }],
+  operationParams: [
+    { name: 'window', type: 'number', required: false, defaultValue: WINDOW_DEFAULT_VALUE },
+  ],
   getPossibleOperation: (indexPattern) => {
     if (hasDateField(indexPattern)) {
       return {
@@ -130,6 +132,28 @@ export const movingAverageOperation: OperationDefinition<
   },
   timeScalingMode: 'optional',
   filterable: true,
+  documentation: {
+    section: 'calculation',
+    signature: i18n.translate('xpack.lens.indexPattern.moving_average.signature', {
+      defaultMessage: 'metric: number, [window]: number',
+    }),
+    description: i18n.translate('xpack.lens.indexPattern.movingAverage.documentation', {
+      defaultMessage: `
+Calculates the moving average of a metric over time, averaging the last n-th values to calculate the current value. To use this function, you need to configure a date histogram dimension as well.
+The default window value is {defaultValue}.
+
+This calculation will be done separately for separate series defined by filters or top values dimensions.
+
+Takes a named parameter \`window\` which specifies how many last values to include in the average calculation for the current value.
+
+Example: Smooth a line of measurements:
+\`moving_average(sum(bytes), window=5)\`
+      `,
+      values: {
+        defaultValue: WINDOW_DEFAULT_VALUE,
+      },
+    }),
+  },
   shiftable: true,
 };
 

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/cardinality.tsx

@@ -116,4 +116,21 @@ export const cardinalityOperation: OperationDefinition<CardinalityIndexPatternCo
       sourceField: field.name,
     };
   },
+  documentation: {
+    section: 'elasticsearch',
+    signature: i18n.translate('xpack.lens.indexPattern.cardinality.signature', {
+      defaultMessage: 'field: string',
+    }),
+    description: i18n.translate('xpack.lens.indexPattern.cardinality.documentation', {
+      defaultMessage: `
+Calculates the number of unique values of a specified field. Works for number, string, date and boolean values.
+
+Example: Calculate the number of different products:
+\`unique_count(product.name)\`
+
+Example: Calculate the number of different products from the "clothes" group:
+\`unique_count(product.name, kql='product.group=clothes')\`
+      `,
+    }),
+  },
 };

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/count.tsx

@@ -111,5 +111,20 @@ export const countOperation: OperationDefinition<CountIndexPatternColumn, 'field
   },
   timeScalingMode: 'optional',
   filterable: true,
+  documentation: {
+    section: 'elasticsearch',
+    signature: '',
+    description: i18n.translate('xpack.lens.indexPattern.count.documentation', {
+      defaultMessage: `
+Calculates the number of documents.
+
+Example: Calculate the number of documents: 
+\`count()\`
+
+Example: Calculate the number of documents matching a certain filter:
+\`count(kql='price > 500')\`
+      `,
+    }),
+  },
   shiftable: true,
 };

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/date_histogram.test.tsx

@@ -98,6 +98,9 @@ const defaultOptions = {
   http: {} as HttpSetup,
   indexPattern: indexPattern1,
   operationDefinitionMap: {},
+  isFullscreen: false,
+  toggleFullscreen: jest.fn(),
+  setIsCloseable: jest.fn(),
 };
 
 describe('date_histogram', () => {

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/filters/filters.test.tsx

@@ -28,6 +28,9 @@ const defaultProps = {
   http: {} as HttpSetup,
   indexPattern: createMockedIndexPattern(),
   operationDefinitionMap: {},
+  isFullscreen: false,
+  toggleFullscreen: jest.fn(),
+  setIsCloseable: jest.fn(),
 };
 
 // mocking random id generator function

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/formula.scss

@@ -0,0 +1,167 @@
+.lnsFormula {
+  display: flex;
+  flex-direction: column;
+
+  .lnsIndexPatternDimensionEditor-isFullscreen & {
+    height: 100%;
+  }
+
+  & > * {
+    flex: 1;
+    min-height: 0;
+  }
+
+  & > * + * {
+    border-top: $euiBorderThin;
+  }
+}
+
+.lnsFormula__editor {
+  border-bottom: $euiBorderThin;
+
+  .lnsIndexPatternDimensionEditor-isFullscreen & {
+    border-bottom: none;
+    display: flex;
+    flex-direction: column;
+  }
+
+  & > * + * {
+    border-top: $euiBorderThin;
+  }
+}
+
+.lnsFormula__editorHeader,
+.lnsFormula__editorFooter {
+  padding: $euiSizeS;
+}
+
+.lnsFormula__editorFooter {
+  // make sure docs are rendered in front of monaco
+  z-index: 1;
+  background-color: $euiColorLightestShade;
+}
+
+.lnsFormula__editorHeaderGroup,
+.lnsFormula__editorFooterGroup {
+  display: block; // Overrides EUI's styling of `display: flex` on `EuiFlexItem` components
+}
+
+.lnsFormula__editorContent {
+  position: relative;
+  height: 201px;
+}
+
+.lnsFormula__editorPlaceholder {
+  position: absolute;
+  top: 0;
+  left: $euiSize;
+  right: 0;
+  color: $euiTextSubduedColor;
+  // Matches monaco editor
+  font-family: Menlo, Monaco, 'Courier New', monospace;
+  pointer-events: none;
+}
+
+.lnsIndexPatternDimensionEditor-isFullscreen .lnsFormula__editorContent {
+  flex: 1;
+  min-height: 201px;
+}
+
+.lnsFormula__warningText + .lnsFormula__warningText {
+  margin-top: $euiSizeS;
+  border-top: $euiBorderThin;
+  padding-top: $euiSizeS;
+}
+
+.lnsFormula__editorHelp--inline {
+  align-items: center;
+  display: flex;
+  padding: $euiSizeXS;
+
+  & > * + * {
+    margin-left: $euiSizeXS;
+  }
+}
+
+.lnsFormula__editorError {
+  white-space: nowrap;
+}
+
+.lnsFormula__docs {
+  background: $euiColorEmptyShade;
+}
+
+.lnsFormula__docs--inline {
+  display: flex;
+  flex-direction: column;
+  // make sure docs are rendered in front of monaco
+  z-index: 1;
+}
+
+.lnsFormula__docsContent {
+  .lnsFormula__docs--overlay & {
+    height: 40vh;
+    width:  #{'min(75vh, 90vw)'};
+  }
+
+  .lnsFormula__docs--inline & {
+    flex: 1;
+    min-height: 0;
+  }
+
+  & > * + * {
+    border-left: $euiBorderThin;
+  }
+}
+
+.lnsFormula__docsSidebar {
+  background: $euiColorLightestShade;
+}
+
+.lnsFormula__docsSidebarInner {
+  min-height: 0;
+
+  & > * + * {
+    border-top: $euiBorderThin;
+  }
+}
+
+.lnsFormula__docsSearch {
+  padding: $euiSizeS;
+}
+
+.lnsFormula__docsNav {
+  @include euiYScroll;
+}
+
+.lnsFormula__docsNavGroup {
+  padding: $euiSizeS;
+
+  & + & {
+    border-top: $euiBorderThin;
+  }
+}
+
+.lnsFormula__docsNavGroupLink {
+  font-weight: inherit;
+}
+
+.lnsFormula__docsText {
+  @include euiYScroll;
+  padding: $euiSize;
+}
+
+.lnsFormula__docsTextGroup,
+.lnsFormula__docsTextItem {
+  margin-top: $euiSizeXXL;
+}
+
+.lnsFormula__docsTextGroup {
+  border-top: $euiBorderThin;
+  padding-top: $euiSizeXXL;
+}
+
+.lnsFormulaOverflow {
+  // Needs to be higher than the modal and all flyouts
+  z-index: $euiZLevel9 + 1;
+}

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/formula_editor.tsx

@@ -0,0 +1,791 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import React, { useCallback, useEffect, useState, useMemo } from 'react';
+import { i18n } from '@kbn/i18n';
+import {
+  EuiButtonIcon,
+  EuiButtonEmpty,
+  EuiFlexGroup,
+  EuiFlexItem,
+  EuiIcon,
+  EuiLink,
+  EuiPopover,
+  EuiText,
+  EuiToolTip,
+  EuiSpacer,
+} from '@elastic/eui';
+import useUnmount from 'react-use/lib/useUnmount';
+import { monaco } from '@kbn/monaco';
+import classNames from 'classnames';
+import { CodeEditor } from '../../../../../../../../../src/plugins/kibana_react/public';
+import type { CodeEditorProps } from '../../../../../../../../../src/plugins/kibana_react/public';
+import { useDebounceWithOptions } from '../../../../../shared_components';
+import { ParamEditorProps } from '../../index';
+import { getManagedColumnsFrom } from '../../../layer_helpers';
+import { ErrorWrapper, runASTValidation, tryToParse } from '../validation';
+import {
+  LensMathSuggestion,
+  SUGGESTION_TYPE,
+  suggest,
+  getSuggestion,
+  getSignatureHelp,
+  getHover,
+  getTokenInfo,
+  offsetToRowColumn,
+  monacoPositionToOffset,
+} from './math_completion';
+import { LANGUAGE_ID } from './math_tokenization';
+import { MemoizedFormulaHelp } from './formula_help';
+import { trackUiEvent } from '../../../../../lens_ui_telemetry';
+
+import './formula.scss';
+import { FormulaIndexPatternColumn } from '../formula';
+import { regenerateLayerFromAst } from '../parse';
+import { filterByVisibleOperation } from '../util';
+
+export const MemoizedFormulaEditor = React.memo(FormulaEditor);
+
+export function FormulaEditor({
+  layer,
+  updateLayer,
+  currentColumn,
+  columnId,
+  indexPattern,
+  operationDefinitionMap,
+  data,
+  toggleFullscreen,
+  isFullscreen,
+  setIsCloseable,
+}: ParamEditorProps<FormulaIndexPatternColumn>) {
+  const [text, setText] = useState(currentColumn.params.formula);
+  const [warnings, setWarnings] = useState<
+    Array<{ severity: monaco.MarkerSeverity; message: string }>
+  >([]);
+  const [isHelpOpen, setIsHelpOpen] = useState<boolean>(isFullscreen);
+  const [isWarningOpen, setIsWarningOpen] = useState<boolean>(false);
+  const [isWordWrapped, toggleWordWrap] = useState<boolean>(true);
+  const editorModel = React.useRef<monaco.editor.ITextModel>();
+  const overflowDiv1 = React.useRef<HTMLElement>();
+  const disposables = React.useRef<monaco.IDisposable[]>([]);
+  const editor1 = React.useRef<monaco.editor.IStandaloneCodeEditor>();
+
+  const visibleOperationsMap = useMemo(() => filterByVisibleOperation(operationDefinitionMap), [
+    operationDefinitionMap,
+  ]);
+
+  // The Monaco editor needs to have the overflowDiv in the first render. Using an effect
+  // requires a second render to work, so we are using an if statement to guarantee it happens
+  // on first render
+  if (!overflowDiv1?.current) {
+    const node1 = (overflowDiv1.current = document.createElement('div'));
+    node1.setAttribute('data-test-subj', 'lnsFormulaWidget');
+    // Monaco CSS is targeted on the monaco-editor class
+    node1.classList.add('lnsFormulaOverflow', 'monaco-editor');
+    document.body.appendChild(node1);
+  }
+
+  // Clean up the monaco editor and DOM on unmount
+  useEffect(() => {
+    const model = editorModel;
+    const allDisposables = disposables;
+    const editor1ref = editor1;
+    return () => {
+      model.current?.dispose();
+      overflowDiv1.current?.parentNode?.removeChild(overflowDiv1.current);
+      editor1ref.current?.dispose();
+      allDisposables.current?.forEach((d) => d.dispose());
+    };
+  }, []);
+
+  useUnmount(() => {
+    setIsCloseable(true);
+    // If the text is not synced, update the column.
+    if (text !== currentColumn.params.formula) {
+      updateLayer((prevLayer) => {
+        return regenerateLayerFromAst(
+          text || '',
+          prevLayer,
+          columnId,
+          currentColumn,
+          indexPattern,
+          operationDefinitionMap
+        ).newLayer;
+      });
+    }
+  });
+
+  useDebounceWithOptions(
+    () => {
+      if (!editorModel.current) return;
+
+      if (!text) {
+        setWarnings([]);
+        monaco.editor.setModelMarkers(editorModel.current, 'LENS', []);
+        if (currentColumn.params.formula) {
+          // Only submit if valid
+          const { newLayer } = regenerateLayerFromAst(
+            text || '',
+            layer,
+            columnId,
+            currentColumn,
+            indexPattern,
+            operationDefinitionMap
+          );
+          updateLayer(newLayer);
+        }
+
+        return;
+      }
+
+      let errors: ErrorWrapper[] = [];
+
+      const { root, error } = tryToParse(text, visibleOperationsMap);
+      if (error) {
+        errors = [error];
+      } else if (root) {
+        const validationErrors = runASTValidation(root, layer, indexPattern, visibleOperationsMap);
+        if (validationErrors.length) {
+          errors = validationErrors;
+        }
+      }
+
+      if (errors.length) {
+        if (currentColumn.params.isFormulaBroken) {
+          // If the formula is already broken, show the latest error message in the workspace
+          if (currentColumn.params.formula !== text) {
+            updateLayer(
+              regenerateLayerFromAst(
+                text || '',
+                layer,
+                columnId,
+                currentColumn,
+                indexPattern,
+                visibleOperationsMap
+              ).newLayer
+            );
+          }
+        }
+
+        const markers = errors.flatMap((innerError) => {
+          if (innerError.locations.length) {
+            return innerError.locations.map((location) => {
+              const startPosition = offsetToRowColumn(text, location.min);
+              const endPosition = offsetToRowColumn(text, location.max);
+              return {
+                message: innerError.message,
+                startColumn: startPosition.column + 1,
+                startLineNumber: startPosition.lineNumber,
+                endColumn: endPosition.column + 1,
+                endLineNumber: endPosition.lineNumber,
+                severity:
+                  innerError.severity === 'warning'
+                    ? monaco.MarkerSeverity.Warning
+                    : monaco.MarkerSeverity.Error,
+              };
+            });
+          } else {
+            // Parse errors return no location info
+            const startPosition = offsetToRowColumn(text, 0);
+            const endPosition = offsetToRowColumn(text, text.length - 1);
+            return [
+              {
+                message: innerError.message,
+                startColumn: startPosition.column + 1,
+                startLineNumber: startPosition.lineNumber,
+                endColumn: endPosition.column + 1,
+                endLineNumber: endPosition.lineNumber,
+                severity:
+                  innerError.severity === 'warning'
+                    ? monaco.MarkerSeverity.Warning
+                    : monaco.MarkerSeverity.Error,
+              },
+            ];
+          }
+        });
+
+        monaco.editor.setModelMarkers(editorModel.current, 'LENS', markers);
+        setWarnings(markers.map(({ severity, message }) => ({ severity, message })));
+      } else {
+        monaco.editor.setModelMarkers(editorModel.current, 'LENS', []);
+
+        // Only submit if valid
+        const { newLayer, locations } = regenerateLayerFromAst(
+          text || '',
+          layer,
+          columnId,
+          currentColumn,
+          indexPattern,
+          visibleOperationsMap
+        );
+        updateLayer(newLayer);
+
+        const managedColumns = getManagedColumnsFrom(columnId, newLayer.columns);
+        const markers: monaco.editor.IMarkerData[] = managedColumns
+          .flatMap(([id, column]) => {
+            if (locations[id]) {
+              const def = visibleOperationsMap[column.operationType];
+              if (def.getErrorMessage) {
+                const messages = def.getErrorMessage(
+                  newLayer,
+                  id,
+                  indexPattern,
+                  visibleOperationsMap
+                );
+                if (messages) {
+                  const startPosition = offsetToRowColumn(text, locations[id].min);
+                  const endPosition = offsetToRowColumn(text, locations[id].max);
+                  return [
+                    {
+                      message: messages.join(', '),
+                      startColumn: startPosition.column + 1,
+                      startLineNumber: startPosition.lineNumber,
+                      endColumn: endPosition.column + 1,
+                      endLineNumber: endPosition.lineNumber,
+                      severity: monaco.MarkerSeverity.Warning,
+                    },
+                  ];
+                }
+              }
+            }
+            return [];
+          })
+          .filter((marker) => marker);
+        setWarnings(markers.map(({ severity, message }) => ({ severity, message })));
+        monaco.editor.setModelMarkers(editorModel.current, 'LENS', markers);
+      }
+    },
+    // Make it validate on flyout open in case of a broken formula left over
+    // from a previous edit
+    { skipFirstRender: false },
+    256,
+    [text]
+  );
+
+  const errorCount = warnings.filter((marker) => marker.severity === monaco.MarkerSeverity.Error)
+    .length;
+  const warningCount = warnings.filter(
+    (marker) => marker.severity === monaco.MarkerSeverity.Warning
+  ).length;
+
+  /**
+   * The way that Monaco requests autocompletion is not intuitive, but the way we use it
+   * we fetch new suggestions in these scenarios:
+   *
+   * - If the user types one of the trigger characters, suggestions are always fetched
+   * - When the user selects the kql= suggestion, we tell Monaco to trigger new suggestions after
+   * - When the user types the first character into an empty text box, Monaco requests suggestions
+   *
+   * Monaco also triggers suggestions automatically when there are no suggestions being displayed
+   * and the user types a non-whitespace character.
+   *
+   * While suggestions are being displayed, Monaco uses an in-memory cache of the last known suggestions.
+   */
+  const provideCompletionItems = useCallback(
+    async (
+      model: monaco.editor.ITextModel,
+      position: monaco.Position,
+      context: monaco.languages.CompletionContext
+    ) => {
+      const innerText = model.getValue();
+      let aSuggestions: { list: LensMathSuggestion[]; type: SUGGESTION_TYPE } = {
+        list: [],
+        type: SUGGESTION_TYPE.FIELD,
+      };
+      const offset = monacoPositionToOffset(innerText, position);
+
+      if (context.triggerCharacter === '(') {
+        // Monaco usually inserts the end quote and reports the position is after the end quote
+        if (innerText.slice(offset - 1, offset + 1) === '()') {
+          position = position.delta(0, -1);
+        }
+        const wordUntil = model.getWordAtPosition(position.delta(0, -3));
+        if (wordUntil) {
+          // Retrieve suggestions for subexpressions
+          aSuggestions = await suggest({
+            expression: innerText,
+            zeroIndexedOffset: offset,
+            context,
+            indexPattern,
+            operationDefinitionMap: visibleOperationsMap,
+            data,
+          });
+        }
+      } else {
+        aSuggestions = await suggest({
+          expression: innerText,
+          zeroIndexedOffset: offset,
+          context,
+          indexPattern,
+          operationDefinitionMap: visibleOperationsMap,
+          data,
+        });
+      }
+
+      return {
+        suggestions: aSuggestions.list.map((s) =>
+          getSuggestion(s, aSuggestions.type, visibleOperationsMap, context.triggerCharacter)
+        ),
+      };
+    },
+    [indexPattern, visibleOperationsMap, data]
+  );
+
+  const provideSignatureHelp = useCallback(
+    async (
+      model: monaco.editor.ITextModel,
+      position: monaco.Position,
+      token: monaco.CancellationToken,
+      context: monaco.languages.SignatureHelpContext
+    ) => {
+      const innerText = model.getValue();
+      const textRange = model.getFullModelRange();
+
+      const lengthAfterPosition = model.getValueLengthInRange({
+        startLineNumber: position.lineNumber,
+        startColumn: position.column,
+        endLineNumber: textRange.endLineNumber,
+        endColumn: textRange.endColumn,
+      });
+      return getSignatureHelp(
+        model.getValue(),
+        innerText.length - lengthAfterPosition,
+        visibleOperationsMap
+      );
+    },
+    [visibleOperationsMap]
+  );
+
+  const provideHover = useCallback(
+    async (
+      model: monaco.editor.ITextModel,
+      position: monaco.Position,
+      token: monaco.CancellationToken
+    ) => {
+      const innerText = model.getValue();
+      const textRange = model.getFullModelRange();
+
+      const lengthAfterPosition = model.getValueLengthInRange({
+        startLineNumber: position.lineNumber,
+        startColumn: position.column,
+        endLineNumber: textRange.endLineNumber,
+        endColumn: textRange.endColumn,
+      });
+      return getHover(
+        model.getValue(),
+        innerText.length - lengthAfterPosition,
+        visibleOperationsMap
+      );
+    },
+    [visibleOperationsMap]
+  );
+
+  const onTypeHandler = useCallback(
+    (e: monaco.editor.IModelContentChangedEvent, editor: monaco.editor.IStandaloneCodeEditor) => {
+      if (e.isFlush || e.isRedoing || e.isUndoing) {
+        return;
+      }
+      if (e.changes.length === 1) {
+        const char = e.changes[0].text;
+        if (char !== '=' && char !== "'") {
+          return;
+        }
+        const currentPosition = e.changes[0].range;
+        if (currentPosition) {
+          const currentText = editor.getValue();
+          const offset = monacoPositionToOffset(
+            currentText,
+            new monaco.Position(currentPosition.startLineNumber, currentPosition.startColumn)
+          );
+          let tokenInfo = getTokenInfo(currentText, offset + 1);
+
+          if (!tokenInfo && char === "'") {
+            // try again this time replacing the current quote with an escaped quote
+            const line = currentText;
+            const lineEscaped = line.substring(0, offset) + "\\'" + line.substring(offset + 1);
+            tokenInfo = getTokenInfo(lineEscaped, offset + 2);
+          }
+
+          const isSingleQuoteCase = /'LENS_MATH_MARKER/;
+          // Make sure that we are only adding kql='' or lucene='', and also
+          // check that the = sign isn't inside the KQL expression like kql='='
+          if (
+            !tokenInfo ||
+            typeof tokenInfo.ast === 'number' ||
+            tokenInfo.ast.type !== 'namedArgument' ||
+            (tokenInfo.ast.name !== 'kql' && tokenInfo.ast.name !== 'lucene') ||
+            (tokenInfo.ast.value !== 'LENS_MATH_MARKER' &&
+              !isSingleQuoteCase.test(tokenInfo.ast.value))
+          ) {
+            return;
+          }
+
+          let editOperation: monaco.editor.IIdentifiedSingleEditOperation | null = null;
+          const cursorOffset = 2;
+          if (char === '=') {
+            editOperation = {
+              range: {
+                ...currentPosition,
+                // Insert after the current char
+                startColumn: currentPosition.startColumn + 1,
+                endColumn: currentPosition.startColumn + 1,
+              },
+              text: `''`,
+            };
+          }
+          if (char === "'") {
+            editOperation = {
+              range: {
+                ...currentPosition,
+                // Insert after the current char
+                startColumn: currentPosition.startColumn,
+                endColumn: currentPosition.startColumn + 1,
+              },
+              text: `\\'`,
+            };
+          }
+
+          if (editOperation) {
+            setTimeout(() => {
+              editor.executeEdits(
+                'LENS',
+                [editOperation!],
+                [
+                  // After inserting, move the cursor in between the single quotes or after the escaped quote
+                  new monaco.Selection(
+                    currentPosition.startLineNumber,
+                    currentPosition.startColumn + cursorOffset,
+                    currentPosition.startLineNumber,
+                    currentPosition.startColumn + cursorOffset
+                  ),
+                ]
+              );
+
+              // Need to move these sync to prevent race conditions between a fast user typing a single quote
+              // after an = char
+              // Timeout is required because otherwise the cursor position is not updated.
+              editor.setPosition({
+                column: currentPosition.startColumn + cursorOffset,
+                lineNumber: currentPosition.startLineNumber,
+              });
+              editor.trigger('lens', 'editor.action.triggerSuggest', {});
+            }, 0);
+          }
+        }
+      }
+    },
+    []
+  );
+
+  const codeEditorOptions: CodeEditorProps = {
+    languageId: LANGUAGE_ID,
+    value: text ?? '',
+    onChange: setText,
+    options: {
+      automaticLayout: false,
+      fontSize: 14,
+      folding: false,
+      lineNumbers: 'off',
+      scrollBeyondLastLine: false,
+      minimap: { enabled: false },
+      wordWrap: isWordWrapped ? 'on' : 'off',
+      // Disable suggestions that appear when we don't provide a default suggestion
+      wordBasedSuggestions: false,
+      autoIndent: 'brackets',
+      wrappingIndent: 'none',
+      dimension: { width: 320, height: 200 },
+      fixedOverflowWidgets: true,
+      matchBrackets: 'always',
+    },
+  };
+
+  useEffect(() => {
+    // Because the monaco model is owned by Lens, we need to manually attach and remove handlers
+    const { dispose: dispose1 } = monaco.languages.registerCompletionItemProvider(LANGUAGE_ID, {
+      triggerCharacters: ['.', '(', '=', ' ', ':', `'`],
+      provideCompletionItems,
+    });
+    const { dispose: dispose2 } = monaco.languages.registerSignatureHelpProvider(LANGUAGE_ID, {
+      signatureHelpTriggerCharacters: ['(', '='],
+      provideSignatureHelp,
+    });
+    const { dispose: dispose3 } = monaco.languages.registerHoverProvider(LANGUAGE_ID, {
+      provideHover,
+    });
+    return () => {
+      dispose1();
+      dispose2();
+      dispose3();
+    };
+  }, [provideCompletionItems, provideSignatureHelp, provideHover]);
+
+  // The Monaco editor will lazily load Monaco, which takes a render cycle to trigger. This can cause differences
+  // in the behavior of Monaco when it's first loaded and then reloaded.
+  return (
+    <div
+      className={classNames({
+        lnsIndexPatternDimensionEditor: true,
+        'lnsIndexPatternDimensionEditor-isFullscreen': isFullscreen,
+      })}
+    >
+      <div className="lnsIndexPatternDimensionEditor__section lnsIndexPatternDimensionEditor__section--shaded">
+        <div className="lnsFormula">
+          <div className="lnsFormula__editor">
+            <div className="lnsFormula__editorHeader">
+              <EuiFlexGroup alignItems="center" gutterSize="m" responsive={false}>
+                <EuiFlexItem className="lnsFormula__editorHeaderGroup">
+                  {/* TODO: Replace `bolt` with `wordWrap` icon (after latest EUI is deployed) and hook up button to enable/disable word wrapping. */}
+                  <EuiToolTip
+                    content={
+                      isWordWrapped
+                        ? i18n.translate('xpack.lens.formula.disableWordWrapLabel', {
+                            defaultMessage: 'Disable word wrap',
+                          })
+                        : i18n.translate('xpack.lens.formulaEnableWordWrapLabel', {
+                            defaultMessage: 'Enable word wrap',
+                          })
+                    }
+                    position="top"
+                  >
+                    <EuiButtonIcon
+                      iconType="bolt"
+                      display={!isWordWrapped ? 'fill' : undefined}
+                      color={'text'}
+                      aria-label={
+                        isWordWrapped
+                          ? i18n.translate('xpack.lens.formula.disableWordWrapLabel', {
+                              defaultMessage: 'Disable word wrap',
+                            })
+                          : i18n.translate('xpack.lens.formulaEnableWordWrapLabel', {
+                              defaultMessage: 'Enable word wrap',
+                            })
+                      }
+                      isSelected={!isWordWrapped}
+                      onClick={() => {
+                        editor1.current?.updateOptions({
+                          wordWrap: isWordWrapped ? 'off' : 'on',
+                        });
+                        toggleWordWrap(!isWordWrapped);
+                      }}
+                    />
+                  </EuiToolTip>
+                </EuiFlexItem>
+
+                <EuiFlexItem className="lnsFormula__editorHeaderGroup" grow={false}>
+                  {/* TODO: Replace `bolt` with `fullScreenExit` icon (after latest EUI is deployed). */}
+                  <EuiButtonEmpty
+                    onClick={() => {
+                      toggleFullscreen();
+                      // Help text opens when entering full screen, and closes when leaving full screen
+                      setIsHelpOpen(!isFullscreen);
+                      trackUiEvent('toggle_formula_fullscreen');
+                    }}
+                    iconType={isFullscreen ? 'bolt' : 'fullScreen'}
+                    size="xs"
+                    color="text"
+                    flush="right"
+                    data-test-subj="lnsFormula-fullscreen"
+                  >
+                    {isFullscreen
+                      ? i18n.translate('xpack.lens.formula.fullScreenExitLabel', {
+                          defaultMessage: 'Collapse',
+                        })
+                      : i18n.translate('xpack.lens.formula.fullScreenEnterLabel', {
+                          defaultMessage: 'Expand',
+                        })}
+                  </EuiButtonEmpty>
+                </EuiFlexItem>
+              </EuiFlexGroup>
+            </div>
+
+            <div className="lnsFormula__editorContent">
+              <CodeEditor
+                {...codeEditorOptions}
+                options={{
+                  ...codeEditorOptions.options,
+                  // Shared model and overflow node
+                  overflowWidgetsDomNode: overflowDiv1.current,
+                }}
+                editorDidMount={(editor) => {
+                  editor1.current = editor;
+                  const model = editor.getModel();
+                  if (model) {
+                    editorModel.current = model;
+                  }
+                  disposables.current.push(
+                    editor.onDidFocusEditorWidget(() => {
+                      setTimeout(() => {
+                        setIsCloseable(false);
+                      });
+                    })
+                  );
+                  disposables.current.push(
+                    editor.onDidBlurEditorWidget(() => {
+                      setTimeout(() => {
+                        setIsCloseable(true);
+                      });
+                    })
+                  );
+                  // If we ever introduce a second Monaco editor, we need to toggle
+                  // the typing handler to the active editor to maintain the cursor
+                  disposables.current.push(
+                    editor.onDidChangeModelContent((e) => {
+                      onTypeHandler(e, editor);
+                    })
+                  );
+                }}
+              />
+
+              {!text ? (
+                <div className="lnsFormula__editorPlaceholder">
+                  <EuiText color="subdued" size="s">
+                    {i18n.translate('xpack.lens.formulaPlaceholderText', {
+                      defaultMessage: 'Type a formula by combining functions with math, like:',
+                    })}
+                  </EuiText>
+                  <EuiSpacer size="s" />
+                  <pre>count() + 1</pre>
+                </div>
+              ) : null}
+            </div>
+
+            <div className="lnsFormula__editorFooter">
+              <EuiFlexGroup alignItems="center" gutterSize="m" responsive={false}>
+                <EuiFlexItem className="lnsFormula__editorFooterGroup">
+                  {isFullscreen ? (
+                    <EuiToolTip
+                      content={
+                        isHelpOpen
+                          ? i18n.translate('xpack.lens.formula.editorHelpInlineHideToolTip', {
+                              defaultMessage: 'Hide function reference',
+                            })
+                          : i18n.translate('xpack.lens.formula.editorHelpInlineShowToolTip', {
+                              defaultMessage: 'Show function reference',
+                            })
+                      }
+                      delay="long"
+                      position="top"
+                    >
+                      <EuiLink
+                        aria-label={i18n.translate('xpack.lens.formula.editorHelpInlineHideLabel', {
+                          defaultMessage: 'Hide function reference',
+                        })}
+                        className="lnsFormula__editorHelp lnsFormula__editorHelp--inline"
+                        color="text"
+                        onClick={() => setIsHelpOpen(!isHelpOpen)}
+                      >
+                        <EuiIcon type="help" />
+                        <EuiIcon type={isHelpOpen ? 'arrowDown' : 'arrowUp'} />
+                      </EuiLink>
+                    </EuiToolTip>
+                  ) : (
+                    <EuiToolTip
+                      content={i18n.translate('xpack.lens.formula.editorHelpOverlayToolTip', {
+                        defaultMessage: 'Function reference',
+                      })}
+                      position="top"
+                    >
+                      <EuiPopover
+                        panelClassName="lnsFormula__docs lnsFormula__docs--overlay"
+                        panelPaddingSize="none"
+                        anchorPosition="leftCenter"
+                        isOpen={isHelpOpen}
+                        closePopover={() => setIsHelpOpen(false)}
+                        ownFocus={false}
+                        button={
+                          <EuiButtonIcon
+                            className="lnsFormula__editorHelp lnsFormula__editorHelp--overlay"
+                            onClick={() => setIsHelpOpen(!isHelpOpen)}
+                            iconType="help"
+                            color="text"
+                            size="s"
+                            aria-label={i18n.translate(
+                              'xpack.lens.formula.editorHelpInlineShowToolTip',
+                              {
+                                defaultMessage: 'Show function reference',
+                              }
+                            )}
+                          />
+                        }
+                      >
+                        <MemoizedFormulaHelp
+                          isFullscreen={isFullscreen}
+                          indexPattern={indexPattern}
+                          operationDefinitionMap={visibleOperationsMap}
+                        />
+                      </EuiPopover>
+                    </EuiToolTip>
+                  )}
+                </EuiFlexItem>
+
+                {errorCount || warningCount ? (
+                  <EuiFlexItem className="lnsFormula__editorFooterGroup" grow={false}>
+                    <EuiPopover
+                      ownFocus={false}
+                      isOpen={isWarningOpen}
+                      closePopover={() => setIsWarningOpen(false)}
+                      button={
+                        <EuiButtonEmpty
+                          color={errorCount ? 'danger' : 'warning'}
+                          className="lnsFormula__editorError"
+                          iconType="alert"
+                          size="xs"
+                          flush="right"
+                          onClick={() => {
+                            setIsWarningOpen(!isWarningOpen);
+                          }}
+                        >
+                          {errorCount
+                            ? i18n.translate('xpack.lens.formulaErrorCount', {
+                                defaultMessage:
+                                  '{count} {count, plural, one {error} other {errors}}',
+                                values: { count: errorCount },
+                              })
+                            : null}
+                          {warningCount
+                            ? i18n.translate('xpack.lens.formulaWarningCount', {
+                                defaultMessage:
+                                  '{count} {count, plural, one {warning} other {warnings}}',
+                                values: { count: warningCount },
+                              })
+                            : null}
+                        </EuiButtonEmpty>
+                      }
+                    >
+                      {warnings.map(({ message, severity }, index) => (
+                        <div key={index} className="lnsFormula__warningText">
+                          <EuiText
+                            size="s"
+                            color={
+                              severity === monaco.MarkerSeverity.Warning ? 'warning' : 'danger'
+                            }
+                          >
+                            {message}
+                          </EuiText>
+                        </div>
+                      ))}
+                    </EuiPopover>
+                  </EuiFlexItem>
+                ) : null}
+              </EuiFlexGroup>
+            </div>
+          </div>
+
+          {isFullscreen && isHelpOpen ? (
+            <div className="lnsFormula__docs lnsFormula__docs--inline">
+              <MemoizedFormulaHelp
+                isFullscreen={isFullscreen}
+                indexPattern={indexPattern}
+                operationDefinitionMap={visibleOperationsMap}
+              />
+            </div>
+          ) : null}
+        </div>
+      </div>
+    </div>
+  );
+}

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/formula_help.tsx

@@ -0,0 +1,469 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import React, { useEffect, useRef, useState, useMemo } from 'react';
+import { i18n } from '@kbn/i18n';
+import {
+  EuiFlexGroup,
+  EuiFlexItem,
+  EuiLink,
+  EuiPopoverTitle,
+  EuiText,
+  EuiListGroupItem,
+  EuiListGroup,
+  EuiTitle,
+  EuiFieldSearch,
+  EuiHighlight,
+} from '@elastic/eui';
+import { Markdown } from '../../../../../../../../../src/plugins/kibana_react/public';
+import { IndexPattern } from '../../../../types';
+import { tinymathFunctions } from '../util';
+import { getPossibleFunctions } from './math_completion';
+import { hasFunctionFieldArgument } from '../validation';
+
+import type {
+  GenericOperationDefinition,
+  IndexPatternColumn,
+  OperationDefinition,
+  ParamEditorProps,
+} from '../../index';
+import type { FormulaIndexPatternColumn } from '../formula';
+
+function FormulaHelp({
+  indexPattern,
+  operationDefinitionMap,
+  isFullscreen,
+}: {
+  indexPattern: IndexPattern;
+  operationDefinitionMap: Record<string, GenericOperationDefinition>;
+  isFullscreen: boolean;
+}) {
+  const [selectedFunction, setSelectedFunction] = useState<string | undefined>();
+  const scrollTargets = useRef<Record<string, HTMLElement>>({});
+
+  useEffect(() => {
+    if (selectedFunction && scrollTargets.current[selectedFunction]) {
+      scrollTargets.current[selectedFunction].scrollIntoView();
+    }
+  }, [selectedFunction]);
+
+  const helpGroups: Array<{
+    label: string;
+    description?: string;
+    items: Array<{ label: string; description?: JSX.Element }>;
+  }> = [];
+
+  helpGroups.push({
+    label: i18n.translate('xpack.lens.formulaDocumentationHeading', {
+      defaultMessage: 'How it works',
+    }),
+    items: [],
+  });
+
+  helpGroups.push({
+    label: i18n.translate('xpack.lens.formulaDocumentation.elasticsearchSection', {
+      defaultMessage: 'Elasticsearch',
+    }),
+    description: i18n.translate('xpack.lens.formulaDocumentation.elasticsearchSectionDescription', {
+      defaultMessage:
+        'These functions will be executed on the raw documents for each row of the resulting table, aggregating all documents matching the break down dimensions into a single value.',
+    }),
+    items: [],
+  });
+
+  const availableFunctions = getPossibleFunctions(indexPattern);
+
+  // Es aggs
+  helpGroups[1].items.push(
+    ...availableFunctions
+      .filter(
+        (key) =>
+          key in operationDefinitionMap &&
+          operationDefinitionMap[key].documentation?.section === 'elasticsearch'
+      )
+      .sort()
+      .map((key) => ({
+        label: key,
+        description: (
+          <>
+            <h3>
+              {key}({operationDefinitionMap[key].documentation?.signature})
+            </h3>
+
+            {operationDefinitionMap[key].documentation?.description ? (
+              <Markdown markdown={operationDefinitionMap[key].documentation!.description} />
+            ) : null}
+          </>
+        ),
+      }))
+  );
+
+  helpGroups.push({
+    label: i18n.translate('xpack.lens.formulaDocumentation.columnCalculationSection', {
+      defaultMessage: 'Column-wise calculation',
+    }),
+    description: i18n.translate(
+      'xpack.lens.formulaDocumentation.columnCalculationSectionDescription',
+      {
+        defaultMessage:
+          'These functions will be executed for reach row of the resulting table, using data from cells from other rows as well as the current value.',
+      }
+    ),
+    items: [],
+  });
+
+  // Calculations aggs
+  helpGroups[2].items.push(
+    ...availableFunctions
+      .filter(
+        (key) =>
+          key in operationDefinitionMap &&
+          operationDefinitionMap[key].documentation?.section === 'calculation'
+      )
+      .sort()
+      .map((key) => ({
+        label: key,
+        description: (
+          <>
+            <h3>
+              {key}({operationDefinitionMap[key].documentation?.signature})
+            </h3>
+
+            {operationDefinitionMap[key].documentation?.description ? (
+              <Markdown markdown={operationDefinitionMap[key].documentation!.description} />
+            ) : null}
+          </>
+        ),
+        checked:
+          selectedFunction === `${key}: ${operationDefinitionMap[key].displayName}`
+            ? ('on' as const)
+            : undefined,
+      }))
+  );
+
+  helpGroups.push({
+    label: i18n.translate('xpack.lens.formulaDocumentation.mathSection', {
+      defaultMessage: 'Math',
+    }),
+    description: i18n.translate('xpack.lens.formulaDocumentation.mathSectionDescription', {
+      defaultMessage:
+        'These functions will be executed for reach row of the resulting table using single values from the same row calculated using other functions.',
+    }),
+    items: [],
+  });
+
+  const tinymathFns = useMemo(() => {
+    return getPossibleFunctions(indexPattern)
+      .filter((key) => key in tinymathFunctions)
+      .sort()
+      .map((key) => {
+        const [description, examples] = tinymathFunctions[key].help.split(`\`\`\``);
+        return {
+          label: key,
+          description: description.replace(/\n/g, '\n\n'),
+          examples: examples ? `\`\`\`${examples}\`\`\`` : '',
+        };
+      });
+  }, [indexPattern]);
+
+  helpGroups[3].items.push(
+    ...tinymathFns.map(({ label, description, examples }) => {
+      return {
+        label,
+        description: (
+          <>
+            <h3>{getFunctionSignatureLabel(label, operationDefinitionMap)}</h3>
+
+            <Markdown markdown={`${description}${examples}`} />
+          </>
+        ),
+      };
+    })
+  );
+
+  const [searchText, setSearchText] = useState('');
+
+  const normalizedSearchText = searchText.trim().toLocaleLowerCase();
+
+  const filteredHelpGroups = helpGroups
+    .map((group) => {
+      const items = group.items.filter((helpItem) => {
+        return (
+          !normalizedSearchText || helpItem.label.toLocaleLowerCase().includes(normalizedSearchText)
+        );
+      });
+      return { ...group, items };
+    })
+    .filter((group) => {
+      if (group.items.length > 0 || !normalizedSearchText) {
+        return true;
+      }
+      return group.label.toLocaleLowerCase().includes(normalizedSearchText);
+    });
+
+  return (
+    <>
+      <EuiPopoverTitle className="lnsFormula__docsHeader" paddingSize="s">
+        {i18n.translate('xpack.lens.formulaDocumentation.header', {
+          defaultMessage: 'Formula reference',
+        })}
+      </EuiPopoverTitle>
+
+      <EuiFlexGroup
+        className="lnsFormula__docsContent"
+        gutterSize="none"
+        responsive={false}
+        alignItems="stretch"
+      >
+        <EuiFlexItem className="lnsFormula__docsSidebar" grow={1}>
+          <EuiFlexGroup
+            className="lnsFormula__docsSidebarInner"
+            direction="column"
+            gutterSize="none"
+            responsive={false}
+          >
+            <EuiFlexItem className="lnsFormula__docsSearch" grow={false}>
+              <EuiFieldSearch
+                value={searchText}
+                onChange={(e) => {
+                  setSearchText(e.target.value);
+                }}
+                placeholder={i18n.translate('xpack.lens.formulaSearchPlaceholder', {
+                  defaultMessage: 'Search functions',
+                })}
+              />
+            </EuiFlexItem>
+
+            <EuiFlexItem className="lnsFormula__docsNav">
+              {filteredHelpGroups.map((helpGroup, index) => {
+                return (
+                  <nav className="lnsFormula__docsNavGroup" key={helpGroup.label}>
+                    <EuiTitle size="xxs">
+                      <h6>
+                        <EuiLink
+                          className="lnsFormula__docsNavGroupLink"
+                          color="text"
+                          onClick={() => {
+                            setSelectedFunction(helpGroup.label);
+                          }}
+                        >
+                          <EuiHighlight search={searchText}>{helpGroup.label}</EuiHighlight>
+                        </EuiLink>
+                      </h6>
+                    </EuiTitle>
+
+                    <EuiListGroup gutterSize="none">
+                      {helpGroup.items.map((helpItem) => {
+                        return (
+                          <EuiListGroupItem
+                            key={helpItem.label}
+                            label={
+                              <EuiHighlight search={searchText}>{helpItem.label}</EuiHighlight>
+                            }
+                            size="s"
+                            onClick={() => {
+                              setSelectedFunction(helpItem.label);
+                            }}
+                          />
+                        );
+                      })}
+                    </EuiListGroup>
+                  </nav>
+                );
+              })}
+            </EuiFlexItem>
+          </EuiFlexGroup>
+        </EuiFlexItem>
+
+        <EuiFlexItem className="lnsFormula__docsText" grow={2}>
+          <EuiText size="s">
+            <section
+              className="lnsFormula__docsTextIntro"
+              ref={(el) => {
+                if (el) {
+                  scrollTargets.current[helpGroups[0].label] = el;
+                }
+              }}
+            >
+              <Markdown
+                markdown={i18n.translate('xpack.lens.formulaDocumentation', {
+                  defaultMessage: `
+## How it works
+
+Lens formulas let you do math using a combination of Elasticsearch aggregations and
+math functions. There are three main types of functions:
+
+* Elasticsearch metrics, like \`sum(bytes)\`
+* Time series functions use Elasticsearch metrics as input, like \`cumulative_sum()\`
+* Math functions like \`round()\`
+
+An example formula that uses all of these:
+
+\`\`\`
+round(100 * moving_average(
+  average(cpu.load.pct),
+  window=10,
+  kql='datacenter.name: east*'
+))
+\`\`\`
+
+Elasticsearch functions take a field name, which can be in quotes. \`sum(bytes)\` is the same
+as \`sum("bytes")\`.
+
+Some functions take named arguments, like moving_average(count(), window=5)
+
+Elasticsearch metrics can be filtered using KQL or Lucene syntax. To add a filter, use the named
+parameter \`kql='field: value'\` or \`lucene=''\`. Always use single quotes when writing KQL or Lucene
+queries. If your search has a single quote in it, use a backslash to escape, like: \`kql='Women's'\'
+
+Math functions can take positional arguments, like pow(count(), 3) is the same as count() * count() * count()
+
+Use the symbols +, -, /, and * to perform basic math.
+                  `,
+                  description:
+                    'Text is in markdown. Do not translate function names or field names like sum(bytes)',
+                })}
+              />
+            </section>
+
+            {helpGroups.slice(1).map((helpGroup, index) => {
+              return (
+                <section
+                  className="lnsFormula__docsTextGroup"
+                  key={helpGroup.label}
+                  ref={(el) => {
+                    if (el) {
+                      scrollTargets.current[helpGroup.label] = el;
+                    }
+                  }}
+                >
+                  <h2>{helpGroup.label}</h2>
+
+                  <p>{helpGroup.description}</p>
+
+                  {helpGroups[index + 1].items.map((helpItem) => {
+                    return (
+                      <article
+                        className="lnsFormula__docsTextItem"
+                        key={helpItem.label}
+                        ref={(el) => {
+                          if (el) {
+                            scrollTargets.current[helpItem.label] = el;
+                          }
+                        }}
+                      >
+                        {helpItem.description}
+                      </article>
+                    );
+                  })}
+                </section>
+              );
+            })}
+          </EuiText>
+        </EuiFlexItem>
+      </EuiFlexGroup>
+    </>
+  );
+}
+
+export const MemoizedFormulaHelp = React.memo(FormulaHelp);
+
+export function getFunctionSignatureLabel(
+  name: string,
+  operationDefinitionMap: ParamEditorProps<FormulaIndexPatternColumn>['operationDefinitionMap'],
+  firstParam?: { label: string | [number, number] } | null
+): string {
+  if (tinymathFunctions[name]) {
+    return `${name}(${tinymathFunctions[name].positionalArguments
+      .map(({ name: argName, optional, type }) => `[${argName}]${optional ? '?' : ''}: ${type}`)
+      .join(', ')})`;
+  }
+  if (operationDefinitionMap[name]) {
+    const def = operationDefinitionMap[name];
+    let extraArgs = '';
+    if (def.filterable) {
+      extraArgs += hasFunctionFieldArgument(name) || 'operationParams' in def ? ',' : '';
+      extraArgs += i18n.translate('xpack.lens.formula.kqlExtraArguments', {
+        defaultMessage: '[kql]?: string, [lucene]?: string',
+      });
+    }
+    return `${name}(${def.documentation?.signature}${extraArgs})`;
+  }
+  return '';
+}
+
+function getFunctionArgumentsStringified(
+  params: Required<
+    OperationDefinition<IndexPatternColumn, 'field' | 'fullReference'>
+  >['operationParams']
+) {
+  return params
+    .map(
+      ({ name, type: argType, defaultValue = 5 }) =>
+        `${name}=${argType === 'string' ? `"${defaultValue}"` : defaultValue}`
+    )
+    .join(', ');
+}
+
+/**
+ * Get an array of strings containing all possible information about a specific
+ * operation type: examples and infos.
+ */
+export function getHelpTextContent(
+  type: string,
+  operationDefinitionMap: ParamEditorProps<FormulaIndexPatternColumn>['operationDefinitionMap']
+): { description: string; examples: string[] } {
+  const definition = operationDefinitionMap[type];
+  const description = definition.documentation?.description ?? '';
+
+  // as for the time being just add examples text.
+  // Later will enrich with more information taken from the operation definitions.
+  const examples: string[] = [];
+  // If the description already contain examples skip it
+  if (!/Example/.test(description)) {
+    if (!hasFunctionFieldArgument(type)) {
+      // ideally this should have the same example automation as the operations below
+      examples.push(`${type}()`);
+      return { description, examples };
+    }
+    if (definition.input === 'field') {
+      const mandatoryArgs = definition.operationParams?.filter(({ required }) => required) || [];
+      if (mandatoryArgs.length === 0) {
+        examples.push(`${type}(bytes)`);
+      }
+      if (mandatoryArgs.length) {
+        const additionalArgs = getFunctionArgumentsStringified(mandatoryArgs);
+        examples.push(`${type}(bytes, ${additionalArgs})`);
+      }
+      if (
+        definition.operationParams &&
+        mandatoryArgs.length !== definition.operationParams.length
+      ) {
+        const additionalArgs = getFunctionArgumentsStringified(definition.operationParams);
+        examples.push(`${type}(bytes, ${additionalArgs})`);
+      }
+    }
+    if (definition.input === 'fullReference') {
+      const mandatoryArgs = definition.operationParams?.filter(({ required }) => required) || [];
+      if (mandatoryArgs.length === 0) {
+        examples.push(`${type}(sum(bytes))`);
+      }
+      if (mandatoryArgs.length) {
+        const additionalArgs = getFunctionArgumentsStringified(mandatoryArgs);
+        examples.push(`${type}(sum(bytes), ${additionalArgs})`);
+      }
+      if (
+        definition.operationParams &&
+        mandatoryArgs.length !== definition.operationParams.length
+      ) {
+        const additionalArgs = getFunctionArgumentsStringified(definition.operationParams);
+        examples.push(`${type}(sum(bytes), ${additionalArgs})`);
+      }
+    }
+  }
+  return { description, examples };
+}

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/index.ts

@@ -0,0 +1,8 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+export * from './formula_editor';

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/math_completion.test.ts

@@ -0,0 +1,386 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import { parse } from '@kbn/tinymath';
+import { monaco } from '@kbn/monaco';
+import { createMockedIndexPattern } from '../../../../mocks';
+import { GenericOperationDefinition } from '../../index';
+import type { IndexPatternField } from '../../../../types';
+import type { OperationMetadata } from '../../../../../types';
+import { dataPluginMock } from '../../../../../../../../../src/plugins/data/public/mocks';
+import { tinymathFunctions } from '../util';
+import {
+  getSignatureHelp,
+  getHover,
+  suggest,
+  monacoPositionToOffset,
+  getInfoAtZeroIndexedPosition,
+} from './math_completion';
+
+const buildGenericColumn = (type: string) => {
+  return ({ field }: { field?: IndexPatternField }) => {
+    return {
+      label: type,
+      dataType: 'number',
+      operationType: type,
+      sourceField: field?.name ?? undefined,
+      isBucketed: false,
+      scale: 'ratio',
+      timeScale: false,
+    };
+  };
+};
+
+const numericOperation = () => ({ dataType: 'number', isBucketed: false });
+const stringOperation = () => ({ dataType: 'string', isBucketed: true });
+
+// Only one of each type is needed
+const operationDefinitionMap: Record<string, GenericOperationDefinition> = {
+  sum: ({
+    type: 'sum',
+    input: 'field',
+    buildColumn: buildGenericColumn('sum'),
+    getPossibleOperationForField: (field: IndexPatternField) =>
+      field.type === 'number' ? numericOperation() : null,
+    documentation: {
+      section: 'elasticsearch',
+      signature: 'field: string',
+      description: 'description',
+    },
+  } as unknown) as GenericOperationDefinition,
+  count: ({
+    type: 'count',
+    input: 'field',
+    buildColumn: buildGenericColumn('count'),
+    getPossibleOperationForField: (field: IndexPatternField) =>
+      field.name === 'Records' ? numericOperation() : null,
+  } as unknown) as GenericOperationDefinition,
+  last_value: ({
+    type: 'last_value',
+    input: 'field',
+    buildColumn: buildGenericColumn('last_value'),
+    getPossibleOperationForField: (field: IndexPatternField) => ({
+      dataType: field.type,
+      isBucketed: false,
+    }),
+  } as unknown) as GenericOperationDefinition,
+  moving_average: ({
+    type: 'moving_average',
+    input: 'fullReference',
+    requiredReferences: [
+      {
+        input: ['field', 'managedReference'],
+        validateMetadata: (meta: OperationMetadata) =>
+          meta.dataType === 'number' && !meta.isBucketed,
+      },
+    ],
+    operationParams: [{ name: 'window', type: 'number', required: true }],
+    buildColumn: buildGenericColumn('moving_average'),
+    getPossibleOperation: numericOperation,
+  } as unknown) as GenericOperationDefinition,
+  cumulative_sum: ({
+    type: 'cumulative_sum',
+    input: 'fullReference',
+    buildColumn: buildGenericColumn('cumulative_sum'),
+    getPossibleOperation: numericOperation,
+  } as unknown) as GenericOperationDefinition,
+  terms: ({
+    type: 'terms',
+    input: 'field',
+    getPossibleOperationForField: stringOperation,
+  } as unknown) as GenericOperationDefinition,
+};
+
+describe('math completion', () => {
+  describe('signature help', () => {
+    function unwrapSignatures(signatureResult: monaco.languages.SignatureHelpResult) {
+      return signatureResult.value.signatures[0];
+    }
+
+    it('should silently handle parse errors', () => {
+      expect(unwrapSignatures(getSignatureHelp('sum(', 4, operationDefinitionMap))).toBeUndefined();
+    });
+
+    it('should return a signature for a field-based ES function', () => {
+      expect(unwrapSignatures(getSignatureHelp('sum()', 4, operationDefinitionMap))).toEqual({
+        label: 'sum(field: string)',
+        documentation: { value: 'description' },
+        parameters: [{ label: 'field' }],
+      });
+    });
+
+    it('should return a signature for count', () => {
+      expect(unwrapSignatures(getSignatureHelp('count()', 6, operationDefinitionMap))).toEqual({
+        label: 'count(undefined)',
+        documentation: { value: '' },
+        parameters: [],
+      });
+    });
+
+    it('should return a signature for a function with named parameters', () => {
+      expect(
+        unwrapSignatures(
+          getSignatureHelp('2 * moving_average(count(), window=)', 35, operationDefinitionMap)
+        )
+      ).toEqual({
+        label: expect.stringContaining('moving_average('),
+        documentation: { value: '' },
+        parameters: [
+          { label: 'function' },
+          {
+            label: 'window=number',
+            documentation: 'Required',
+          },
+        ],
+      });
+    });
+
+    it('should return a signature for an inner function', () => {
+      expect(
+        unwrapSignatures(
+          getSignatureHelp('2 * moving_average(count())', 25, operationDefinitionMap)
+        )
+      ).toEqual({
+        label: expect.stringContaining('count('),
+        parameters: [],
+        documentation: { value: '' },
+      });
+    });
+
+    it('should return a signature for a complex tinymath function', () => {
+      // 15 is the whitespace between the two arguments
+      expect(
+        unwrapSignatures(getSignatureHelp('clamp(count(), 5)', 15, operationDefinitionMap))
+      ).toEqual({
+        label: expect.stringContaining('clamp('),
+        documentation: { value: '' },
+        parameters: [
+          { label: 'value', documentation: '' },
+          { label: 'min', documentation: '' },
+          { label: 'max', documentation: '' },
+        ],
+      });
+    });
+  });
+
+  describe('hover provider', () => {
+    it('should silently handle parse errors', () => {
+      expect(getHover('sum(', 2, operationDefinitionMap)).toEqual({ contents: [] });
+    });
+
+    it('should show signature for a field-based ES function', () => {
+      expect(getHover('sum()', 2, operationDefinitionMap)).toEqual({
+        contents: [{ value: 'sum(field: string)' }],
+      });
+    });
+
+    it('should show signature for count', () => {
+      expect(getHover('count()', 2, operationDefinitionMap)).toEqual({
+        contents: [{ value: expect.stringContaining('count(') }],
+      });
+    });
+
+    it('should show signature for a function with named parameters', () => {
+      expect(getHover('2 * moving_average(count())', 10, operationDefinitionMap)).toEqual({
+        contents: [{ value: expect.stringContaining('moving_average(') }],
+      });
+    });
+
+    it('should show signature for an inner function', () => {
+      expect(getHover('2 * moving_average(count())', 22, operationDefinitionMap)).toEqual({
+        contents: [{ value: expect.stringContaining('count(') }],
+      });
+    });
+
+    it('should show signature for a complex tinymath function', () => {
+      expect(getHover('clamp(count(), 5)', 2, operationDefinitionMap)).toEqual({
+        contents: [{ value: expect.stringContaining('clamp([value]: number') }],
+      });
+    });
+  });
+
+  describe('autocomplete', () => {
+    it('should list all valid functions at the top level (fake test)', async () => {
+      // This test forces an invalid scenario, since the autocomplete actually requires
+      // some typing
+      const results = await suggest({
+        expression: '',
+        zeroIndexedOffset: 1,
+        context: {
+          triggerKind: monaco.languages.CompletionTriggerKind.TriggerCharacter,
+          triggerCharacter: '',
+        },
+        indexPattern: createMockedIndexPattern(),
+        operationDefinitionMap,
+        data: dataPluginMock.createStartContract(),
+      });
+      expect(results.list).toHaveLength(4 + Object.keys(tinymathFunctions).length);
+      ['sum', 'moving_average', 'cumulative_sum', 'last_value'].forEach((key) => {
+        expect(results.list).toEqual(expect.arrayContaining([{ label: key, type: 'operation' }]));
+      });
+      Object.keys(tinymathFunctions).forEach((key) => {
+        expect(results.list).toEqual(expect.arrayContaining([{ label: key, type: 'operation' }]));
+      });
+    });
+
+    it('should list all valid sub-functions for a fullReference', async () => {
+      const results = await suggest({
+        expression: 'moving_average()',
+        zeroIndexedOffset: 15,
+        context: {
+          triggerKind: monaco.languages.CompletionTriggerKind.TriggerCharacter,
+          triggerCharacter: '(',
+        },
+        indexPattern: createMockedIndexPattern(),
+        operationDefinitionMap,
+        data: dataPluginMock.createStartContract(),
+      });
+      expect(results.list).toHaveLength(2);
+      ['sum', 'last_value'].forEach((key) => {
+        expect(results.list).toEqual(expect.arrayContaining([{ label: key, type: 'operation' }]));
+      });
+    });
+
+    it('should list all valid named arguments for a fullReference', async () => {
+      const results = await suggest({
+        expression: 'moving_average(count(),)',
+        zeroIndexedOffset: 23,
+        context: {
+          triggerKind: monaco.languages.CompletionTriggerKind.TriggerCharacter,
+          triggerCharacter: ',',
+        },
+        indexPattern: createMockedIndexPattern(),
+        operationDefinitionMap,
+        data: dataPluginMock.createStartContract(),
+      });
+      expect(results.list).toEqual(['window']);
+    });
+
+    it('should not list named arguments when they are already in use', async () => {
+      const results = await suggest({
+        expression: 'moving_average(count(), window=5, )',
+        zeroIndexedOffset: 34,
+        context: {
+          triggerKind: monaco.languages.CompletionTriggerKind.TriggerCharacter,
+          triggerCharacter: ',',
+        },
+        indexPattern: createMockedIndexPattern(),
+        operationDefinitionMap,
+        data: dataPluginMock.createStartContract(),
+      });
+      expect(results.list).toEqual([]);
+    });
+
+    it('should list all valid positional arguments for a tinymath function used by name', async () => {
+      const results = await suggest({
+        expression: 'divide(count(), )',
+        zeroIndexedOffset: 16,
+        context: {
+          triggerKind: monaco.languages.CompletionTriggerKind.TriggerCharacter,
+          triggerCharacter: ',',
+        },
+        indexPattern: createMockedIndexPattern(),
+        operationDefinitionMap,
+        data: dataPluginMock.createStartContract(),
+      });
+      expect(results.list).toHaveLength(4 + Object.keys(tinymathFunctions).length);
+      ['sum', 'moving_average', 'cumulative_sum', 'last_value'].forEach((key) => {
+        expect(results.list).toEqual(expect.arrayContaining([{ label: key, type: 'math' }]));
+      });
+      Object.keys(tinymathFunctions).forEach((key) => {
+        expect(results.list).toEqual(expect.arrayContaining([{ label: key, type: 'math' }]));
+      });
+    });
+
+    it('should list all valid positional arguments for a tinymath function used with alias', async () => {
+      const results = await suggest({
+        expression: 'count() / ',
+        zeroIndexedOffset: 10,
+        context: {
+          triggerKind: monaco.languages.CompletionTriggerKind.TriggerCharacter,
+          triggerCharacter: ',',
+        },
+        indexPattern: createMockedIndexPattern(),
+        operationDefinitionMap,
+        data: dataPluginMock.createStartContract(),
+      });
+      expect(results.list).toHaveLength(4 + Object.keys(tinymathFunctions).length);
+      ['sum', 'moving_average', 'cumulative_sum', 'last_value'].forEach((key) => {
+        expect(results.list).toEqual(expect.arrayContaining([{ label: key, type: 'math' }]));
+      });
+      Object.keys(tinymathFunctions).forEach((key) => {
+        expect(results.list).toEqual(expect.arrayContaining([{ label: key, type: 'math' }]));
+      });
+    });
+
+    it('should not autocomplete any fields for the count function', async () => {
+      const results = await suggest({
+        expression: 'count()',
+        zeroIndexedOffset: 6,
+        context: {
+          triggerKind: monaco.languages.CompletionTriggerKind.TriggerCharacter,
+          triggerCharacter: '(',
+        },
+        indexPattern: createMockedIndexPattern(),
+        operationDefinitionMap,
+        data: dataPluginMock.createStartContract(),
+      });
+      expect(results.list).toHaveLength(0);
+    });
+
+    it('should autocomplete and validate the right type of field', async () => {
+      const results = await suggest({
+        expression: 'sum()',
+        zeroIndexedOffset: 4,
+        context: {
+          triggerKind: monaco.languages.CompletionTriggerKind.TriggerCharacter,
+          triggerCharacter: '(',
+        },
+        indexPattern: createMockedIndexPattern(),
+        operationDefinitionMap,
+        data: dataPluginMock.createStartContract(),
+      });
+      expect(results.list).toEqual(['bytes', 'memory']);
+    });
+
+    it('should autocomplete only operations that provide numeric output', async () => {
+      const results = await suggest({
+        expression: 'last_value()',
+        zeroIndexedOffset: 11,
+        context: {
+          triggerKind: monaco.languages.CompletionTriggerKind.TriggerCharacter,
+          triggerCharacter: '(',
+        },
+        indexPattern: createMockedIndexPattern(),
+        operationDefinitionMap,
+        data: dataPluginMock.createStartContract(),
+      });
+      expect(results.list).toEqual(['bytes', 'memory']);
+    });
+  });
+
+  describe('monacoPositionToOffset', () => {
+    it('should work with multi-line strings accounting for newline characters', () => {
+      const input = `012
+456
+89')`;
+      expect(input[monacoPositionToOffset(input, new monaco.Position(1, 1))]).toEqual('0');
+      expect(input[monacoPositionToOffset(input, new monaco.Position(3, 2))]).toEqual('9');
+    });
+  });
+
+  describe('getInfoAtZeroIndexedPosition', () => {
+    it('should return the location for a function inside multiple levels of math', () => {
+      const expression = `count() + 5 + average(LENS_MATH_MARKER)`;
+      const ast = parse(expression);
+      expect(getInfoAtZeroIndexedPosition(ast, 22)).toEqual({
+        ast: expect.objectContaining({ value: 'LENS_MATH_MARKER' }),
+        parent: expect.objectContaining({ name: 'average' }),
+      });
+    });
+  });
+});

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/math_completion.ts

@@ -0,0 +1,594 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import { uniq, startsWith } from 'lodash';
+import { i18n } from '@kbn/i18n';
+import { monaco } from '@kbn/monaco';
+import {
+  parse,
+  TinymathLocation,
+  TinymathAST,
+  TinymathFunction,
+  TinymathNamedArgument,
+} from '@kbn/tinymath';
+import type {
+  DataPublicPluginStart,
+  QuerySuggestion,
+} from '../../../../../../../../../src/plugins/data/public';
+import { IndexPattern } from '../../../../types';
+import { memoizedGetAvailableOperationsByMetadata } from '../../../operations';
+import { tinymathFunctions, groupArgsByType } from '../util';
+import type { GenericOperationDefinition } from '../..';
+import { getFunctionSignatureLabel, getHelpTextContent } from './formula_help';
+import { hasFunctionFieldArgument } from '../validation';
+
+export enum SUGGESTION_TYPE {
+  FIELD = 'field',
+  NAMED_ARGUMENT = 'named_argument',
+  FUNCTIONS = 'functions',
+  KQL = 'kql',
+}
+
+export type LensMathSuggestion =
+  | string
+  | {
+      label: string;
+      type: 'operation' | 'math';
+    }
+  | QuerySuggestion;
+
+export interface LensMathSuggestions {
+  list: LensMathSuggestion[];
+  type: SUGGESTION_TYPE;
+}
+
+function inLocation(cursorPosition: number, location: TinymathLocation) {
+  return cursorPosition >= location.min && cursorPosition < location.max;
+}
+
+const MARKER = 'LENS_MATH_MARKER';
+
+export function getInfoAtZeroIndexedPosition(
+  ast: TinymathAST,
+  zeroIndexedPosition: number,
+  parent?: TinymathFunction
+): undefined | { ast: TinymathAST; parent?: TinymathFunction } {
+  if (typeof ast === 'number') {
+    return;
+  }
+  // +, -, *, and / do not have location any more
+  if (ast.location && !inLocation(zeroIndexedPosition, ast.location)) {
+    return;
+  }
+  if (ast.type === 'function') {
+    const [match] = ast.args
+      .map((arg) => getInfoAtZeroIndexedPosition(arg, zeroIndexedPosition, ast))
+      .filter((a) => a);
+    if (match) {
+      return match;
+    } else if (ast.location) {
+      return { ast };
+    } else {
+      // None of the arguments match, but we don't know the position so it's not a match
+      return;
+    }
+  }
+  return {
+    ast,
+    parent,
+  };
+}
+
+export function offsetToRowColumn(expression: string, offset: number): monaco.Position {
+  const lines = expression.split(/\n/);
+  let remainingChars = offset;
+  let lineNumber = 1;
+  for (const line of lines) {
+    if (line.length >= remainingChars) {
+      return new monaco.Position(lineNumber, remainingChars);
+    }
+    remainingChars -= line.length + 1;
+    lineNumber++;
+  }
+
+  throw new Error('Algorithm failure');
+}
+
+export function monacoPositionToOffset(expression: string, position: monaco.Position): number {
+  const lines = expression.split(/\n/);
+  return lines
+    .slice(0, position.lineNumber)
+    .reduce(
+      (prev, current, index) =>
+        prev + (index === position.lineNumber - 1 ? position.column - 1 : current.length + 1),
+      0
+    );
+}
+
+export async function suggest({
+  expression,
+  zeroIndexedOffset,
+  context,
+  indexPattern,
+  operationDefinitionMap,
+  data,
+}: {
+  expression: string;
+  zeroIndexedOffset: number;
+  context: monaco.languages.CompletionContext;
+  indexPattern: IndexPattern;
+  operationDefinitionMap: Record<string, GenericOperationDefinition>;
+  data: DataPublicPluginStart;
+}): Promise<{ list: LensMathSuggestion[]; type: SUGGESTION_TYPE }> {
+  const text =
+    expression.substr(0, zeroIndexedOffset) + MARKER + expression.substr(zeroIndexedOffset);
+  try {
+    const ast = parse(text);
+
+    const tokenInfo = getInfoAtZeroIndexedPosition(ast, zeroIndexedOffset);
+    const tokenAst = tokenInfo?.ast;
+
+    const isNamedArgument =
+      tokenInfo?.parent &&
+      typeof tokenAst !== 'number' &&
+      tokenAst &&
+      'type' in tokenAst &&
+      tokenAst.type === 'namedArgument';
+    if (tokenInfo?.parent && (context.triggerCharacter === '=' || isNamedArgument)) {
+      return await getNamedArgumentSuggestions({
+        ast: tokenAst as TinymathNamedArgument,
+        data,
+        indexPattern,
+      });
+    } else if (tokenInfo?.parent) {
+      return getArgumentSuggestions(
+        tokenInfo.parent,
+        tokenInfo.parent.args.findIndex((a) => a === tokenAst),
+        indexPattern,
+        operationDefinitionMap
+      );
+    }
+    if (
+      typeof tokenAst === 'object' &&
+      Boolean(tokenAst.type === 'variable' || tokenAst.type === 'function')
+    ) {
+      const nameWithMarker = tokenAst.type === 'function' ? tokenAst.name : tokenAst.value;
+      return getFunctionSuggestions(
+        nameWithMarker.split(MARKER)[0],
+        indexPattern,
+        operationDefinitionMap
+      );
+    }
+  } catch (e) {
+    // Fail silently
+  }
+  return { list: [], type: SUGGESTION_TYPE.FIELD };
+}
+
+export function getPossibleFunctions(
+  indexPattern: IndexPattern,
+  operationDefinitionMap?: Record<string, GenericOperationDefinition>
+) {
+  const available = memoizedGetAvailableOperationsByMetadata(indexPattern, operationDefinitionMap);
+  const possibleOperationNames: string[] = [];
+  available.forEach((a) => {
+    if (a.operationMetaData.dataType === 'number' && !a.operationMetaData.isBucketed) {
+      possibleOperationNames.push(
+        ...a.operations.filter((o) => o.type !== 'managedReference').map((o) => o.operationType)
+      );
+    }
+  });
+
+  return [...uniq(possibleOperationNames), ...Object.keys(tinymathFunctions)];
+}
+
+function getFunctionSuggestions(
+  prefix: string,
+  indexPattern: IndexPattern,
+  operationDefinitionMap: Record<string, GenericOperationDefinition>
+) {
+  return {
+    list: uniq(
+      getPossibleFunctions(indexPattern, operationDefinitionMap).filter((func) =>
+        startsWith(func, prefix)
+      )
+    ).map((func) => ({ label: func, type: 'operation' as const })),
+    type: SUGGESTION_TYPE.FUNCTIONS,
+  };
+}
+
+function getArgumentSuggestions(
+  ast: TinymathFunction,
+  position: number,
+  indexPattern: IndexPattern,
+  operationDefinitionMap: Record<string, GenericOperationDefinition>
+) {
+  const { name } = ast;
+  const operation = operationDefinitionMap[name];
+  if (!operation && !tinymathFunctions[name]) {
+    return { list: [], type: SUGGESTION_TYPE.FIELD };
+  }
+
+  const tinymathFunction = tinymathFunctions[name];
+  if (tinymathFunction) {
+    if (tinymathFunction.positionalArguments[position]) {
+      return {
+        list: uniq(getPossibleFunctions(indexPattern, operationDefinitionMap)).map((f) => ({
+          type: 'math' as const,
+          label: f,
+        })),
+        type: SUGGESTION_TYPE.FUNCTIONS,
+      };
+    }
+    return { list: [], type: SUGGESTION_TYPE.FIELD };
+  }
+
+  if (position > 0 || !hasFunctionFieldArgument(operation.type)) {
+    const { namedArguments } = groupArgsByType(ast.args);
+    const list = [];
+    if (operation.filterable) {
+      if (!namedArguments.find((arg) => arg.name === 'kql')) {
+        list.push('kql');
+      }
+      if (!namedArguments.find((arg) => arg.name === 'lucene')) {
+        list.push('lucene');
+      }
+    }
+    if ('operationParams' in operation) {
+      // Exclude any previously used named args
+      list.push(
+        ...operation
+          .operationParams!.filter(
+            (param) =>
+              // Keep the param if it's the first use
+              !namedArguments.find((arg) => arg.name === param.name)
+          )
+          .map((p) => p.name)
+      );
+    }
+    return { list, type: SUGGESTION_TYPE.NAMED_ARGUMENT };
+  }
+
+  if (operation.input === 'field' && position === 0) {
+    const available = memoizedGetAvailableOperationsByMetadata(
+      indexPattern,
+      operationDefinitionMap
+    );
+    // TODO: This only allow numeric functions, will reject last_value(string) for example.
+    const validOperation = available.find(
+      ({ operationMetaData }) =>
+        operationMetaData.dataType === 'number' && !operationMetaData.isBucketed
+    );
+    if (validOperation) {
+      const fields = validOperation.operations
+        .filter((op) => op.operationType === operation.type)
+        .map((op) => ('field' in op ? op.field : undefined))
+        .filter((field) => field);
+      return { list: fields as string[], type: SUGGESTION_TYPE.FIELD };
+    } else {
+      return { list: [], type: SUGGESTION_TYPE.FIELD };
+    }
+  }
+
+  if (operation.input === 'fullReference') {
+    const available = memoizedGetAvailableOperationsByMetadata(
+      indexPattern,
+      operationDefinitionMap
+    );
+    const possibleOperationNames: string[] = [];
+    available.forEach((a) => {
+      if (
+        operation.requiredReferences.some((requirement) =>
+          requirement.validateMetadata(a.operationMetaData)
+        )
+      ) {
+        possibleOperationNames.push(
+          ...a.operations
+            .filter((o) =>
+              operation.requiredReferences.some((requirement) => requirement.input.includes(o.type))
+            )
+            .map((o) => o.operationType)
+        );
+      }
+    });
+    return {
+      list: uniq(possibleOperationNames).map((n) => ({ label: n, type: 'operation' as const })),
+      type: SUGGESTION_TYPE.FUNCTIONS,
+    };
+  }
+
+  return { list: [], type: SUGGESTION_TYPE.FIELD };
+}
+
+export async function getNamedArgumentSuggestions({
+  ast,
+  data,
+  indexPattern,
+}: {
+  ast: TinymathNamedArgument;
+  indexPattern: IndexPattern;
+  data: DataPublicPluginStart;
+}) {
+  if (ast.name !== 'kql' && ast.name !== 'lucene') {
+    return { list: [], type: SUGGESTION_TYPE.KQL };
+  }
+  if (!data.autocomplete.hasQuerySuggestions(ast.name === 'kql' ? 'kuery' : 'lucene')) {
+    return { list: [], type: SUGGESTION_TYPE.KQL };
+  }
+
+  const query = ast.value.split(MARKER)[0];
+  const position = ast.value.indexOf(MARKER) + 1;
+
+  const suggestions = await data.autocomplete.getQuerySuggestions({
+    language: ast.name === 'kql' ? 'kuery' : 'lucene',
+    query,
+    selectionStart: position,
+    selectionEnd: position,
+    indexPatterns: [indexPattern],
+    boolFilter: [],
+  });
+  return {
+    list: suggestions ?? [],
+    type: SUGGESTION_TYPE.KQL,
+  };
+}
+
+const TRIGGER_SUGGESTION_COMMAND = {
+  title: 'Trigger Suggestion Dialog',
+  id: 'editor.action.triggerSuggest',
+};
+
+export function getSuggestion(
+  suggestion: LensMathSuggestion,
+  type: SUGGESTION_TYPE,
+  operationDefinitionMap: Record<string, GenericOperationDefinition>,
+  triggerChar: string | undefined
+): monaco.languages.CompletionItem {
+  let kind: monaco.languages.CompletionItemKind = monaco.languages.CompletionItemKind.Method;
+  let label: string =
+    typeof suggestion === 'string'
+      ? suggestion
+      : 'label' in suggestion
+      ? suggestion.label
+      : suggestion.text;
+  let insertText: string | undefined;
+  let insertTextRules: monaco.languages.CompletionItem['insertTextRules'];
+  let detail: string = '';
+  let command: monaco.languages.CompletionItem['command'];
+  let sortText: string = '';
+  const filterText: string = label;
+
+  switch (type) {
+    case SUGGESTION_TYPE.FIELD:
+      kind = monaco.languages.CompletionItemKind.Value;
+      break;
+    case SUGGESTION_TYPE.FUNCTIONS:
+      insertText = `${label}($0)`;
+      insertTextRules = monaco.languages.CompletionItemInsertTextRule.InsertAsSnippet;
+      if (typeof suggestion !== 'string') {
+        if ('text' in suggestion) break;
+        label = getFunctionSignatureLabel(suggestion.label, operationDefinitionMap);
+        const tinymathFunction = tinymathFunctions[suggestion.label];
+        if (tinymathFunction) {
+          detail = 'TinyMath';
+          kind = monaco.languages.CompletionItemKind.Method;
+        } else {
+          kind = monaco.languages.CompletionItemKind.Constant;
+          detail = 'Elasticsearch';
+          // Always put ES functions first
+          sortText = `0${label}`;
+          command = TRIGGER_SUGGESTION_COMMAND;
+        }
+      }
+      break;
+    case SUGGESTION_TYPE.NAMED_ARGUMENT:
+      kind = monaco.languages.CompletionItemKind.Keyword;
+      if (label === 'kql' || label === 'lucene') {
+        command = TRIGGER_SUGGESTION_COMMAND;
+        insertText = `${label}='$0'`;
+        insertTextRules = monaco.languages.CompletionItemInsertTextRule.InsertAsSnippet;
+        sortText = `zzz${label}`;
+      }
+      label = `${label}=`;
+      detail = '';
+      break;
+    case SUGGESTION_TYPE.KQL:
+      if (triggerChar === ':') {
+        insertText = `${triggerChar} ${label}`;
+      } else {
+        // concatenate KQL suggestion for faster query composition
+        command = TRIGGER_SUGGESTION_COMMAND;
+      }
+      if (label.includes(`'`)) {
+        insertText = (insertText || label).replaceAll(`'`, "\\'");
+      }
+      break;
+  }
+
+  return {
+    detail,
+    kind,
+    label,
+    insertText: insertText ?? label,
+    insertTextRules,
+    command,
+    additionalTextEdits: [],
+    // @ts-expect-error Monaco says this type is required, but provides a default value
+    range: undefined,
+    sortText,
+    filterText,
+  };
+}
+
+function getOperationTypeHelp(
+  name: string,
+  operationDefinitionMap: Record<string, GenericOperationDefinition>
+) {
+  const { description: descriptionInMarkdown, examples } = getHelpTextContent(
+    name,
+    operationDefinitionMap
+  );
+  const examplesInMarkdown = examples.length
+    ? `\n\n**${i18n.translate('xpack.lens.formulaExampleMarkdown', {
+        defaultMessage: 'Examples',
+      })}**
+
+  ${examples.map((example) => `\`${example}\``).join('\n\n')}`
+    : '';
+  return {
+    value: `${descriptionInMarkdown}${examplesInMarkdown}`,
+  };
+}
+
+function getSignaturesForFunction(
+  name: string,
+  operationDefinitionMap: Record<string, GenericOperationDefinition>
+) {
+  if (tinymathFunctions[name]) {
+    const stringify = getFunctionSignatureLabel(name, operationDefinitionMap);
+    const documentation = tinymathFunctions[name].help.replace(/\n/g, '\n\n');
+    return [
+      {
+        label: stringify,
+        documentation: { value: documentation },
+        parameters: tinymathFunctions[name].positionalArguments.map((arg) => ({
+          label: arg.name,
+          documentation: arg.optional
+            ? i18n.translate('xpack.lens.formula.optionalArgument', {
+                defaultMessage: 'Optional. Default value is {defaultValue}',
+                values: {
+                  defaultValue: arg.defaultValue,
+                },
+              })
+            : '',
+        })),
+      },
+    ];
+  }
+  if (operationDefinitionMap[name]) {
+    const def = operationDefinitionMap[name];
+
+    const firstParam: monaco.languages.ParameterInformation | null = hasFunctionFieldArgument(name)
+      ? {
+          label: def.input === 'field' ? 'field' : def.input === 'fullReference' ? 'function' : '',
+        }
+      : null;
+
+    const functionLabel = getFunctionSignatureLabel(name, operationDefinitionMap, firstParam);
+    const documentation = getOperationTypeHelp(name, operationDefinitionMap);
+    if ('operationParams' in def && def.operationParams) {
+      return [
+        {
+          label: functionLabel,
+          parameters: [
+            ...(firstParam ? [firstParam] : []),
+            ...def.operationParams.map((arg) => ({
+              label: `${arg.name}=${arg.type}`,
+              documentation: arg.required
+                ? i18n.translate('xpack.lens.formula.requiredArgument', {
+                    defaultMessage: 'Required',
+                  })
+                : '',
+            })),
+          ],
+          documentation,
+        },
+      ];
+    }
+    return [
+      {
+        label: functionLabel,
+        parameters: firstParam ? [firstParam] : [],
+        documentation,
+      },
+    ];
+  }
+  return [];
+}
+
+export function getSignatureHelp(
+  expression: string,
+  position: number,
+  operationDefinitionMap: Record<string, GenericOperationDefinition>
+): monaco.languages.SignatureHelpResult {
+  const text = expression.substr(0, position) + MARKER + expression.substr(position);
+  try {
+    const ast = parse(text);
+
+    const tokenInfo = getInfoAtZeroIndexedPosition(ast, position);
+
+    let signatures: ReturnType<typeof getSignaturesForFunction> = [];
+    let index = 0;
+    if (tokenInfo?.parent) {
+      const name = tokenInfo.parent.name;
+      // reference equality is fine here because of the way the getInfo function works
+      index = tokenInfo.parent.args.findIndex((arg) => arg === tokenInfo.ast);
+      signatures = getSignaturesForFunction(name, operationDefinitionMap);
+    } else if (typeof tokenInfo?.ast === 'object' && tokenInfo.ast.type === 'function') {
+      const name = tokenInfo.ast.name;
+      signatures = getSignaturesForFunction(name, operationDefinitionMap);
+    }
+    if (signatures.length) {
+      return {
+        value: {
+          // remove the documentation
+          signatures: signatures.map(({ documentation, ...signature }) => ({
+            ...signature,
+            // extract only the first section (usually few lines)
+            documentation: { value: documentation.value.split('\n\n')[0] },
+          })),
+          activeParameter: index,
+          activeSignature: 0,
+        },
+        dispose: () => {},
+      };
+    }
+  } catch (e) {
+    // do nothing
+  }
+  return { value: { signatures: [], activeParameter: 0, activeSignature: 0 }, dispose: () => {} };
+}
+
+export function getHover(
+  expression: string,
+  position: number,
+  operationDefinitionMap: Record<string, GenericOperationDefinition>
+): monaco.languages.Hover {
+  try {
+    const ast = parse(expression);
+
+    const tokenInfo = getInfoAtZeroIndexedPosition(ast, position);
+
+    if (!tokenInfo || typeof tokenInfo.ast === 'number' || !('name' in tokenInfo.ast)) {
+      return { contents: [] };
+    }
+
+    const name = tokenInfo.ast.name;
+    const signatures = getSignaturesForFunction(name, operationDefinitionMap);
+    if (signatures.length) {
+      const { label } = signatures[0];
+
+      return {
+        contents: [{ value: label }],
+      };
+    }
+  } catch (e) {
+    // do nothing
+  }
+  return { contents: [] };
+}
+
+export function getTokenInfo(expression: string, position: number) {
+  const text = expression.substr(0, position) + MARKER + expression.substr(position);
+  try {
+    const ast = parse(text);
+
+    return getInfoAtZeroIndexedPosition(ast, position);
+  } catch (e) {
+    return;
+  }
+}

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/math_tokenization.tsx

@@ -0,0 +1,66 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import { monaco } from '@kbn/monaco';
+
+export const LANGUAGE_ID = 'lens_math';
+monaco.languages.register({ id: LANGUAGE_ID });
+
+export const languageConfiguration: monaco.languages.LanguageConfiguration = {
+  wordPattern: /[^()'"\s]+/g,
+  brackets: [['(', ')']],
+  autoClosingPairs: [
+    { open: '(', close: ')' },
+    { open: `'`, close: `'` },
+    { open: '"', close: '"' },
+  ],
+  surroundingPairs: [
+    { open: '(', close: ')' },
+    { open: `'`, close: `'` },
+    { open: '"', close: '"' },
+  ],
+};
+
+export const lexerRules = {
+  defaultToken: 'invalid',
+  tokenPostfix: '',
+  ignoreCase: true,
+  brackets: [{ open: '(', close: ')', token: 'delimiter.parenthesis' }],
+  escapes: /\\(?:[\\"'])/,
+  tokenizer: {
+    root: [
+      [/\s+/, 'whitespace'],
+      [/-?(\d*\.)?\d+([eE][+\-]?\d+)?/, 'number'],
+      [/[a-zA-Z0-9][a-zA-Z0-9_\-\.]*/, 'keyword'],
+      [/[,=:]/, 'delimiter'],
+      // strings double quoted
+      [/"([^"\\]|\\.)*$/, 'string.invalid'], // string without termination
+      [/"/, 'string', '@string_dq'],
+      // strings single quoted
+      [/'([^'\\]|\\.)*$/, 'string.invalid'], // string without termination
+      [/'/, 'string', '@string_sq'],
+      [/\+|\-|\*|\//, 'keyword.operator'],
+      [/[\(]/, 'delimiter'],
+      [/[\)]/, 'delimiter'],
+    ],
+    string_dq: [
+      [/[^\\"]+/, 'string'],
+      [/@escapes/, 'string.escape'],
+      [/\\./, 'string.escape.invalid'],
+      [/"/, 'string', '@pop'],
+    ],
+    string_sq: [
+      [/[^\\']+/, 'string'],
+      [/@escapes/, 'string.escape'],
+      [/\\./, 'string.escape.invalid'],
+      [/'/, 'string', '@pop'],
+    ],
+  },
+} as monaco.languages.IMonarchLanguage;
+
+monaco.languages.setMonarchTokensProvider(LANGUAGE_ID, lexerRules);
+monaco.languages.setLanguageConfiguration(LANGUAGE_ID, languageConfiguration);

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/formula.test.tsx

@@ -14,8 +14,10 @@ import { tinymathFunctions } from './util';
 
 jest.mock('../../layer_helpers', () => {
   return {
-    getColumnOrder: ({ columns }: { columns: Record<string, IndexPatternColumn> }) =>
-      Object.keys(columns),
+    getColumnOrder: jest.fn(({ columns }: { columns: Record<string, IndexPatternColumn> }) =>
+      Object.keys(columns)
+    ),
+    getManagedColumnsFrom: jest.fn().mockReturnValue([]),
   };
 });
 
@@ -142,7 +144,7 @@ describe('formula', () => {
           indexPattern,
         })
       ).toEqual({
-        label: 'Formula',
+        label: 'average(bytes)',
         dataType: 'number',
         operationType: 'formula',
         isBucketed: false,
@@ -170,7 +172,7 @@ describe('formula', () => {
           indexPattern,
         })
       ).toEqual({
-        label: 'Formula',
+        label: 'average(bytes)',
         dataType: 'number',
         operationType: 'formula',
         isBucketed: false,
@@ -204,7 +206,7 @@ describe('formula', () => {
           indexPattern,
         })
       ).toEqual({
-        label: 'Formula',
+        label: `average(bytes, kql='category.keyword: "Men\\'s Clothing" or category.keyword: "Men\\'s Shoes"')`,
         dataType: 'number',
         operationType: 'formula',
         isBucketed: false,
@@ -233,7 +235,7 @@ describe('formula', () => {
           indexPattern,
         })
       ).toEqual({
-        label: 'Formula',
+        label: `count(lucene='*')`,
         dataType: 'number',
         operationType: 'formula',
         isBucketed: false,
@@ -291,7 +293,7 @@ describe('formula', () => {
           operationDefinitionMap
         )
       ).toEqual({
-        label: 'Formula',
+        label: 'moving_average(average(bytes), window=3)',
         dataType: 'number',
         operationType: 'formula',
         isBucketed: false,
@@ -375,6 +377,7 @@ describe('formula', () => {
           ...layer.columns,
           col1: {
             ...currentColumn,
+            label: formula,
             params: {
               ...currentColumn.params,
               formula,
@@ -415,6 +418,7 @@ describe('formula', () => {
           ...layer.columns,
           col1: {
             ...currentColumn,
+            label: 'average(bytes)',
             references: ['col1X1'],
             params: {
               ...currentColumn.params,
@@ -565,7 +569,7 @@ describe('formula', () => {
       ).toEqual({
         col1X0: { min: 15, max: 29 },
         col1X2: { min: 0, max: 41 },
-        col1X3: { min: 43, max: 50 },
+        col1X3: { min: 42, max: 50 },
       });
     });
   });
@@ -787,6 +791,34 @@ invalid: "
       }
     });
 
+    it('returns an error if formula or math operations are used', () => {
+      const formulaFormulas = ['formula()', 'formula(bytes)', 'formula(formula())'];
+
+      for (const formula of formulaFormulas) {
+        expect(
+          formulaOperation.getErrorMessage!(
+            getNewLayerWithFormula(formula),
+            'col1',
+            indexPattern,
+            operationDefinitionMap
+          )
+        ).toEqual(['Operation formula not found']);
+      }
+
+      const mathFormulas = ['math()', 'math(bytes)', 'math(math())'];
+
+      for (const formula of mathFormulas) {
+        expect(
+          formulaOperation.getErrorMessage!(
+            getNewLayerWithFormula(formula),
+            'col1',
+            indexPattern,
+            operationDefinitionMap
+          )
+        ).toEqual(['Operation math not found']);
+      }
+    });
+
     it('returns an error if field operation in formula have the wrong first argument', () => {
       const formulas = [
         'average(7)',
@@ -897,6 +929,150 @@ invalid: "
       ).toEqual(undefined);
     });
 
+    it('returns no error for a query edge case', () => {
+      const formulas = [
+        `count(kql='')`,
+        `count(lucene='')`,
+        `moving_average(count(kql=''), window=7)`,
+        `count(kql='bytes >= 4000')`,
+        `count(kql='bytes <= 4000')`,
+        `count(kql='bytes = 4000')`,
+      ];
+      for (const formula of formulas) {
+        expect(
+          formulaOperation.getErrorMessage!(
+            getNewLayerWithFormula(formula),
+            'col1',
+            indexPattern,
+            operationDefinitionMap
+          )
+        ).toEqual(undefined);
+      }
+    });
+
+    it('returns an error for a query not wrapped in single quotes', () => {
+      const formulas = [
+        `count(kql="")`,
+        `count(kql='")`,
+        `count(kql="')`,
+        `count(kql="category.keyword: *")`,
+        `count(kql='category.keyword: *")`,
+        `count(kql="category.keyword: *')`,
+        `count(kql='category.keyword: *)`,
+        `count(kql=category.keyword: *')`,
+        `count(kql=category.keyword: *)`,
+        `count(kql="category.keyword: "Men's Clothing" or category.keyword: "Men's Shoes"")`,
+        `count(lucene="category.keyword: *")`,
+        `count(lucene=category.keyword: *)`,
+        `count(lucene=category.keyword: *) + average(bytes)`,
+        `count(lucene='category.keyword: *') + count(kql=category.keyword: *)`,
+        `count(lucene='category.keyword: *") + count(kql='category.keyword: *")`,
+        `count(lucene='category.keyword: *') + count(kql=category.keyword: *, kql='category.keyword: *')`,
+        `count(lucene='category.keyword: *') + count(kql="category.keyword: *")`,
+        `moving_average(count(kql=category.keyword: *), window=7, kql=category.keywork: *)`,
+        `moving_average(
+          cumulative_sum(
+             7 * clamp(sum(bytes), 0, last_value(memory) + max(memory))
+          ), window=10, kql=category.keywork: *
+        )`,
+      ];
+      for (const formula of formulas) {
+        expect(
+          formulaOperation.getErrorMessage!(
+            getNewLayerWithFormula(formula),
+            'col1',
+            indexPattern,
+            operationDefinitionMap
+          )
+        ).toEqual(expect.arrayContaining([expect.stringMatching(`Single quotes are required`)]));
+      }
+    });
+
+    it('it returns parse fail error rather than query message if the formula is only a query condition (false positive cases for query checks)', () => {
+      const formulas = [
+        `kql="category.keyword: *"`,
+        `kql=category.keyword: *`,
+        `kql='category.keyword: *'`,
+        `(kql="category.keyword: *")`,
+        `(kql=category.keyword: *)`,
+        `(lucene="category.keyword: *")`,
+        `(lucene=category.keyword: *)`,
+        `(lucene='category.keyword: *') + (kql=category.keyword: *)`,
+        `(lucene='category.keyword: *') + (kql=category.keyword: *, kql='category.keyword: *')`,
+        `(lucene='category.keyword: *') + (kql="category.keyword: *")`,
+        `((kql=category.keyword: *), window=7, kql=category.keywork: *)`,
+        `(, window=10, kql=category.keywork: *)`,
+        `(
+          , window=10, kql=category.keywork: *
+        )`,
+      ];
+      for (const formula of formulas) {
+        expect(
+          formulaOperation.getErrorMessage!(
+            getNewLayerWithFormula(formula),
+            'col1',
+            indexPattern,
+            operationDefinitionMap
+          )
+        ).toEqual([`The Formula ${formula} cannot be parsed`]);
+      }
+    });
+
+    it('returns no error for a query wrapped in single quotes but with some whitespaces', () => {
+      const formulas = [
+        `count(kql ='category.keyword: *')`,
+        `count(kql = 'category.keyword: *')`,
+        `count(kql =    'category.keyword: *')`,
+      ];
+      for (const formula of formulas) {
+        expect(
+          formulaOperation.getErrorMessage!(
+            getNewLayerWithFormula(formula),
+            'col1',
+            indexPattern,
+            operationDefinitionMap
+          )
+        ).toEqual(undefined);
+      }
+    });
+
+    it('returns an error for multiple queries submitted for the same function', () => {
+      expect(
+        formulaOperation.getErrorMessage!(
+          getNewLayerWithFormula(`count(kql='category.keyword: *', lucene='category.keyword: *')`),
+          'col1',
+          indexPattern,
+          operationDefinitionMap
+        )
+      ).toEqual(['Use only one of kql= or lucene=, not both']);
+    });
+
+    it("returns a clear error when there's a missing field for a function", () => {
+      for (const fn of ['average', 'terms', 'max', 'sum']) {
+        expect(
+          formulaOperation.getErrorMessage!(
+            getNewLayerWithFormula(`${fn}()`),
+            'col1',
+            indexPattern,
+            operationDefinitionMap
+          )
+        ).toEqual([`The first argument for ${fn} should be a field name. Found no field`]);
+      }
+    });
+
+    it("returns a clear error when there's a missing function for a fullReference operation", () => {
+      for (const fn of ['cumulative_sum', 'derivative']) {
+        expect(
+          formulaOperation.getErrorMessage!(
+            getNewLayerWithFormula(`${fn}()`),
+            'col1',
+            indexPattern,
+            operationDefinitionMap
+          )
+        ).toEqual([`The first argument for ${fn} should be a operation name. Found no operation`]);
+      }
+    });
+
     it('returns no error if a math operation is passed to fullReference operations', () => {
       const formulas = [
         'derivative(7+1)',

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/formula.tsx

@@ -10,8 +10,11 @@ import { OperationDefinition } from '../index';
 import { ReferenceBasedIndexPatternColumn } from '../column_types';
 import { IndexPattern } from '../../../types';
 import { runASTValidation, tryToParse } from './validation';
+import { MemoizedFormulaEditor } from './editor';
 import { regenerateLayerFromAst } from './parse';
 import { generateFormula } from './generate';
+import { filterByVisibleOperation } from './util';
+import { getManagedColumnsFrom } from '../../layer_helpers';
 
 const defaultLabel = i18n.translate('xpack.lens.indexPattern.formulaLabel', {
   defaultMessage: 'Formula',
@@ -38,7 +41,7 @@ export const formulaOperation: OperationDefinition<
 > = {
   type: 'formula',
   displayName: defaultLabel,
-  getDefaultLabel: (column, indexPattern) => defaultLabel,
+  getDefaultLabel: (column, indexPattern) => column.params.formula ?? defaultLabel,
   input: 'managedReference',
   hidden: true,
   getDisabledStatus(indexPattern: IndexPattern) {
@@ -49,13 +52,32 @@ export const formulaOperation: OperationDefinition<
     if (!column.params.formula || !operationDefinitionMap) {
       return;
     }
-    const { root, error } = tryToParse(column.params.formula);
+
+    const visibleOperationsMap = filterByVisibleOperation(operationDefinitionMap);
+    const { root, error } = tryToParse(column.params.formula, visibleOperationsMap);
     if (error || !root) {
       return [error!.message];
     }
 
-    const errors = runASTValidation(root, layer, indexPattern, operationDefinitionMap);
-    return errors.length ? errors.map(({ message }) => message) : undefined;
+    const errors = runASTValidation(root, layer, indexPattern, visibleOperationsMap);
+
+    if (errors.length) {
+      return errors.map(({ message }) => message);
+    }
+
+    const managedColumns = getManagedColumnsFrom(columnId, layer.columns);
+    const innerErrors = managedColumns
+      .flatMap(([id, col]) => {
+        const def = visibleOperationsMap[col.operationType];
+        if (def?.getErrorMessage) {
+          const messages = def.getErrorMessage(layer, id, indexPattern, visibleOperationsMap);
+          return messages ? { message: messages.join(', ') } : [];
+        }
+        return [];
+      })
+      .filter((marker) => marker);
+
+    return innerErrors.length ? innerErrors.map(({ message }) => message) : undefined;
   },
   getPossibleOperation() {
     return {
@@ -72,8 +94,8 @@ export const formulaOperation: OperationDefinition<
     const label = !params?.isFormulaBroken
       ? useDisplayLabel
         ? currentColumn.label
-        : params?.formula
-      : '';
+        : params?.formula ?? defaultLabel
+      : defaultLabel;
 
     return [
       {
@@ -81,21 +103,23 @@ export const formulaOperation: OperationDefinition<
         function: 'mapColumn',
         arguments: {
           id: [columnId],
-          name: [label || ''],
+          name: [label || defaultLabel],
           exp: [
             {
               type: 'expression',
-              chain: [
-                {
-                  type: 'function',
-                  function: 'math',
-                  arguments: {
-                    expression: [
-                      currentColumn.references.length ? `"${currentColumn.references[0]}"` : ``,
-                    ],
-                  },
-                },
-              ],
+              chain: currentColumn.references.length
+                ? [
+                    {
+                      type: 'function',
+                      function: 'math',
+                      arguments: {
+                        expression: [
+                          currentColumn.references.length ? `"${currentColumn.references[0]}"` : ``,
+                        ],
+                      },
+                    },
+                  ]
+                : [],
             },
           ],
         },
@@ -119,7 +143,7 @@ export const formulaOperation: OperationDefinition<
       prevFormat = { format: previousColumn.params.format };
     }
     return {
-      label: 'Formula',
+      label: previousFormula || defaultLabel,
       dataType: 'number',
       operationType: 'formula',
       isBucketed: false,
@@ -152,4 +176,6 @@ export const formulaOperation: OperationDefinition<
     );
     return newLayer;
   },
+
+  paramEditor: MemoizedFormulaEditor,
 };

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/math_examples.md

@@ -0,0 +1,28 @@
+Basic numeric functions that we already support in Lens:
+
+count()
+count(normalize_unit='1s')
+sum(field name)
+avg(field name)
+moving_average(sum(field name), window=5)
+moving_average(sum(field name), window=5, normalize_unit='1s')
+counter_rate(field name, normalize_unit='1s')
+differences(count())
+differences(sum(bytes), normalize_unit='1s')
+last_value(bytes, sort=timestamp)
+percentile(bytes, percent=95)
+
+Adding features beyond what we already support. New features are:
+
+* Filtering
+* Math across series
+* Time offset
+
+count() * 100
+(count() / count(offset=-7d)) + min(field name)
+sum(field name, filter='field.keyword: "KQL autocomplete inside math" AND field.value > 100')
+
+What about custom formatting using string manipulation? Probably not...
+
+(avg(bytes) / 1000) + 'kb'
+  

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/parse.ts

@@ -5,6 +5,7 @@
  * 2.0.
  */
 
+import { i18n } from '@kbn/i18n';
 import { isObject } from 'lodash';
 import type { TinymathAST, TinymathVariable, TinymathLocation } from '@kbn/tinymath';
 import { OperationDefinition, GenericOperationDefinition, IndexPatternColumn } from '../index';
@@ -12,7 +13,12 @@ import { IndexPattern, IndexPatternLayer } from '../../../types';
 import { mathOperation } from './math';
 import { documentField } from '../../../document_field';
 import { runASTValidation, shouldHaveFieldArgument, tryToParse } from './validation';
-import { findVariables, getOperationParams, groupArgsByType } from './util';
+import {
+  filterByVisibleOperation,
+  findVariables,
+  getOperationParams,
+  groupArgsByType,
+} from './util';
 import { FormulaIndexPatternColumn } from './formula';
 import { getColumnOrder } from '../../layer_helpers';
 
@@ -27,7 +33,7 @@ function parseAndExtract(
   indexPattern: IndexPattern,
   operationDefinitionMap: Record<string, GenericOperationDefinition>
 ) {
-  const { root, error } = tryToParse(text);
+  const { root, error } = tryToParse(text, operationDefinitionMap);
   if (error || !root) {
     return { extracted: [], isValid: false };
   }
@@ -61,9 +67,9 @@ function extractColumns(
     const nodeOperation = operations[node.name];
     if (!nodeOperation) {
       // it's a regular math node
-      const consumedArgs = node.args.map(parseNode).filter(Boolean) as Array<
-        number | TinymathVariable
-      >;
+      const consumedArgs = node.args
+        .map(parseNode)
+        .filter((n) => typeof n !== 'undefined' && n !== null) as Array<number | TinymathVariable>;
       return {
         ...node,
         args: consumedArgs,
@@ -168,7 +174,7 @@ export function regenerateLayerFromAst(
     layer,
     columnId,
     indexPattern,
-    operationDefinitionMap
+    filterByVisibleOperation(operationDefinitionMap)
   );
 
   const columns = { ...layer.columns };
@@ -188,6 +194,12 @@ export function regenerateLayerFromAst(
 
   columns[columnId] = {
     ...currentColumn,
+    label: !currentColumn.customLabel
+      ? text ??
+        i18n.translate('xpack.lens.indexPattern.formulaLabel', {
+          defaultMessage: 'Formula',
+        })
+      : currentColumn.label,
     params: {
       ...currentColumn.params,
       formula: text,

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/util.ts

@@ -13,7 +13,7 @@ import type {
   TinymathNamedArgument,
   TinymathVariable,
 } from 'packages/kbn-tinymath';
-import type { OperationDefinition, IndexPatternColumn } from '../index';
+import type { OperationDefinition, IndexPatternColumn, GenericOperationDefinition } from '../index';
 import type { GroupedNodes } from './types';
 
 export function groupArgsByType(args: TinymathAST[]) {
@@ -66,6 +66,16 @@ export function getOperationParams(
   }, {});
 }
 
+function getTypeI18n(type: string) {
+  if (type === 'number') {
+    return i18n.translate('xpack.lens.formula.number', { defaultMessage: 'number' });
+  }
+  if (type === 'string') {
+    return i18n.translate('xpack.lens.formula.string', { defaultMessage: 'string' });
+  }
+  return '';
+}
+
 // Todo: i18n everything here
 export const tinymathFunctions: Record<
   string,
@@ -73,145 +83,254 @@ export const tinymathFunctions: Record<
     positionalArguments: Array<{
       name: string;
       optional?: boolean;
+      defaultValue?: string | number;
+      type?: string;
     }>;
-    // help: React.ReactElement;
     // Help is in Markdown format
     help: string;
   }
 > = {
   add: {
     positionalArguments: [
-      { name: i18n.translate('xpack.lens.formula.left', { defaultMessage: 'left' }) },
-      { name: i18n.translate('xpack.lens.formula.right', { defaultMessage: 'right' }) },
+      {
+        name: i18n.translate('xpack.lens.formula.left', { defaultMessage: 'left' }),
+        type: getTypeI18n('number'),
+      },
+      {
+        name: i18n.translate('xpack.lens.formula.right', { defaultMessage: 'right' }),
+        type: getTypeI18n('number'),
+      },
     ],
     help: `
+Adds up two numbers.
 Also works with + symbol
-Example: ${'`count() + sum(bytes)`'}
-Example: ${'`add(count(), 5)`'}
+
+Example: Calculate the sum of two fields
+
+${'`sum(price) + sum(tax)`'}
+
+Example: Offset count by a static value
+
+${'`add(count(), 5)`'}
     `,
   },
   subtract: {
     positionalArguments: [
-      { name: i18n.translate('xpack.lens.formula.left', { defaultMessage: 'left' }) },
-      { name: i18n.translate('xpack.lens.formula.right', { defaultMessage: 'right' }) },
+      {
+        name: i18n.translate('xpack.lens.formula.left', { defaultMessage: 'left' }),
+        type: getTypeI18n('number'),
+      },
+      {
+        name: i18n.translate('xpack.lens.formula.right', { defaultMessage: 'right' }),
+        type: getTypeI18n('number'),
+      },
     ],
     help: `
+Subtracts the first number from the second number.
 Also works with ${'`-`'} symbol
-Example: ${'`subtract(sum(bytes), avg(bytes))`'}
+
+Example: Calculate the range of a field
+${'`subtract(max(bytes), min(bytes))`'}
     `,
   },
   multiply: {
     positionalArguments: [
-      { name: i18n.translate('xpack.lens.formula.left', { defaultMessage: 'left' }) },
-      { name: i18n.translate('xpack.lens.formula.right', { defaultMessage: 'right' }) },
+      {
+        name: i18n.translate('xpack.lens.formula.left', { defaultMessage: 'left' }),
+        type: getTypeI18n('number'),
+      },
+      {
+        name: i18n.translate('xpack.lens.formula.right', { defaultMessage: 'right' }),
+        type: getTypeI18n('number'),
+      },
     ],
     help: `
-Also works with ${'`*`'} symbol
-Example: ${'`multiply(sum(bytes), 2)`'}
+Multiplies two numbers.
+Also works with ${'`*`'} symbol.
+
+Example: Calculate price after current tax rate
+${'`sum(bytes) * last_value(tax_rate)`'}
+
+Example: Calculate price after constant tax rate
+${'`multiply(sum(price), 1.2)`'}
     `,
   },
   divide: {
     positionalArguments: [
-      { name: i18n.translate('xpack.lens.formula.left', { defaultMessage: 'left' }) },
-      { name: i18n.translate('xpack.lens.formula.right', { defaultMessage: 'right' }) },
+      {
+        name: i18n.translate('xpack.lens.formula.left', { defaultMessage: 'left' }),
+        type: getTypeI18n('number'),
+      },
+      {
+        name: i18n.translate('xpack.lens.formula.right', { defaultMessage: 'right' }),
+        type: getTypeI18n('number'),
+      },
     ],
     help: `
+Divides the first number by the second number.
 Also works with ${'`/`'} symbol
-Example: ${'`ceil(sum(bytes))`'}
+
+Example: Calculate profit margin 
+${'`sum(profit) / sum(revenue)`'}
+
+Example: ${'`divide(sum(bytes), 2)`'}
     `,
   },
   abs: {
     positionalArguments: [
-      { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) },
+      {
+        name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }),
+        type: getTypeI18n('number'),
+      },
     ],
     help: `
-Absolute value
-Example: ${'`abs(sum(bytes))`'}
+Calculates absolute value. A negative value is multiplied by -1, a positive value stays the same.
+
+Example: Calculate average distance to sea level ${'`abs(average(altitude))`'}
     `,
   },
   cbrt: {
     positionalArguments: [
-      { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) },
+      {
+        name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }),
+        type: getTypeI18n('number'),
+      },
     ],
     help: `
-Cube root of value
-Example: ${'`cbrt(sum(bytes))`'}
+Cube root of value.
+
+Example: Calculate side length from volume
+${'`cbrt(last_value(volume))`'}
     `,
   },
   ceil: {
     positionalArguments: [
-      { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) },
+      {
+        name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }),
+        type: getTypeI18n('number'),
+      },
     ],
+    // signature: 'ceil(value: number)',
     help: `
-Ceiling of value, rounds up
-Example: ${'`ceil(sum(bytes))`'}
+Ceiling of value, rounds up.
+
+Example: Round up price to the next dollar 
+${'`ceil(sum(price))`'}
     `,
   },
   clamp: {
     positionalArguments: [
-      { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) },
-      { name: i18n.translate('xpack.lens.formula.min', { defaultMessage: 'min' }) },
-      { name: i18n.translate('xpack.lens.formula.max', { defaultMessage: 'max' }) },
+      {
+        name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }),
+        type: getTypeI18n('number'),
+      },
+      {
+        name: i18n.translate('xpack.lens.formula.min', { defaultMessage: 'min' }),
+        type: getTypeI18n('number'),
+      },
+      {
+        name: i18n.translate('xpack.lens.formula.max', { defaultMessage: 'max' }),
+        type: getTypeI18n('number'),
+      },
     ],
+    // signature: 'clamp(value: number, minimum: number, maximum: number)',
     help: `
-Limits the value from a minimum to maximum
-Example: ${'`ceil(sum(bytes))`'}
-    `,
+Limits the value from a minimum to maximum.
+
+Example: Make sure to catch outliers
+\`\`\`
+clamp(
+  average(bytes),
+  percentile(bytes, percentile=5),
+  percentile(bytes, percentile=95)
+)
+\`\`\`
+`,
   },
   cube: {
     positionalArguments: [
-      { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) },
+      {
+        name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }),
+        type: getTypeI18n('number'),
+      },
     ],
     help: `
-Limits the value from a minimum to maximum
-Example: ${'`ceil(sum(bytes))`'}
+Calculates the cube of a number.
+
+Example: Calculate volume from side length
+${'`cube(last_value(length))`'}
     `,
   },
   exp: {
     positionalArguments: [
-      { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) },
+      {
+        name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }),
+        type: getTypeI18n('number'),
+      },
     ],
     help: `
-Raises <em>e</em> to the nth power.
-Example: ${'`exp(sum(bytes))`'}
+Raises *e* to the nth power.
+
+Example: Calculate the natural exponential function
+
+${'`exp(last_value(duration))`'}
     `,
   },
   fix: {
     positionalArguments: [
-      { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) },
+      {
+        name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }),
+        type: getTypeI18n('number'),
+      },
     ],
     help: `
 For positive values, takes the floor. For negative values, takes the ceiling.
-Example: ${'`fix(sum(bytes))`'}
+
+Example: Rounding towards zero
+${'`fix(sum(profit))`'}
     `,
   },
   floor: {
     positionalArguments: [
-      { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) },
+      {
+        name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }),
+        type: getTypeI18n('number'),
+      },
     ],
     help: `
 Round down to nearest integer value
-Example: ${'`floor(sum(bytes))`'}
+
+Example: Round down a price
+${'`floor(sum(price))`'}
     `,
   },
   log: {
     positionalArguments: [
-      { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) },
+      {
+        name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }),
+        type: getTypeI18n('number'),
+      },
       {
         name: i18n.translate('xpack.lens.formula.base', { defaultMessage: 'base' }),
         optional: true,
+        defaultValue: 'e',
+        type: getTypeI18n('number'),
       },
     ],
     help: `
-Logarithm with optional base. The natural base <em>e</em> is used as default.
-Example: ${'`log(sum(bytes))`'}
-Example: ${'`log(sum(bytes), 2)`'}
+Logarithm with optional base. The natural base *e* is used as default.
+
+Example: Calculate number of bits required to store values
+\`\`\`
+log(sum(bytes))
+log(sum(bytes), 2)
+\`\`\`
     `,
   },
   // TODO: check if this is valid for Tinymath
   //   log10: {
   //     positionalArguments: [
-  //       { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) },
+  //       { name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }), type: getTypeI18n('number') },
   //     ],
   //     help: `
   // Base 10 logarithm.
@@ -220,59 +339,89 @@ Example: ${'`log(sum(bytes), 2)`'}
   //   },
   mod: {
     positionalArguments: [
-      { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) },
+      {
+        name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }),
+        type: getTypeI18n('number'),
+      },
       {
         name: i18n.translate('xpack.lens.formula.base', { defaultMessage: 'base' }),
-        optional: true,
+        type: getTypeI18n('number'),
       },
     ],
     help: `
 Remainder after dividing the function by a number
-Example: ${'`mod(sum(bytes), 2)`'}
+
+Example: Calculate last three digits of a value
+${'`mod(sum(price), 1000)`'}
     `,
   },
   pow: {
     positionalArguments: [
-      { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) },
+      {
+        name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }),
+        type: getTypeI18n('number'),
+      },
       {
         name: i18n.translate('xpack.lens.formula.base', { defaultMessage: 'base' }),
+        type: getTypeI18n('number'),
       },
     ],
     help: `
 Raises the value to a certain power. The second argument is required
-Example: ${'`pow(sum(bytes), 3)`'}
+
+Example: Calculate volume based on side length
+${'`pow(last_value(length), 3)`'}
     `,
   },
   round: {
     positionalArguments: [
-      { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) },
+      {
+        name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }),
+        type: getTypeI18n('number'),
+      },
       {
         name: i18n.translate('xpack.lens.formula.decimals', { defaultMessage: 'decimals' }),
         optional: true,
+        defaultValue: 0,
+        type: getTypeI18n('number'),
       },
     ],
     help: `
 Rounds to a specific number of decimal places, default of 0
-Example: ${'`round(sum(bytes))`'}
-Example: ${'`round(sum(bytes), 2)`'}
+
+Examples: Round to the cent
+\`\`\`
+round(sum(bytes))
+round(sum(bytes), 2)
+\`\`\`
     `,
   },
   sqrt: {
     positionalArguments: [
-      { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) },
+      {
+        name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }),
+        type: getTypeI18n('number'),
+      },
     ],
     help: `
 Square root of a positive value only
-Example: ${'`sqrt(sum(bytes))`'}
+
+Example: Calculate side length based on area
+${'`sqrt(last_value(area))`'}
     `,
   },
   square: {
     positionalArguments: [
-      { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) },
+      {
+        name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }),
+        type: getTypeI18n('number'),
+      },
     ],
     help: `
 Raise the value to the 2nd power
-Example: ${'`square(sum(bytes))`'}
+
+Example: Calculate area based on side length
+${'`square(last_value(length))`'}
     `,
   },
 };
@@ -315,3 +464,11 @@ export function findVariables(node: TinymathAST | string): TinymathVariable[] {
   }
   return node.args.flatMap(findVariables);
 }
+
+export function filterByVisibleOperation(
+  operationDefinitionMap: Record<string, GenericOperationDefinition>
+) {
+  return Object.fromEntries(
+    Object.entries(operationDefinitionMap).filter(([, operation]) => !operation.hidden)
+  );
+}

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/validation.ts

@@ -5,7 +5,7 @@
  * 2.0.
  */
 
-import { isObject } from 'lodash';
+import { isObject, partition } from 'lodash';
 import { i18n } from '@kbn/i18n';
 import { parse, TinymathLocation } from '@kbn/tinymath';
 import type { TinymathAST, TinymathFunction, TinymathNamedArgument } from '@kbn/tinymath';
@@ -58,6 +58,10 @@ interface ValidationErrors {
     message: string;
     type: { operation: string; count: number; params: string };
   };
+  tooManyQueries: {
+    message: string;
+    type: {};
+  };
 }
 type ErrorTypes = keyof ValidationErrors;
 type ErrorValues<K extends ErrorTypes> = ValidationErrors[K]['type'];
@@ -90,15 +94,76 @@ export function hasInvalidOperations(
   return {
     // avoid duplicates
     names: Array.from(new Set(nodes.map(({ name }) => name))),
-    locations: nodes.map(({ location }) => location),
+    locations: nodes.map(({ location }) => location).filter((a) => a) as TinymathLocation[],
   };
 }
 
+export const getRawQueryValidationError = (text: string, operations: Record<string, unknown>) => {
+  // try to extract the query context here
+  const singleLine = text.split('\n').join('');
+  const allArgs = singleLine.split(',').filter((args) => /(kql|lucene)/.test(args));
+  // check for the presence of a valid ES operation
+  const containsOneValidOperation = Object.keys(operations).some((operation) =>
+    singleLine.includes(operation)
+  );
+  // no args or no valid operation, no more work to do here
+  if (allArgs.length === 0 || !containsOneValidOperation) {
+    return;
+  }
+  // at this point each entry in allArgs may contain one or more
+  // in the worst case it would be a math chain of count operation
+  // For instance: count(kql=...) + count(lucene=...) - count(kql=...)
+  // therefore before partition them, split them by "count" keywork and filter only string with a length
+  const flattenArgs = allArgs.flatMap((arg) =>
+    arg.split('count').filter((subArg) => /(kql|lucene)/.test(subArg))
+  );
+  const [kqlQueries, luceneQueries] = partition(flattenArgs, (arg) => /kql/.test(arg));
+  const errors = [];
+  for (const kqlQuery of kqlQueries) {
+    const result = validateQueryQuotes(kqlQuery, 'kql');
+    if (result) {
+      errors.push(result);
+    }
+  }
+  for (const luceneQuery of luceneQueries) {
+    const result = validateQueryQuotes(luceneQuery, 'lucene');
+    if (result) {
+      errors.push(result);
+    }
+  }
+  return errors.length ? errors : undefined;
+};
+
+const validateQueryQuotes = (rawQuery: string, language: 'kql' | 'lucene') => {
+  // check if the raw argument has the minimal requirements
+  // use the rest operator here to handle cases where comparison operations are used in the query
+  const [, ...rawValue] = rawQuery.split('=');
+  const fullRawValue = (rawValue || ['']).join('');
+  const cleanedRawValue = fullRawValue.trim();
+  // it must start with a single quote, and quotes must have a closure
+  if (
+    cleanedRawValue.length &&
+    (cleanedRawValue[0] !== "'" || !/'\s*([^']+?)\s*'/.test(fullRawValue)) &&
+    // there's a special case when it's valid as two single quote strings
+    cleanedRawValue !== "''"
+  ) {
+    return i18n.translate('xpack.lens.indexPattern.formulaOperationQueryError', {
+      defaultMessage: `Single quotes are required for {language}='' at {rawQuery}`,
+      values: { language, rawQuery },
+    });
+  }
+};
+
 export const getQueryValidationError = (
-  query: string,
-  language: 'kql' | 'lucene',
+  { value: query, name: language, text }: TinymathNamedArgument,
   indexPattern: IndexPattern
 ): string | undefined => {
+  // check if the raw argument has the minimal requirements
+  const result = validateQueryQuotes(text, language as 'kql' | 'lucene');
+  // forward the error here is ok?
+  if (result) {
+    return result;
+  }
   try {
     if (language === 'kql') {
       esKuery.toElasticsearchQuery(esKuery.fromKueryExpression(query), indexPattern);
@@ -113,7 +178,7 @@ export const getQueryValidationError = (
 
 function getMessageFromId<K extends ErrorTypes>({
   messageId,
-  values: { ...values },
+  values,
   locations,
 }: {
   messageId: K;
@@ -203,6 +268,11 @@ function getMessageFromId<K extends ErrorTypes>({
         values: { operation: out.operation, count: out.count, params: out.params },
       });
       break;
+    case 'tooManyQueries':
+      message = i18n.translate('xpack.lens.indexPattern.formulaOperationDoubleQueryError', {
+        defaultMessage: 'Use only one of kql= or lucene=, not both',
+      });
+      break;
     // case 'mathRequiresFunction':
     //   message = i18n.translate('xpack.lens.indexPattern.formulaMathRequiresFunctionLabel', {
     //     defaultMessage; 'The function {name} requires an Elasticsearch function',
@@ -218,12 +288,22 @@ function getMessageFromId<K extends ErrorTypes>({
 }
 
 export function tryToParse(
-  formula: string
+  formula: string,
+  operations: Record<string, unknown>
 ): { root: TinymathAST; error: null } | { root: null; error: ErrorWrapper } {
   let root;
   try {
     root = parse(formula);
   } catch (e) {
+    // A tradeoff is required here, unless we want to reimplement a full parser
+    // Internally the function has the following logic:
+    // * if the formula contains no existing ES operation, assume it's a plain parse failure
+    // * if the formula contains at least one existing operation, check for query problems
+    const maybeQueryProblems = getRawQueryValidationError(formula, operations);
+    if (maybeQueryProblems) {
+      // need to emulate an error shape here
+      return { root: null, error: { message: maybeQueryProblems[0], locations: [] } };
+    }
     return {
       root: null,
       error: getMessageFromId({
@@ -319,7 +399,10 @@ function getQueryValidationErrors(
   const errors: ErrorWrapper[] = [];
   (namedArguments ?? []).forEach((arg) => {
     if (arg.name === 'kql' || arg.name === 'lucene') {
-      const message = getQueryValidationError(arg.value, arg.name, indexPattern);
+      const message = getQueryValidationError(
+        arg as TinymathNamedArgument & { name: 'kql' | 'lucene' },
+        indexPattern
+      );
       if (message) {
         errors.push({
           message,
@@ -331,6 +414,12 @@ function getQueryValidationErrors(
   return errors;
 }
 
+function checkSingleQuery(namedArguments: TinymathNamedArgument[] | undefined) {
+  return namedArguments
+    ? namedArguments.filter((arg) => arg.name === 'kql' || arg.name === 'lucene').length > 1
+    : undefined;
+}
+
 function validateNameArguments(
   node: TinymathFunction,
   nodeOperation:
@@ -349,7 +438,7 @@ function validateNameArguments(
           operation: node.name,
           params: missingParams.map(({ name }) => name).join(', '),
         },
-        locations: [node.location],
+        locations: node.location ? [node.location] : [],
       })
     );
   }
@@ -362,7 +451,7 @@ function validateNameArguments(
           operation: node.name,
           params: wrongTypeParams.map(({ name }) => name).join(', '),
         },
-        locations: [node.location],
+        locations: node.location ? [node.location] : [],
       })
     );
   }
@@ -375,7 +464,7 @@ function validateNameArguments(
           operation: node.name,
           params: duplicateParams.join(', '),
         },
-        locations: [node.location],
+        locations: node.location ? [node.location] : [],
       })
     );
   }
@@ -383,6 +472,16 @@ function validateNameArguments(
   if (queryValidationErrors.length) {
     errors.push(...queryValidationErrors);
   }
+  const hasTooManyQueries = checkSingleQuery(namedArguments);
+  if (hasTooManyQueries) {
+    errors.push(
+      getMessageFromId({
+        messageId: 'tooManyQueries',
+        values: {},
+        locations: node.location ? [node.location] : [],
+      })
+    );
+  }
   return errors;
 }
 
@@ -426,7 +525,7 @@ function runFullASTValidation(
                     type: 'field',
                     argument: `math operation`,
                   },
-                  locations: [node.location],
+                  locations: node.location ? [node.location] : [],
                 })
               );
             } else {
@@ -436,9 +535,13 @@ function runFullASTValidation(
                   values: {
                     operation: node.name,
                     type: 'field',
-                    argument: getValueOrName(firstArg),
+                    argument:
+                      getValueOrName(firstArg) ||
+                      i18n.translate('xpack.lens.indexPattern.formulaNoFieldForOperation', {
+                        defaultMessage: 'no field',
+                      }),
                   },
-                  locations: [node.location],
+                  locations: node.location ? [node.location] : [],
                 })
               );
             }
@@ -452,7 +555,7 @@ function runFullASTValidation(
                 values: {
                   operation: node.name,
                 },
-                locations: [node.location],
+                locations: node.location ? [node.location] : [],
               })
             );
           }
@@ -464,7 +567,7 @@ function runFullASTValidation(
               values: {
                 operation: node.name,
               },
-              locations: [node.location],
+              locations: node.location ? [node.location] : [],
             })
           );
         } else {
@@ -493,9 +596,13 @@ function runFullASTValidation(
               values: {
                 operation: node.name,
                 type: 'operation',
-                argument: getValueOrName(firstArg),
+                argument:
+                  getValueOrName(firstArg) ||
+                  i18n.translate('xpack.lens.indexPattern.formulaNoOperation', {
+                    defaultMessage: 'no operation',
+                  }),
               },
-              locations: [node.location],
+              locations: node.location ? [node.location] : [],
             })
           );
         }
@@ -506,7 +613,7 @@ function runFullASTValidation(
               values: {
                 operation: node.name,
               },
-              locations: [node.location],
+              locations: node.location ? [node.location] : [],
             })
           );
         } else {
@@ -606,7 +713,11 @@ export function validateParams(
 }
 
 export function shouldHaveFieldArgument(node: TinymathFunction) {
-  return !['count'].includes(node.name);
+  return hasFunctionFieldArgument(node.name);
+}
+
+export function hasFunctionFieldArgument(type: string) {
+  return !['count'].includes(type);
 }
 
 export function isFirstArgumentValidType(arg: TinymathAST, type: TinymathNodeTypes['type']) {
@@ -628,7 +739,7 @@ export function validateMathNodes(root: TinymathAST, missingVariableSet: Set<str
             type: 'operation',
             argument: `()`,
           },
-          locations: [node.location],
+          locations: node.location ? [node.location] : [],
         })
       );
     }
@@ -640,7 +751,7 @@ export function validateMathNodes(root: TinymathAST, missingVariableSet: Set<str
           values: {
             operation: node.name,
           },
-          locations: [node.location],
+          locations: node.location ? [node.location] : [],
         })
       );
     }
@@ -659,7 +770,7 @@ export function validateMathNodes(root: TinymathAST, missingVariableSet: Set<str
           values: {
             operation: node.name,
           },
-          locations: [node.location],
+          locations: node.location ? [node.location] : [],
         })
       );
     }
@@ -678,7 +789,7 @@ export function validateMathNodes(root: TinymathAST, missingVariableSet: Set<str
             count: mandatoryArguments.length - node.args.length,
             params: missingArgs.map(({ name }) => name).join(', '),
           },
-          locations: [node.location],
+          locations: node.location ? [node.location] : [],
         })
       );
     }

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/index.ts

@@ -153,6 +153,9 @@ export interface ParamEditorProps<C> {
   updateLayer: (
     setter: IndexPatternLayer | ((prevLayer: IndexPatternLayer) => IndexPatternLayer)
   ) => void;
+  toggleFullscreen: () => void;
+  setIsCloseable: (isCloseable: boolean) => void;
+  isFullscreen: boolean;
   columnId: string;
   indexPattern: IndexPattern;
   uiSettings: IUiSettingsClient;
@@ -279,6 +282,11 @@ interface BaseOperationDefinitionProps<C extends BaseIndexPatternColumn> {
    * Operations can be used as middleware for other operations, hence not shown in the panel UI
    */
   hidden?: boolean;
+  documentation?: {
+    signature: string;
+    description: string;
+    section: 'elasticsearch' | 'calculation';
+  };
 }
 
 interface BaseBuildColumnArgs {
@@ -290,6 +298,7 @@ interface OperationParam {
   name: string;
   type: string;
   required?: boolean;
+  defaultValue?: string | number;
 }
 
 interface FieldlessOperationDefinition<C extends BaseIndexPatternColumn> {

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/last_value.test.tsx

@@ -30,6 +30,9 @@ const defaultProps = {
     hasRestrictions: false,
   } as IndexPattern,
   operationDefinitionMap: {},
+  isFullscreen: false,
+  toggleFullscreen: jest.fn(),
+  setIsCloseable: jest.fn(),
 };
 
 describe('last_value', () => {

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/last_value.tsx

@@ -277,4 +277,20 @@ export const lastValueOperation: OperationDefinition<LastValueIndexPatternColumn
       </>
     );
   },
+  documentation: {
+    section: 'elasticsearch',
+    signature: i18n.translate('xpack.lens.indexPattern.lastValue.signature', {
+      defaultMessage: 'field: string',
+    }),
+    description: i18n.translate('xpack.lens.indexPattern.lastValue.documentation', {
+      defaultMessage: `
+Returns the value of a field from the last document, ordered by the default time field of the index pattern.
+
+This function is usefull the retrieve the latest state of an entity.
+
+Example: Get the current status of server A: 
+\`last_value(server.status, kql=\'server.name="A"\')\`
+      `,
+    }),
+  },
 };

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/metrics.tsx

@@ -42,6 +42,7 @@ const supportedTypes = ['number', 'histogram'];
 function buildMetricOperation<T extends MetricColumn<string>>({
   type,
   displayName,
+  description,
   ofName,
   priority,
   optionalTimeScaling,
@@ -51,6 +52,7 @@ function buildMetricOperation<T extends MetricColumn<string>>({
   ofName: (name: string) => string;
   priority?: number;
   optionalTimeScaling?: boolean;
+  description?: string;
 }) {
   const labelLookup = (name: string, column?: BaseIndexPatternColumn) => {
     const label = ofName(name);
@@ -67,6 +69,7 @@ function buildMetricOperation<T extends MetricColumn<string>>({
     type,
     priority,
     displayName,
+    description,
     input: 'field',
     timeScalingMode: optionalTimeScaling ? 'optional' : undefined,
     getPossibleOperationForField: ({ aggregationRestrictions, aggregatable, type: fieldType }) => {
@@ -131,6 +134,26 @@ function buildMetricOperation<T extends MetricColumn<string>>({
     getErrorMessage: (layer, columnId, indexPattern) =>
       getInvalidFieldMessage(layer.columns[columnId] as FieldBasedIndexPatternColumn, indexPattern),
     filterable: true,
+    documentation: {
+      section: 'elasticsearch',
+      signature: i18n.translate('xpack.lens.indexPattern.metric.signature', {
+        defaultMessage: 'field: string',
+      }),
+      description: i18n.translate('xpack.lens.indexPattern.metric.documentation', {
+        defaultMessage: `
+Returns the {metric} of a field. This function only works for number fields.
+
+Example: Get the {metric} of price:
+\`{metric}(price)\`
+
+Example: Get the {metric} of price for orders from the UK:
+\`{metric}(price, kql='location:UK')\`
+      `,
+        values: {
+          metric: type,
+        },
+      }),
+    },
     shiftable: true,
   } as OperationDefinition<T, 'field'>;
 }
@@ -151,6 +174,10 @@ export const minOperation = buildMetricOperation<MinIndexPatternColumn>({
       defaultMessage: 'Minimum of {name}',
       values: { name },
     }),
+  description: i18n.translate('xpack.lens.indexPattern.min.description', {
+    defaultMessage:
+      'A single-value metrics aggregation that returns the minimum value among the numeric values extracted from the aggregated documents.',
+  }),
 });
 
 export const maxOperation = buildMetricOperation<MaxIndexPatternColumn>({
@@ -163,6 +190,10 @@ export const maxOperation = buildMetricOperation<MaxIndexPatternColumn>({
       defaultMessage: 'Maximum of {name}',
       values: { name },
     }),
+  description: i18n.translate('xpack.lens.indexPattern.max.description', {
+    defaultMessage:
+      'A single-value metrics aggregation that returns the maximum value among the numeric values extracted from the aggregated documents.',
+  }),
 });
 
 export const averageOperation = buildMetricOperation<AvgIndexPatternColumn>({
@@ -176,6 +207,10 @@ export const averageOperation = buildMetricOperation<AvgIndexPatternColumn>({
       defaultMessage: 'Average of {name}',
       values: { name },
     }),
+  description: i18n.translate('xpack.lens.indexPattern.avg.description', {
+    defaultMessage:
+      'A single-value metric aggregation that computes the average of numeric values that are extracted from the aggregated documents',
+  }),
 });
 
 export const sumOperation = buildMetricOperation<SumIndexPatternColumn>({
@@ -190,6 +225,10 @@ export const sumOperation = buildMetricOperation<SumIndexPatternColumn>({
       values: { name },
     }),
   optionalTimeScaling: true,
+  description: i18n.translate('xpack.lens.indexPattern.sum.description', {
+    defaultMessage:
+      'A single-value metrics aggregation that sums up numeric values that are extracted from the aggregated documents.',
+  }),
 });
 
 export const medianOperation = buildMetricOperation<MedianIndexPatternColumn>({
@@ -203,4 +242,8 @@ export const medianOperation = buildMetricOperation<MedianIndexPatternColumn>({
       defaultMessage: 'Median of {name}',
       values: { name },
     }),
+  description: i18n.translate('xpack.lens.indexPattern.median.description', {
+    defaultMessage:
+      'A single-value metrics aggregation that computes the median value that are extracted from the aggregated documents.',
+  }),
 });

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/percentile.test.tsx

@@ -32,6 +32,9 @@ const defaultProps = {
     hasRestrictions: false,
   } as IndexPattern,
   operationDefinitionMap: {},
+  isFullscreen: false,
+  toggleFullscreen: jest.fn(),
+  setIsCloseable: jest.fn(),
 };
 
 describe('percentile', () => {

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/percentile.tsx

@@ -59,7 +59,9 @@ export const percentileOperation: OperationDefinition<PercentileIndexPatternColu
     defaultMessage: 'Percentile',
   }),
   input: 'field',
-  operationParams: [{ name: 'percentile', type: 'number', required: false }],
+  operationParams: [
+    { name: 'percentile', type: 'number', required: false, defaultValue: DEFAULT_PERCENTILE_VALUE },
+  ],
   filterable: true,
   shiftable: true,
   getPossibleOperationForField: ({ aggregationRestrictions, aggregatable, type: fieldType }) => {
@@ -213,4 +215,18 @@ export const percentileOperation: OperationDefinition<PercentileIndexPatternColu
       </EuiFormRow>
     );
   },
+  documentation: {
+    section: 'elasticsearch',
+    signature: i18n.translate('xpack.lens.indexPattern.percentile.signature', {
+      defaultMessage: 'field: string, [percentile]: number',
+    }),
+    description: i18n.translate('xpack.lens.indexPattern.percentile.documentation', {
+      defaultMessage: `
+Returns the specified percentile of the values of a field. This is the value n percent of the values occuring in documents are smaller.
+
+Example: Get the number of bytes larger than 95 % of values:
+\`percentile(bytes, percentile=95)\`
+      `,
+    }),
+  },
 };

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/ranges/ranges.test.tsx

@@ -102,6 +102,9 @@ const defaultOptions = {
     ]),
   },
   operationDefinitionMap: {},
+  isFullscreen: false,
+  toggleFullscreen: jest.fn(),
+  setIsCloseable: jest.fn(),
 };
 
 describe('ranges', () => {

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/terms/terms.test.tsx

@@ -35,6 +35,9 @@ const defaultProps = {
   http: {} as HttpSetup,
   indexPattern: createMockedIndexPattern(),
   operationDefinitionMap: {},
+  isFullscreen: false,
+  toggleFullscreen: jest.fn(),
+  setIsCloseable: jest.fn(),
 };
 
 describe('terms', () => {

x-pack/plugins/lens/public/indexpattern_datasource/operations/layer_helpers.test.ts

@@ -24,7 +24,7 @@ import type { IndexPattern, IndexPatternLayer } from '../types';
 import { documentField } from '../document_field';
 import { getFieldByNameFactory } from '../pure_helpers';
 import { generateId } from '../../id_generator';
-import { createMockedFullReference } from './mocks';
+import { createMockedFullReference, createMockedManagedReference } from './mocks';
 
 jest.mock('../operations');
 jest.mock('../../id_generator');
@@ -91,10 +91,13 @@ describe('state_helpers', () => {
 
     // @ts-expect-error we are inserting an invalid type
     operationDefinitionMap.testReference = createMockedFullReference();
+    // @ts-expect-error we are inserting an invalid type
+    operationDefinitionMap.managedReference = createMockedManagedReference();
   });
 
   afterEach(() => {
     delete operationDefinitionMap.testReference;
+    delete operationDefinitionMap.managedReference;
   });
 
   describe('copyColumn', () => {
@@ -102,19 +105,19 @@ describe('state_helpers', () => {
       const source = {
         dataType: 'number' as const,
         isBucketed: false,
-        label: 'Formula',
+        label: 'moving_average(sum(bytes), window=5)',
         operationType: 'formula' as const,
         params: {
           formula: 'moving_average(sum(bytes), window=5)',
           isFormulaBroken: false,
         },
-        references: ['formulaX3'],
+        references: ['formulaX1'],
       };
       const math = {
         customLabel: true,
         dataType: 'number' as const,
         isBucketed: false,
-        label: 'math',
+        label: 'formulaX2',
         operationType: 'math' as const,
         params: { tinymathAst: 'formulaX2' },
         references: ['formulaX2'],
@@ -135,7 +138,7 @@ describe('state_helpers', () => {
         label: 'formulaX2',
         operationType: 'moving_average' as const,
         params: { window: 5 },
-        references: ['formulaX1'],
+        references: ['formulaX0'],
       };
       expect(
         copyColumn({
@@ -387,6 +390,42 @@ describe('state_helpers', () => {
       ).toEqual(expect.objectContaining({ columnOrder: ['col1', 'col2', 'col3'] }));
     });
 
+    it('should not change order of metrics and references on inserting new buckets', () => {
+      const layer: IndexPatternLayer = {
+        indexPatternId: '1',
+        columnOrder: ['col1', 'col2'],
+        columns: {
+          col1: {
+            label: 'Cumulative sum of count of records',
+            dataType: 'number',
+            isBucketed: false,
+
+            // Private
+            operationType: 'cumulative_sum',
+            references: ['col2'],
+          },
+          col2: {
+            label: 'Count of records',
+            dataType: 'document',
+            isBucketed: false,
+
+            // Private
+            operationType: 'count',
+            sourceField: 'Records',
+          },
+        },
+      };
+      expect(
+        insertNewColumn({
+          layer,
+          indexPattern,
+          columnId: 'col3',
+          op: 'filters',
+          visualizationGroups: [],
+        })
+      ).toEqual(expect.objectContaining({ columnOrder: ['col3', 'col1', 'col2'] }));
+    });
+
     it('should insert both incomplete states if the aggregation does not support the field', () => {
       expect(
         insertNewColumn({
@@ -2655,6 +2694,36 @@ describe('state_helpers', () => {
       expect(errors).toHaveLength(1);
     });
 
+    it('should only collect the top level errors from managed references', () => {
+      const notCalledMock = jest.fn();
+      const mock = jest.fn().mockReturnValue(['error 1']);
+      operationDefinitionMap.testReference.getErrorMessage = notCalledMock;
+      operationDefinitionMap.managedReference.getErrorMessage = mock;
+      const errors = getErrorMessages(
+        {
+          indexPatternId: '1',
+          columnOrder: [],
+          columns: {
+            col1:
+              // @ts-expect-error not statically analyzed
+              { operationType: 'managedReference', references: ['col2'] },
+            col2: {
+              // @ts-expect-error not statically analyzed
+              operationType: 'testReference',
+              references: [],
+            },
+          },
+        },
+        indexPattern,
+        {},
+        '1',
+        {}
+      );
+      expect(notCalledMock).not.toHaveBeenCalled();
+      expect(mock).toHaveBeenCalledTimes(1);
+      expect(errors).toHaveLength(1);
+    });
+
     it('should ignore incompleteColumns when checking for errors', () => {
       const savedRef = jest.fn().mockReturnValue(['error 1']);
       const incompleteRef = jest.fn();

x-pack/plugins/lens/public/indexpattern_datasource/operations/layer_helpers.ts

@@ -169,6 +169,10 @@ export function insertNewColumn({
     if (field) {
       throw new Error(`Can't create operation ${op} with the provided field ${field.name}`);
     }
+    if (operationDefinition.input === 'managedReference') {
+      // TODO: need to create on the fly the new columns for Formula,
+      // like we do for fullReferences to show a seamless transition
+    }
     const possibleOperation = operationDefinition.getPossibleOperation();
     const isBucketed = Boolean(possibleOperation?.isBucketed);
     const addOperationFn = isBucketed ? addBucket : addMetric;
@@ -358,9 +362,9 @@ export function replaceColumn({
     tempLayer = resetIncomplete(tempLayer, columnId);
 
     if (previousDefinition.input === 'managedReference') {
-      // Every transition away from a managedReference resets it, we don't have a way to keep the state
+      // If the transition is incomplete, leave the managed state until it's finished.
       tempLayer = deleteColumn({ layer: tempLayer, columnId, indexPattern });
-      return insertNewColumn({
+      const hypotheticalLayer = insertNewColumn({
         layer: tempLayer,
         columnId,
         indexPattern,
@@ -368,6 +372,14 @@ export function replaceColumn({
         field,
         visualizationGroups,
       });
+      if (hypotheticalLayer.incompleteColumns && hypotheticalLayer.incompleteColumns[columnId]) {
+        return {
+          ...layer,
+          incompleteColumns: hypotheticalLayer.incompleteColumns,
+        };
+      } else {
+        return hypotheticalLayer;
+      }
     }
 
     if (operationDefinition.input === 'fullReference') {
@@ -859,7 +871,10 @@ function addBucket(
   visualizationGroups: VisualizationDimensionGroupConfig[],
   targetGroup?: string
 ): IndexPatternLayer {
-  const [buckets, metrics, references] = getExistingColumnGroups(layer);
+  const [buckets, metrics] = partition(
+    layer.columnOrder,
+    (colId) => layer.columns[colId].isBucketed
+  );
 
   const oldDateHistogramIndex = layer.columnOrder.findIndex(
     (columnId) => layer.columns[columnId].operationType === 'date_histogram'
@@ -873,12 +888,11 @@ function addBucket(
       addedColumnId,
       ...buckets.slice(oldDateHistogramIndex, buckets.length),
       ...metrics,
-      ...references,
     ];
   } else {
     // Insert the new bucket after existing buckets. Users will see the same data
     // they already had, with an extra level of detail.
-    updatedColumnOrder = [...buckets, addedColumnId, ...metrics, ...references];
+    updatedColumnOrder = [...buckets, addedColumnId, ...metrics];
   }
   updatedColumnOrder = reorderByGroups(
     visualizationGroups,
@@ -1169,8 +1183,20 @@ export function getErrorMessages(
         }
     >
   | undefined {
-  const errors = Object.entries(layer.columns)
+  const columns = Object.entries(layer.columns);
+  const visibleManagedReferences = columns.filter(
+    ([columnId, column]) =>
+      !isReferenced(layer, columnId) &&
+      operationDefinitionMap[column.operationType].input === 'managedReference'
+  );
+  const skippedColumns = visibleManagedReferences.flatMap(([columnId]) =>
+    getManagedColumnsFrom(columnId, layer.columns).map(([id]) => id)
+  );
+  const errors = columns
     .flatMap(([columnId, column]) => {
+      if (skippedColumns.includes(columnId)) {
+        return;
+      }
       const def = operationDefinitionMap[column.operationType];
       if (def.getErrorMessage) {
         return def.getErrorMessage(layer, columnId, indexPattern, operationDefinitionMap);
@@ -1218,6 +1244,25 @@ export function isReferenced(layer: IndexPatternLayer, columnId: string): boolea
   return allReferences.includes(columnId);
 }
 
+export function getReferencedColumnIds(layer: IndexPatternLayer, columnId: string): string[] {
+  const referencedIds: string[] = [];
+  function collect(id: string) {
+    const column = layer.columns[id];
+    if (column && 'references' in column) {
+      const columnReferences = column.references;
+      // only record references which have created columns yet
+      const existingReferences = columnReferences.filter((reference) =>
+        Boolean(layer.columns[reference])
+      );
+      referencedIds.push(...existingReferences);
+      existingReferences.forEach(collect);
+    }
+  }
+  collect(columnId);
+
+  return referencedIds;
+}
+
 export function isOperationAllowedAsReference({
   operationType,
   validation,

x-pack/plugins/lens/public/indexpattern_datasource/operations/mocks.ts

@@ -40,3 +40,28 @@ export const createMockedFullReference = () => {
     getErrorMessage: jest.fn(),
   };
 };
+
+export const createMockedManagedReference = () => {
+  return {
+    input: 'managedReference',
+    displayName: 'Managed reference test',
+    type: 'managedReference' as OperationType,
+    selectionStyle: 'full',
+    buildColumn: jest.fn((args) => {
+      return {
+        label: 'Test reference',
+        isBucketed: false,
+        dataType: 'number',
+
+        operationType: 'testReference',
+        references: args.referenceIds,
+      };
+    }),
+    filterable: true,
+    isTransferable: jest.fn(),
+    toExpression: jest.fn().mockReturnValue([]),
+    getPossibleOperation: jest.fn().mockReturnValue({ dataType: 'number', isBucketed: false }),
+    getDefaultLabel: jest.fn().mockReturnValue('Default label'),
+    getErrorMessage: jest.fn(),
+  };
+};

x-pack/plugins/lens/public/indexpattern_datasource/types.ts

@@ -88,6 +88,8 @@ export interface IndexPatternPrivateState {
   isFirstExistenceFetch: boolean;
   existenceFetchFailed?: boolean;
   existenceFetchTimeout?: boolean;
+
+  isDimensionClosePrevented?: boolean;
 }
 
 export interface IndexPatternRef {

x-pack/plugins/lens/public/mocks.tsx

@@ -166,6 +166,9 @@ export function mockDataPlugin(sessionIdSubject = new Subject<string>()) {
     nowProvider: {
       get: jest.fn(),
     },
+    fieldFormats: {
+      deserialize: jest.fn(),
+    },
   } as unknown) as DataPublicPluginStart;
 }
 

x-pack/plugins/lens/public/types.ts

@@ -198,6 +198,11 @@ export interface Datasource<T = unknown, P = unknown> {
     }
   ) => { dropTypes: DropType[]; nextLabel?: string } | undefined;
   onDrop: (props: DatasourceDimensionDropHandlerProps<T>) => false | true | { deleted: string };
+  /**
+   * The datasource is allowed to cancel a close event on the dimension editor,
+   * mainly used for formulas
+   */
+  canCloseDimensionEditor?: (state: T) => boolean;
   getCustomWorkspaceRenderer?: (
     state: T,
     dragging: DraggingIdentifier
@@ -300,11 +305,15 @@ export type DatasourceDimensionEditorProps<T = unknown> = DatasourceDimensionPro
   // Not a StateSetter because we have this unique use case of determining valid columns
   setState: (
     newState: Parameters<StateSetter<T>>[0],
-    publishToVisualization?: { shouldReplaceDimension?: boolean; shouldRemoveDimension?: boolean }
+    publishToVisualization?: {
+      isDimensionComplete?: boolean;
+    }
   ) => void;
   core: Pick<CoreSetup, 'http' | 'notifications' | 'uiSettings'>;
   dateRange: DateRange;
   dimensionGroups: VisualizationDimensionGroupConfig[];
+  toggleFullscreen: () => void;
+  isFullscreen: boolean;
 };
 
 export type DatasourceDimensionTriggerProps<T> = DatasourceDimensionProps<T>;

x-pack/plugins/lens/server/usage/schema.ts

@@ -14,6 +14,12 @@ const eventsSchema: MakeSchemaFrom<LensUsage['events_30_days']> = {
     type: 'long',
     _meta: { description: 'Number of times the user opened one of the in-product help popovers.' },
   },
+  toggle_fullscreen_formula: {
+    type: 'long',
+    _meta: {
+      description: 'Number of times the user toggled fullscreen mode on formula.',
+    },
+  },
   indexpattern_field_info_click: { type: 'long' },
   loaded: { type: 'long' },
   app_filters_updated: { type: 'long' },
@@ -162,6 +168,10 @@ const eventsSchema: MakeSchemaFrom<LensUsage['events_30_days']> = {
     type: 'long',
     _meta: { description: 'Number of times the moving average function was selected' },
   },
+  indexpattern_dimension_operation_formula: {
+    type: 'long',
+    _meta: { description: 'Number of times the formula function was selected' },
+  },
 };
 
 const suggestionEventsSchema: MakeSchemaFrom<LensUsage['suggestion_events_30_days']> = {
@@ -183,6 +193,12 @@ const savedSchema: MakeSchemaFrom<LensUsage['saved_overall']> = {
   lnsDatatable: { type: 'long' },
   lnsPie: { type: 'long' },
   lnsMetric: { type: 'long' },
+  formula: {
+    type: 'long',
+    _meta: {
+      description: 'Number of saved lens visualizations which are using at least one formula',
+    },
+  },
 };
 
 export const lensUsageSchema: MakeSchemaFrom<LensUsage> = {

x-pack/plugins/lens/server/usage/visualization_counts.ts

@@ -43,6 +43,31 @@ export async function getVisualizationCounts(
                 size: 100,
               },
             },
+            usesFormula: {
+              filter: {
+                match: {
+                  operation_type: 'formula',
+                },
+              },
+            },
+          },
+        },
+      },
+      runtime_mappings: {
+        operation_type: {
+          type: 'keyword',
+          script: {
+            lang: 'painless',
+            source: `try {
+              if(doc['lens.state'].size() == 0) return;
+              HashMap layers = params['_source'].get('lens').get('state').get('datasourceStates').get('indexpattern').get('layers');
+              for(layerId in layers.keySet()) {
+                HashMap columns = layers.get(layerId).get('columns');
+                for(columnId in columns.keySet()) {
+                  emit(columns.get(columnId).get('operationType'))
+                }
+              }
+            } catch(Exception e) {}`,
           },
         },
       },
@@ -56,16 +81,19 @@ export async function getVisualizationCounts(
   // eslint-disable-next-line @typescript-eslint/no-explicit-any
   function bucketsToObject(arg: any) {
     const obj: Record<string, number> = {};
-    arg.buckets.forEach((bucket: { key: string; doc_count: number }) => {
+    arg.byType.buckets.forEach((bucket: { key: string; doc_count: number }) => {
       obj[bucket.key] = bucket.doc_count + (obj[bucket.key] ?? 0);
     });
+    if (arg.usesFormula.doc_count > 0) {
+      obj.formula = arg.usesFormula.doc_count;
+    }
     return obj;
   }
 
   return {
-    saved_overall: bucketsToObject(buckets.overall.byType),
-    saved_30_days: bucketsToObject(buckets.last30.byType),
-    saved_90_days: bucketsToObject(buckets.last90.byType),
+    saved_overall: bucketsToObject(buckets.overall),
+    saved_30_days: bucketsToObject(buckets.last30),
+    saved_90_days: bucketsToObject(buckets.last90),
     saved_overall_total: buckets.overall.doc_count,
     saved_30_days_total: buckets.last30.doc_count,
     saved_90_days_total: buckets.last90.doc_count,

x-pack/plugins/telemetry_collection_xpack/schema/xpack_plugins.json

@@ -2162,6 +2162,12 @@
                 "description": "Number of times the user opened one of the in-product help popovers."
               }
             },
+            "toggle_fullscreen_formula": {
+              "type": "long",
+              "_meta": {
+                "description": "Number of times the user toggled fullscreen mode on formula."
+              }
+            },
             "indexpattern_field_info_click": {
               "type": "long"
             },
@@ -2371,6 +2377,12 @@
               "_meta": {
                 "description": "Number of times the moving average function was selected"
               }
+            },
+            "indexpattern_dimension_operation_formula": {
+              "type": "long",
+              "_meta": {
+                "description": "Number of times the formula function was selected"
+              }
             }
           }
         },
@@ -2385,6 +2397,12 @@
                 "description": "Number of times the user opened one of the in-product help popovers."
               }
             },
+            "toggle_fullscreen_formula": {
+              "type": "long",
+              "_meta": {
+                "description": "Number of times the user toggled fullscreen mode on formula."
+              }
+            },
             "indexpattern_field_info_click": {
               "type": "long"
             },
@@ -2594,6 +2612,12 @@
               "_meta": {
                 "description": "Number of times the moving average function was selected"
               }
+            },
+            "indexpattern_dimension_operation_formula": {
+              "type": "long",
+              "_meta": {
+                "description": "Number of times the formula function was selected"
+              }
             }
           }
         },
@@ -2666,6 +2690,12 @@
             },
             "lnsMetric": {
               "type": "long"
+            },
+            "formula": {
+              "type": "long",
+              "_meta": {
+                "description": "Number of saved lens visualizations which are using at least one formula"
+              }
             }
           }
         },
@@ -2709,6 +2739,12 @@
             },
             "lnsMetric": {
               "type": "long"
+            },
+            "formula": {
+              "type": "long",
+              "_meta": {
+                "description": "Number of saved lens visualizations which are using at least one formula"
+              }
             }
           }
         },
@@ -2752,6 +2788,12 @@
             },
             "lnsMetric": {
               "type": "long"
+            },
+            "formula": {
+              "type": "long",
+              "_meta": {
+                "description": "Number of saved lens visualizations which are using at least one formula"
+              }
             }
           }
         }

x-pack/test/functional/apps/lens/formula.ts

@@ -0,0 +1,198 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import expect from '@kbn/expect';
+import { FtrProviderContext } from '../../ftr_provider_context';
+
+export default function ({ getService, getPageObjects }: FtrProviderContext) {
+  const PageObjects = getPageObjects(['visualize', 'lens', 'common', 'header']);
+  const find = getService('find');
+  const listingTable = getService('listingTable');
+  const browser = getService('browser');
+  const testSubjects = getService('testSubjects');
+
+  describe('lens formula', () => {
+    it('should transition from count to formula', async () => {
+      await PageObjects.visualize.gotoVisualizationLandingPage();
+      await listingTable.searchForItemWithName('lnsXYvis');
+      await PageObjects.lens.clickVisualizeListItemTitle('lnsXYvis');
+      await PageObjects.lens.goToTimeRange();
+
+      await PageObjects.lens.configureDimension({
+        dimension: 'lnsXY_yDimensionPanel > lns-dimensionTrigger',
+        operation: 'average',
+        field: 'bytes',
+        keepOpen: true,
+      });
+
+      await PageObjects.lens.switchToFormula();
+      await PageObjects.header.waitUntilLoadingHasFinished();
+      // .echLegendItem__title is the only viable way of getting the xy chart's
+      // legend item(s), so we're using a class selector here.
+      // 4th item is the other bucket
+      expect(await find.allByCssSelector('.echLegendItem')).to.have.length(3);
+    });
+
+    it('should update and delete a formula', async () => {
+      await PageObjects.visualize.navigateToNewVisualization();
+      await PageObjects.visualize.clickVisType('lens');
+      await PageObjects.lens.goToTimeRange();
+      await PageObjects.lens.switchToVisualization('lnsDatatable');
+
+      await PageObjects.lens.configureDimension({
+        dimension: 'lnsDatatable_metrics > lns-empty-dimension',
+        operation: 'formula',
+        formula: `count(kql=`,
+        keepOpen: true,
+      });
+
+      const input = await find.activeElement();
+      await input.type('*');
+
+      await PageObjects.header.waitUntilLoadingHasFinished();
+      expect(await PageObjects.lens.getDatatableCellText(0, 0)).to.eql('14,005');
+    });
+
+    it('should insert single quotes and escape when needed to create valid KQL', async () => {
+      await PageObjects.visualize.navigateToNewVisualization();
+      await PageObjects.visualize.clickVisType('lens');
+      await PageObjects.lens.goToTimeRange();
+      await PageObjects.lens.switchToVisualization('lnsDatatable');
+
+      await PageObjects.lens.configureDimension({
+        dimension: 'lnsDatatable_metrics > lns-empty-dimension',
+        operation: 'formula',
+        formula: `count(kql=`,
+        keepOpen: true,
+      });
+
+      let input = await find.activeElement();
+      await input.type(' ');
+      await input.pressKeys(browser.keys.ARROW_LEFT);
+      await input.type(`Men's Clothing`);
+
+      await PageObjects.common.sleep(100);
+
+      let element = await find.byCssSelector('.monaco-editor');
+      expect(await element.getVisibleText()).to.equal(`count(kql='Men\\'s Clothing ')`);
+
+      await PageObjects.lens.typeFormula('count(kql=');
+      input = await find.activeElement();
+      await input.type(`Men\'s Clothing`);
+
+      element = await find.byCssSelector('.monaco-editor');
+      expect(await element.getVisibleText()).to.equal(`count(kql='Men\\'s Clothing')`);
+    });
+
+    it('should persist a broken formula on close', async () => {
+      await PageObjects.visualize.navigateToNewVisualization();
+      await PageObjects.visualize.clickVisType('lens');
+      await PageObjects.lens.goToTimeRange();
+      await PageObjects.lens.switchToVisualization('lnsDatatable');
+
+      // Close immediately
+      await PageObjects.lens.configureDimension({
+        dimension: 'lnsDatatable_metrics > lns-empty-dimension',
+        operation: 'formula',
+        formula: `asdf`,
+      });
+
+      expect(await PageObjects.lens.getErrorCount()).to.eql(1);
+    });
+
+    it('should keep the formula when entering expanded mode', async () => {
+      await PageObjects.visualize.navigateToNewVisualization();
+      await PageObjects.visualize.clickVisType('lens');
+      await PageObjects.lens.goToTimeRange();
+      await PageObjects.lens.switchToVisualization('lnsDatatable');
+
+      // Close immediately
+      await PageObjects.lens.configureDimension({
+        dimension: 'lnsDatatable_metrics > lns-empty-dimension',
+        operation: 'formula',
+        formula: `count()`,
+        keepOpen: true,
+      });
+
+      await PageObjects.lens.toggleFullscreen();
+
+      const element = await find.byCssSelector('.monaco-editor');
+      expect(await element.getVisibleText()).to.equal('count()');
+    });
+
+    it('should allow an empty formula combined with a valid formula', async () => {
+      await PageObjects.visualize.navigateToNewVisualization();
+      await PageObjects.visualize.clickVisType('lens');
+      await PageObjects.lens.goToTimeRange();
+      await PageObjects.lens.switchToVisualization('lnsDatatable');
+
+      await PageObjects.lens.configureDimension({
+        dimension: 'lnsDatatable_metrics > lns-empty-dimension',
+        operation: 'formula',
+        formula: `count()`,
+      });
+      await PageObjects.lens.configureDimension({
+        dimension: 'lnsDatatable_metrics > lns-empty-dimension',
+        operation: 'formula',
+      });
+
+      await PageObjects.header.waitUntilLoadingHasFinished();
+      expect(await PageObjects.lens.getErrorCount()).to.eql(0);
+    });
+
+    it('should duplicate a moving average formula and be a valid table', async () => {
+      await PageObjects.visualize.navigateToNewVisualization();
+      await PageObjects.visualize.clickVisType('lens');
+      await PageObjects.lens.goToTimeRange();
+      await PageObjects.lens.switchToVisualization('lnsDatatable');
+
+      await PageObjects.lens.configureDimension({
+        dimension: 'lnsDatatable_rows > lns-empty-dimension',
+        operation: 'date_histogram',
+        field: '@timestamp',
+      });
+
+      await PageObjects.lens.configureDimension({
+        dimension: 'lnsDatatable_metrics > lns-empty-dimension',
+        operation: 'formula',
+        formula: `moving_average(sum(bytes), window=5`,
+        keepOpen: true,
+      });
+      await PageObjects.lens.closeDimensionEditor();
+
+      await PageObjects.lens.dragDimensionToDimension(
+        'lnsDatatable_metrics > lns-dimensionTrigger',
+        'lnsDatatable_metrics > lns-empty-dimension'
+      );
+      expect(await PageObjects.lens.getDatatableCellText(1, 1)).to.eql('222420');
+      expect(await PageObjects.lens.getDatatableCellText(1, 2)).to.eql('222420');
+    });
+
+    it('should keep the formula if the user does not fully transition to a quick function', async () => {
+      await PageObjects.visualize.navigateToNewVisualization();
+      await PageObjects.visualize.clickVisType('lens');
+      await PageObjects.lens.goToTimeRange();
+      await PageObjects.lens.switchToVisualization('lnsDatatable');
+
+      await PageObjects.lens.configureDimension({
+        dimension: 'lnsDatatable_metrics > lns-empty-dimension',
+        operation: 'formula',
+        formula: `count()`,
+        keepOpen: true,
+      });
+
+      await PageObjects.lens.switchToQuickFunctions();
+      await testSubjects.click(`lns-indexPatternDimension-min incompatible`);
+      await PageObjects.common.sleep(1000);
+      await PageObjects.lens.closeDimensionEditor();
+
+      expect(await PageObjects.lens.getDimensionTriggerText('lnsDatatable_metrics', 0)).to.eql(
+        'count()'
+      );
+    });
+  });
+}

x-pack/test/functional/apps/lens/index.ts

@@ -41,6 +41,7 @@ export default function ({ getService, loadTestFile }: FtrProviderContext) {
       loadTestFile(require.resolve('./geo_field'));
       loadTestFile(require.resolve('./lens_reporting'));
       loadTestFile(require.resolve('./lens_tagging'));
+      loadTestFile(require.resolve('./formula'));
 
       // has to be last one in the suite because it overrides saved objects
       loadTestFile(require.resolve('./rollup'));

x-pack/test/functional/apps/lens/smokescreen.ts

@@ -172,10 +172,11 @@ export default function ({ getService, getPageObjects }: FtrProviderContext) {
 
       await testSubjects.existOrFail('indexPattern-dimension-formatDecimals');
 
+      await PageObjects.lens.closeDimensionEditor();
+
       expect(await PageObjects.lens.getDimensionTriggerText('lnsXY_yDimensionPanel')).to.eql(
         'Test of label'
       );
-      await PageObjects.lens.closeDimensionEditor();
     });
 
     it('should be able to add very long labels and still be able to remove a dimension', async () => {
@@ -587,6 +588,57 @@ export default function ({ getService, getPageObjects }: FtrProviderContext) {
       );
     });
 
+    it('should not leave an incomplete column in the visualization config with field-based operation', async () => {
+      await PageObjects.visualize.navigateToNewVisualization();
+      await PageObjects.visualize.clickVisType('lens');
+      await PageObjects.lens.goToTimeRange();
+
+      await PageObjects.lens.configureDimension({
+        dimension: 'lnsXY_yDimensionPanel > lns-empty-dimension',
+        operation: 'min',
+      });
+
+      expect(await PageObjects.lens.getDimensionTriggerText('lnsXY_yDimensionPanel')).to.eql(
+        undefined
+      );
+    });
+
+    it('should not leave an incomplete column in the visualization config with reference-based operations', async () => {
+      await PageObjects.visualize.navigateToNewVisualization();
+      await PageObjects.visualize.clickVisType('lens');
+      await PageObjects.lens.goToTimeRange();
+
+      await PageObjects.lens.configureDimension({
+        dimension: 'lnsXY_xDimensionPanel > lns-empty-dimension',
+        operation: 'date_histogram',
+        field: '@timestamp',
+      });
+      await PageObjects.lens.configureDimension({
+        dimension: 'lnsXY_yDimensionPanel > lns-empty-dimension',
+        operation: 'moving_average',
+        field: 'Records',
+      });
+
+      expect(await PageObjects.lens.getDimensionTriggerText('lnsXY_yDimensionPanel')).to.eql(
+        'Moving average of Count of records'
+      );
+
+      await PageObjects.lens.configureDimension({
+        dimension: 'lnsXY_yDimensionPanel > lns-dimensionTrigger',
+        operation: 'median',
+        isPreviousIncompatible: true,
+        keepOpen: true,
+      });
+
+      expect(await PageObjects.lens.isDimensionEditorOpen()).to.eql(true);
+
+      await PageObjects.lens.closeDimensionEditor();
+
+      expect(await PageObjects.lens.getDimensionTriggerText('lnsXY_yDimensionPanel')).to.eql(
+        undefined
+      );
+    });
+
     it('should transition from unique count to last value', async () => {
       await PageObjects.visualize.navigateToNewVisualization();
       await PageObjects.visualize.clickVisType('lens');

x-pack/test/functional/page_objects/lens_page.ts

@@ -107,6 +107,7 @@ export function LensPageProvider({ getService, getPageObjects }: FtrProviderCont
         isPreviousIncompatible?: boolean;
         keepOpen?: boolean;
         palette?: string;
+        formula?: string;
       },
       layerIndex = 0
     ) {
@@ -114,10 +115,15 @@ export function LensPageProvider({ getService, getPageObjects }: FtrProviderCont
         await testSubjects.click(`lns-layerPanel-${layerIndex} > ${opts.dimension}`);
         await testSubjects.exists(`lns-indexPatternDimension-${opts.operation}`);
       });
-      const operationSelector = opts.isPreviousIncompatible
-        ? `lns-indexPatternDimension-${opts.operation} incompatible`
-        : `lns-indexPatternDimension-${opts.operation}`;
-      await testSubjects.click(operationSelector);
+
+      if (opts.operation === 'formula') {
+        await this.switchToFormula();
+      } else {
+        const operationSelector = opts.isPreviousIncompatible
+          ? `lns-indexPatternDimension-${opts.operation} incompatible`
+          : `lns-indexPatternDimension-${opts.operation}`;
+        await testSubjects.click(operationSelector);
+      }
 
       if (opts.field) {
         const target = await testSubjects.find('indexPattern-dimension-field');
@@ -125,6 +131,10 @@ export function LensPageProvider({ getService, getPageObjects }: FtrProviderCont
         await comboBox.setElement(target, opts.field);
       }
 
+      if (opts.formula) {
+        await this.typeFormula(opts.formula);
+      }
+
       if (opts.palette) {
         await this.setPalette(opts.palette);
       }
@@ -357,7 +367,7 @@ export function LensPageProvider({ getService, getPageObjects }: FtrProviderCont
       await retry.try(async () => {
         await testSubjects.click('lns-palettePicker');
         const currentPalette = await (
-          await find.byCssSelector('[aria-selected=true]')
+          await find.byCssSelector('[role=option][aria-selected=true]')
         ).getAttribute('id');
         expect(currentPalette).to.equal(palette);
       });
@@ -379,6 +389,18 @@ export function LensPageProvider({ getService, getPageObjects }: FtrProviderCont
       });
     },
 
+    async isDimensionEditorOpen() {
+      return await testSubjects.exists('lns-indexPattern-dimensionContainerBack');
+    },
+
+    // closes the dimension editor flyout
+    async closeDimensionEditor() {
+      await retry.try(async () => {
+        await testSubjects.click('lns-indexPattern-dimensionContainerBack');
+        await testSubjects.missingOrFail('lns-indexPattern-dimensionContainerBack');
+      });
+    },
+
     async enableTimeShift() {
       await testSubjects.click('indexPattern-advanced-popover');
       await retry.try(async () => {
@@ -398,14 +420,6 @@ export function LensPageProvider({ getService, getPageObjects }: FtrProviderCont
       await testSubjects.click('errorFixAction');
     },
 
-    // closes the dimension editor flyout
-    async closeDimensionEditor() {
-      await retry.try(async () => {
-        await testSubjects.click('lns-indexPattern-dimensionContainerBack');
-        await testSubjects.missingOrFail('lns-indexPattern-dimensionContainerBack');
-      });
-    },
-
     async isTopLevelAggregation() {
       return await testSubjects.isEuiSwitchChecked('indexPattern-nesting-switch');
     },
@@ -549,7 +563,8 @@ export function LensPageProvider({ getService, getPageObjects }: FtrProviderCont
         });
       }
       const errors = await testSubjects.findAll('configuration-failure-error');
-      return errors?.length ?? 0;
+      const expressionErrors = await testSubjects.findAll('expression-failure');
+      return (errors?.length ?? 0) + (expressionErrors?.length ?? 0);
     },
 
     async searchOnChartSwitch(subVisualizationId: string, searchTerm?: string) {
@@ -1025,5 +1040,27 @@ export function LensPageProvider({ getService, getPageObjects }: FtrProviderCont
       );
       await PageObjects.header.waitUntilLoadingHasFinished();
     },
+
+    async switchToQuickFunctions() {
+      await testSubjects.click('lens-dimensionTabs-quickFunctions');
+    },
+
+    async switchToFormula() {
+      await testSubjects.click('lens-dimensionTabs-formula');
+    },
+
+    async toggleFullscreen() {
+      await testSubjects.click('lnsFormula-fullscreen');
+    },
+
+    async typeFormula(formula: string) {
+      // Formula takes time to open
+      await PageObjects.common.sleep(500);
+      await find.byCssSelector('.monaco-editor');
+      await find.clickByCssSelectorWhenNotDisabled('.monaco-editor');
+      const input = await find.activeElement();
+      await input.clearValueWithKeyboard();
+      await input.type(formula);
+    },
   });
 }