[Fleet] Add support for Runtime Fields (#161129)

## Summary Closes https://github.com/elastic/kibana/issues/155255 Closes https://github.com/elastic/package-spec/issues/39 Add support in Fleet for Runtime fields, based on these docs: - Defining runtime fields: - https://www.elastic.co/guide/en/elasticsearch/reference/8.8/runtime-mapping-fields.html - https://www.elastic.co/guide/en/elasticsearch/reference/8.8/runtime-retrieving-fields.html - Mapping runtime fields in dynamic templates: - https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html#dynamic-mapping-runtime-fields - Adding runtime fields under groups Given these field definitions in packages: ```yaml - name: bar type: boolean - name: uptime type: keyword - name: runtime_boolean type: boolean runtime: true - name: runtime.day type: keyword runtime: >- emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT)) - name: to_be_long type: long runtime: true - name: runtime.date type: date date_format: 'yyyy-MM-dd' runtime: >- emit(doc['@timestamp'].value.toEpochMilli()) - name: runtime.epoch_milli type: long runtime: >- emit(doc['@timestamp'].value.toEpochMilli()) - name: lowercase type: keyword runtime: >- emit(doc['uppercase'].value.toLowerCase()) - name: labels.* type: long object_type_mapping_type: double runtime: true - name: responses type: group fields: - name: runtime_group_boolean type: boolean runtime: true - name: foo type: boolean ``` and this definition in the manifest ```yaml elasticsearch: index_template: mappings: runtime: day_of_week_two: type: keyword script: source: "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))" ``` This PR adds the required fields into the `mappings` key when installing the package. For this example, the resulting mappings are (just showing the relevant data for these changes): ```json { ".ds-logs-runtime_fields.foo-default-2023.07.10-000001": { "mappings": { "dynamic_templates": [ { "labels.*": { "path_match": "labels.*", "match_mapping_type": "double", "runtime": { "type": "long" } } } ], "runtime": { "day_of_week_two": { "type": "keyword", "script": { "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))", "lang": "painless" } }, "labels.a": { "type": "long" }, "labels.b": { "type": "long" }, "lowercase": { "type": "keyword", "script": { "source": "emit(doc['uppercase'].value.toLowerCase())", "lang": "painless" } }, "responses.runtime_group_boolean": { "type": "boolean" }, "runtime.date": { "type": "date", "script": { "source": "emit(doc['@timestamp'].value.toEpochMilli())", "lang": "painless" }, "format": "yyyy-MM-dd" }, "runtime.day": { "type": "keyword", "script": { "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))", "lang": "painless" } }, "runtime.epoch_milli": { "type": "long", "script": { "source": "emit(doc['@timestamp'].value.toEpochMilli())", "lang": "painless" } }, "runtime_boolean": { "type": "boolean" }, "to_be_long": { "type": "long" } }, "properties": { "@timestamp": { "type": "date", "ignore_malformed": false }, "bar": { "type": "boolean" }, "data_stream": { "properties": { "dataset": { "type": "constant_keyword" }, "namespace": { "type": "constant_keyword" }, "type": { "type": "constant_keyword" } } }, "labels": { "type": "object" }, "message": { "type": "keyword", "ignore_above": 1024 }, "responses": { "properties": { "foo": { "type": "boolean" } } }, "uppercase": { "type": "keyword", "ignore_above": 1024 }, "user": { "properties": { "id": { "type": "keyword", "ignore_above": 1024 } } } } } } } ``` Tested manually installing a package containing runtime field definitions as the example above. Tested also indexing some documents and retrieving the runtime fields: - Indexing documents: ```json POST /logs-runtime_fields.foo-default/_doc/ { "@timestamp": "2023-07-07T13:32:09.000Z", "datastream": { "dataset": "logs-runtime_fields.foo", "namespace": "default", "type": "logs" }, "user": { "id": "8a4f500d" }, "message": "Login successful", "labels": { "a": 1.6, "b": 2.5 }, "uppercase": "SOMETHING", "to_be_long": 1.6, "runtime_boolean": true, "responses.runtime_group_boolean": false } ``` - Retrieving runtime fields (`_source` disabled): ```json GET logs-runtime_fields.foo-default/_search { "fields": [ "@timestamp", "runtime_boolean", "responses.runtime_group_boolean", "runtime.day", "runtime.date", "runtime.epoch_milli", "labels.*", "uppercase", "lowercase", "to_be_long" ], "_source": false } ``` - Output: ```json ... "hits": [ { "_index": ".ds-logs-runtime_fields.foo-default-2023.07.10-000001", "_id": "_7p1P4kBtEvrlGnsxiFN", "_score": 1, "fields": { "uppercase": [ "SOMETHING" ], "runtime.date": [ "2023-07-10" ], "@timestamp": [ "2023-07-10T09:33:09.000Z" ], "lowercase": [ "something" ], "to_be_long": [ 1 ], "runtime_boolean": [ true ], "runtime.day": [ "Monday" ], "labels.a": [ 1 ], "labels.b": [ 2 ], "responses.runtime_group_boolean": [ false ], "runtime.epoch_milli": [ 1688981589000 ] } } ] ... ``` ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2025-04-24 01:38:56 -04:00 · 2023-07-12 09:53:41 +02:00 · 2023-07-12 09:53:41 +02:00 · 9a7cc5a1d1
commit 9a7cc5a1d1
parent 2e2da69857
19 changed files with 689 additions and 11 deletions
--- a/x-pack/plugins/fleet/common/types/models/epm.ts
+++ b/x-pack/plugins/fleet/common/types/models/epm.ts
@ -582,6 +582,7 @@ export interface PackageAssetReference {
 export interface IndexTemplateMappings {
  properties: any;
  dynamic_templates?: any;
+  runtime?: any;
 }

 // This is an index template v2, see https://github.com/elastic/elasticsearch/issues/53101
--- a/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.test.ts
+++ b/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.test.ts
@ -320,11 +320,64 @@ describe('EPM index template install', () => {
    const packageTemplate = componentTemplates['logs-package.dataset@package'].template;

    if (!('settings' in packageTemplate)) {
-      throw new Error('no mappings on package template');
+      throw new Error('no settings on package template');
    }

    expect(packageTemplate.settings?.index?.mapping).toEqual(
      expect.objectContaining({ ignored_malformed: true })
    );
  });
+
+  it('test prepareTemplate to set a runtime field in index_template.mappings', () => {
+    const dataStream = {
+      type: 'logs',
+      dataset: 'package.dataset',
+      title: 'test data stream',
+      release: 'experimental',
+      package: 'package',
+      path: 'path',
+      ingest_pipeline: 'default',
+      elasticsearch: {
+        'index_template.mappings': {
+          runtime: {
+            day_of_week: {
+              type: 'keyword',
+              script: {
+                source:
+                  "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))",
+              },
+            },
+          },
+        },
+      },
+    } as RegistryDataStream;
+
+    const pkg = {
+      name: 'package',
+      version: '0.0.1',
+    };
+
+    const { componentTemplates } = prepareTemplate({
+      pkg,
+      dataStream,
+    });
+
+    const packageTemplate = componentTemplates['logs-package.dataset@package'].template;
+
+    if (!('mappings' in packageTemplate)) {
+      throw new Error('no mappings on package template');
+    }
+
+    expect(packageTemplate.mappings?.runtime).toEqual(
+      expect.objectContaining({
+        day_of_week: {
+          type: 'keyword',
+          script: {
+            source:
+              "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))",
+          },
+        },
+      })
+    );
+  });
 });
--- a/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.ts
+++ b/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.ts
@ -332,6 +332,8 @@ export function buildComponentTemplates(params: {
    (dynampingTemplate) => Object.keys(dynampingTemplate)[0]
  );

+  const mappingsRuntimeFields = merge(mappings.runtime, indexTemplateMappings.runtime ?? {});
+
  const isTimeSeriesEnabledByDefault = registryElasticsearch?.index_mode === 'time_series';
  const isSyntheticSourceEnabledByDefault = registryElasticsearch?.source_mode === 'synthetic';

@ -359,8 +361,11 @@ export function buildComponentTemplates(params: {
      },
      mappings: {
        properties: mappingsProperties,
+        ...(Object.keys(mappingsRuntimeFields).length > 0
+          ? { runtime: mappingsRuntimeFields }
+          : {}),
        dynamic_templates: mappingsDynamicTemplates.length ? mappingsDynamicTemplates : undefined,
-        ...omit(indexTemplateMappings, 'properties', 'dynamic_templates', '_source'),
+        ...omit(indexTemplateMappings, 'properties', 'dynamic_templates', '_source', 'runtime'),
        ...(indexTemplateMappings?._source || sourceModeSynthetic
          ? {
              _source: {
--- a/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.test.ts
+++ b/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.test.ts
@ -1021,6 +1021,107 @@ describe('EPM template', () => {
    expect(JSON.stringify(mappings)).toEqual(JSON.stringify(fieldMapping));
  });

+  it('tests processing runtime fields without script', () => {
+    const textWithRuntimeFieldsLiteralYml = `
+- name: runtime_field
+  type: boolean
+  runtime: true
+`;
+    const runtimeFieldMapping = {
+      properties: {},
+      runtime: {
+        runtime_field: {
+          type: 'boolean',
+        },
+      },
+    };
+    const fields: Field[] = safeLoad(textWithRuntimeFieldsLiteralYml);
+    const processedFields = processFields(fields);
+    const mappings = generateMappings(processedFields);
+    expect(mappings).toEqual(runtimeFieldMapping);
+  });
+
+  it('tests processing runtime fields with painless script', () => {
+    const textWithRuntimeFieldsLiteralYml = `
+- name: day_of_week
+  type: date
+  runtime: |
+    emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))
+`;
+    const runtimeFieldMapping = {
+      properties: {},
+      runtime: {
+        day_of_week: {
+          type: 'date',
+          script: {
+            source:
+              "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))",
+          },
+        },
+      },
+    };
+    const fields: Field[] = safeLoad(textWithRuntimeFieldsLiteralYml);
+    const processedFields = processFields(fields);
+    const mappings = generateMappings(processedFields);
+    expect(mappings).toEqual(runtimeFieldMapping);
+  });
+
+  it('tests processing runtime fields defined in a group', () => {
+    const textWithRuntimeFieldsLiteralYml = `
+- name: responses
+  type: group
+  fields:
+    - name: day_of_week
+      type: date
+      date_format: date_optional_time
+      runtime: |
+        emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))
+`;
+    const runtimeFieldMapping = {
+      properties: {},
+      runtime: {
+        'responses.day_of_week': {
+          type: 'date',
+          format: 'date_optional_time',
+          script: {
+            source:
+              "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))",
+          },
+        },
+      },
+    };
+    const fields: Field[] = safeLoad(textWithRuntimeFieldsLiteralYml);
+    const processedFields = processFields(fields);
+    const mappings = generateMappings(processedFields);
+    expect(mappings).toEqual(runtimeFieldMapping);
+  });
+
+  it('tests processing runtime fields in a dynamic template', () => {
+    const textWithRuntimeFieldsLiteralYml = `
+- name: labels.*
+  type: keyword
+  runtime: true
+`;
+    const runtimeFieldMapping = {
+      properties: {},
+      dynamic_templates: [
+        {
+          'labels.*': {
+            match_mapping_type: 'string',
+            path_match: 'labels.*',
+            runtime: {
+              type: 'keyword',
+            },
+          },
+        },
+      ],
+    };
+    const fields: Field[] = safeLoad(textWithRuntimeFieldsLiteralYml);
+    const processedFields = processFields(fields);
+    const mappings = generateMappings(processedFields);
+    expect(mappings).toEqual(runtimeFieldMapping);
+  });
+
  it('tests priority and index pattern for data stream without dataset_is_prefix', () => {
    const dataStreamDatasetIsPrefixUnset = {
      type: 'metrics',
--- a/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.ts
+++ b/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.ts
@ -38,6 +38,10 @@ interface MultiFields {
  [key: string]: object;
 }

+interface RuntimeFields {
+  [key: string]: any;
+}
+
 export interface IndexTemplateMapping {
  [key: string]: any;
 }
@ -115,6 +119,7 @@ export function getTemplate({
 export function generateMappings(fields: Field[]): IndexTemplateMappings {
  const dynamicTemplates: Array<Record<string, Properties>> = [];
  const dynamicTemplateNames = new Set<string>();
+  const runtimeFields: RuntimeFields = {};

  const { properties } = _generateMappings(fields, {
    addDynamicMapping: (dynamicMapping: {
@ -122,15 +127,19 @@ export function generateMappings(fields: Field[]): IndexTemplateMappings {
      matchingType: string;
      pathMatch: string;
      properties: string;
+      runtimeProperties?: string;
    }) => {
      const name = dynamicMapping.path;
      if (dynamicTemplateNames.has(name)) {
        return;
      }

-      const dynamicTemplate: Properties = {
-        mapping: dynamicMapping.properties,
-      };
+      const dynamicTemplate: Properties = {};
+      if (dynamicMapping.runtimeProperties !== undefined) {
+        dynamicTemplate.runtime = dynamicMapping.runtimeProperties;
+      } else {
+        dynamicTemplate.mapping = dynamicMapping.properties;
+      }

      if (dynamicMapping.matchingType) {
        dynamicTemplate.match_mapping_type = dynamicMapping.matchingType;
@ -139,17 +148,23 @@ export function generateMappings(fields: Field[]): IndexTemplateMappings {
      if (dynamicMapping.pathMatch) {
        dynamicTemplate.path_match = dynamicMapping.pathMatch;
      }
+
      dynamicTemplateNames.add(name);
      dynamicTemplates.push({ [dynamicMapping.path]: dynamicTemplate });
    },
+    addRuntimeField: (runtimeField: { path: string; properties: Properties }) => {
+      runtimeFields[`${runtimeField.path}`] = runtimeField.properties;
+    },
  });

-  return dynamicTemplates.length
-    ? {
-        properties,
-        dynamic_templates: dynamicTemplates,
-      }
-    : { properties };
+  const indexTemplateMappings: IndexTemplateMappings = { properties };
+  if (dynamicTemplates.length > 0) {
+    indexTemplateMappings.dynamic_templates = dynamicTemplates;
+  }
+  if (Object.keys(runtimeFields).length > 0) {
+    indexTemplateMappings.runtime = runtimeFields;
+  }
+  return indexTemplateMappings;
 }

 /**
@ -164,6 +179,7 @@ function _generateMappings(
  fields: Field[],
  ctx: {
    addDynamicMapping: any;
+    addRuntimeField: any;
    groupFieldName?: string;
  }
 ): {
@ -179,6 +195,55 @@ function _generateMappings(
      // If type is not defined, assume keyword
      const type = field.type || 'keyword';

+      if (field.runtime !== undefined) {
+        const path = ctx.groupFieldName ? `${ctx.groupFieldName}.${field.name}` : field.name;
+        let runtimeFieldProps: Properties = getDefaultProperties(field);
+
+        // Is it a dynamic template?
+        if (type === 'object' && field.object_type) {
+          const pathMatch = path.includes('*') ? path : `${path}.*`;
+
+          let dynProperties: Properties = getDefaultProperties(field);
+          let matchingType: string | undefined;
+          switch (field.object_type) {
+            case 'keyword':
+              dynProperties.type = field.object_type;
+              matchingType = field.object_type_mapping_type ?? 'string';
+              break;
+            case 'double':
+            case 'long':
+            case 'boolean':
+              dynProperties = {
+                type: field.object_type,
+                time_series_metric: field.metric_type,
+              };
+              matchingType = field.object_type_mapping_type ?? field.object_type;
+            default:
+              break;
+          }
+
+          // get the runtime properies of this field assuming type equals to object_type
+          const _field = { ...field, type: field.object_type };
+          const fieldProps = generateRuntimeFieldProps(_field);
+
+          if (dynProperties && matchingType) {
+            ctx.addDynamicMapping({
+              path,
+              pathMatch,
+              matchingType,
+              properties: dynProperties,
+              runtimeProperties: fieldProps,
+            });
+          }
+          return;
+        }
+        const fieldProps = generateRuntimeFieldProps(field);
+        runtimeFieldProps = { ...runtimeFieldProps, ...fieldProps };
+
+        ctx.addRuntimeField({ path, properties: runtimeFieldProps });
+        return; // runtime fields should not be added as a property
+      }
+
      if (type === 'object' && field.object_type) {
        const path = ctx.groupFieldName ? `${ctx.groupFieldName}.${field.name}` : field.name;
        const pathMatch = path.includes('*') ? path : `${path}.*`;
@ -435,6 +500,30 @@ function generateDateMapping(field: Field): IndexTemplateMapping {
  return mapping;
 }

+function generateRuntimeFieldProps(field: Field): IndexTemplateMapping {
+  let mapping: IndexTemplateMapping = {};
+  const type = field.type || keyword;
+  switch (type) {
+    case 'integer':
+      mapping.type = 'long';
+      break;
+    case 'date':
+      const dateMappings = generateDateMapping(field);
+      mapping = { ...mapping, ...dateMappings, type: 'date' };
+      break;
+    default:
+      mapping.type = type;
+  }
+
+  if (typeof field.runtime === 'string') {
+    const scriptObject = {
+      source: field.runtime.trim(),
+    };
+    mapping.script = scriptObject;
+  }
+  return mapping;
+}
+
 /**
 * Generates the template name out of the given information
 */
--- a/x-pack/plugins/fleet/server/services/epm/fields/field.ts
+++ b/x-pack/plugins/fleet/server/services/epm/fields/field.ts
@ -39,6 +39,7 @@ export interface Field {
  null_value?: string;
  dimension?: boolean;
  default_field?: boolean;
+  runtime?: boolean | string;

  // Fields specific of the aggregate_metric_double type
  metrics?: string[];
--- a/x-pack/test/fleet_api_integration/apis/epm/index.js
+++ b/x-pack/test/fleet_api_integration/apis/epm/index.js
@ -43,5 +43,6 @@ export default function loadTests({ loadTestFile, getService }) {
    loadTestFile(require.resolve('./install_hidden_datastreams'));
    loadTestFile(require.resolve('./bulk_get_assets'));
    loadTestFile(require.resolve('./install_dynamic_template_metric'));
+    loadTestFile(require.resolve('./install_runtime_field'));
  });
 }
--- a/x-pack/test/fleet_api_integration/apis/epm/install_runtime_field.ts
+++ b/x-pack/test/fleet_api_integration/apis/epm/install_runtime_field.ts
@ -0,0 +1,118 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+/* eslint-disable no-console */
+
+import expect from '@kbn/expect';
+import { FtrProviderContext } from '../../../api_integration/ftr_provider_context';
+import { skipIfNoDockerRegistry } from '../../helpers';
+import { setupFleetAndAgents } from '../agents/services';
+
+export default function (providerContext: FtrProviderContext) {
+  const { getService } = providerContext;
+  const supertest = getService('supertest');
+  const dockerServers = getService('dockerServers');
+  const es = getService('es');
+
+  const testPackage = 'runtime_fields';
+  const testPackageVersion = '0.0.1';
+  const server = dockerServers.get('registry');
+
+  const deletePackage = async (name: string, version: string) => {
+    await supertest.delete(`/api/fleet/epm/packages/${name}/${version}`).set('kbn-xsrf', 'xxxx');
+  };
+
+  describe('package with runtime fields', async () => {
+    skipIfNoDockerRegistry(providerContext);
+    setupFleetAndAgents(providerContext);
+
+    after(async () => {
+      if (server.enabled) {
+        await deletePackage(testPackage, testPackageVersion);
+      }
+    });
+
+    it('should install with runtime fields added in component template', async function () {
+      const templateName = 'logs-runtime_fields.foo@package';
+
+      await supertest
+        .post(`/api/fleet/epm/packages/${testPackage}/${testPackageVersion}`)
+        .set('kbn-xsrf', 'xxxx')
+        .send({ force: true })
+        .expect(200);
+
+      const { body: resp } = await es.transport.request<any>(
+        {
+          method: 'GET',
+          path: `/_component_template/${templateName}`,
+        },
+        { meta: true }
+      );
+
+      const template = resp.component_templates[0].component_template;
+      const runtimeFieldDefinitions = template.template.mappings.runtime;
+      console.log(JSON.stringify(runtimeFieldDefinitions, null, 2));
+
+      expect(runtimeFieldDefinitions).to.eql({
+        day_of_week_two: {
+          script: {
+            source:
+              "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))",
+          },
+          type: 'keyword',
+        },
+        'responses.runtime_group_boolean': {
+          type: 'boolean',
+        },
+        'runtime.date': {
+          format: 'yyyy-MM-dd',
+          script: {
+            source: "emit(doc['@timestamp'].value.toEpochMilli())",
+          },
+          type: 'date',
+        },
+        'runtime.day': {
+          script: {
+            source:
+              "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))",
+          },
+          type: 'keyword',
+        },
+        'runtime.epoch_milli': {
+          type: 'long',
+          script: {
+            source: "emit(doc['@timestamp'].value.toEpochMilli())",
+          },
+        },
+        runtime_boolean: {
+          type: 'boolean',
+        },
+        to_be_long: {
+          type: 'long',
+        },
+        lowercase: {
+          type: 'keyword',
+          script: {
+            source: "emit(doc['uppercase'].value.toLowerCase())",
+          },
+        },
+      });
+
+      const dynamicTemplates = template.template.mappings.dynamic_templates;
+      expect(dynamicTemplates).to.eql([
+        {
+          'labels.*': {
+            path_match: 'labels.*',
+            match_mapping_type: 'double',
+            runtime: {
+              type: 'long',
+            },
+          },
+        },
+      ]);
+    });
+  });
+}
--- a/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/LICENSE.txt
+++ b/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/LICENSE.txt
@ -0,0 +1,93 @@
+Elastic License 2.0
+
+URL: https://www.elastic.co/licensing/elastic-license
+
+## Acceptance
+
+By using the software, you agree to all of the terms and conditions below.
+
+## Copyright License
+
+The licensor grants you a non-exclusive, royalty-free, worldwide,
+non-sublicensable, non-transferable license to use, copy, distribute, make
+available, and prepare derivative works of the software, in each case subject to
+the limitations and conditions below.
+
+## Limitations
+
+You may not provide the software to third parties as a hosted or managed
+service, where the service provides users with access to any substantial set of
+the features or functionality of the software.
+
+You may not move, change, disable, or circumvent the license key functionality
+in the software, and you may not remove or obscure any functionality in the
+software that is protected by the license key.
+
+You may not alter, remove, or obscure any licensing, copyright, or other notices
+of the licensor in the software. Any use of the licensor’s trademarks is subject
+to applicable law.
+
+## Patents
+
+The licensor grants you a license, under any patent claims the licensor can
+license, or becomes able to license, to make, have made, use, sell, offer for
+sale, import and have imported the software, in each case subject to the
+limitations and conditions in this license. This license does not cover any
+patent claims that you cause to be infringed by modifications or additions to
+the software. If you or your company make any written claim that the software
+infringes or contributes to infringement of any patent, your patent license for
+the software granted under these terms ends immediately. If your company makes
+such a claim, your patent license ends immediately for work on behalf of your
+company.
+
+## Notices
+
+You must ensure that anyone who gets a copy of any part of the software from you
+also gets a copy of these terms.
+
+If you modify the software, you must include in any modified copies of the
+software prominent notices stating that you have modified the software.
+
+## No Other Rights
+
+These terms do not imply any licenses other than those expressly granted in
+these terms.
+
+## Termination
+
+If you use the software in violation of these terms, such use is not licensed,
+and your licenses will automatically terminate. If the licensor provides you
+with a notice of your violation, and you cease all violation of this license no
+later than 30 days after you receive that notice, your licenses will be
+reinstated retroactively. However, if you violate these terms after such
+reinstatement, any additional violation of these terms will cause your licenses
+to terminate automatically and permanently.
+
+## No Liability
+
+*As far as the law allows, the software comes as is, without any warranty or
+condition, and the licensor will not be liable to you for any damages arising
+out of these terms or the use or nature of the software, under any kind of
+legal claim.*
+
+## Definitions
+
+The **licensor** is the entity offering these terms, and the **software** is the
+software the licensor makes available under these terms, including any portion
+of it.
+
+**you** refers to the individual or entity agreeing to these terms.
+
+**your company** is any legal entity, sole proprietorship, or other kind of
+organization that you work for, plus all organizations that have control over,
+are under the control of, or are under common control with that
+organization. **control** means ownership of substantially all the assets of an
+entity, or the power to direct its management and policies by vote, contract, or
+otherwise. Control can be direct or indirect.
+
+**your licenses** are all the licenses granted to you for the software under
+these terms.
+
+**use** means anything you do with the software requiring one of your licenses.
+
+**trademark** means trademarks, service marks, and similar rights.
--- a/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/changelog.yml
+++ b/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/changelog.yml
@ -0,0 +1,6 @@
+# newer versions go on top
+- version: "0.0.1"
+  changes:
+    - description: Initial draft of the package
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/1 # FIXME Replace with the real PR link
--- a/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/data_stream/foo/agent/stream/stream.yml.hbs
+++ b/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/data_stream/foo/agent/stream/stream.yml.hbs
@ -0,0 +1,7 @@
+paths:
+{{#each paths as |path i|}}
+  - {{path}}
+{{/each}}
+exclude_files: [".gz$"]
+processors:
+  - add_locale: ~
--- a/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/data_stream/foo/elasticsearch/ingest_pipeline/default.yml
+++ b/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/data_stream/foo/elasticsearch/ingest_pipeline/default.yml
@ -0,0 +1,13 @@
+---
+description: Pipeline for processing sample logs
+processors:
+- set:
+    field: sample_field
+    value: "1"
+- set:
+    field: my_date
+    copy_from: "@timestamp"
+on_failure:
+- set:
+    field: error.message
+    value: '{{ _ingest.on_failure_message }}'
--- a/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/data_stream/foo/fields/base-fields.yml
+++ b/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/data_stream/foo/fields/base-fields.yml
@ -0,0 +1,12 @@
+- name: data_stream.type
+  type: constant_keyword
+  description: Data stream type.
+- name: data_stream.dataset
+  type: constant_keyword
+  description: Data stream dataset.
+- name: data_stream.namespace
+  type: constant_keyword
+  description: Data stream namespace.
+- name: '@timestamp'
+  type: date
+  description: Event timestamp.
--- a/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/data_stream/foo/fields/fields.yml
+++ b/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/data_stream/foo/fields/fields.yml
@ -0,0 +1,39 @@
+- name: bar
+  type: boolean
+- name: uppercase
+  type: keyword
+- name: runtime_boolean
+  type: boolean
+  runtime: true
+- name: to_be_long
+  type: long
+  runtime: true
+- name: runtime.day
+  type: keyword
+  runtime: >-
+    emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))
+- name: lowercase
+  type: keyword
+  runtime: >-
+    emit(doc['uppercase'].value.toLowerCase())
+- name: runtime.date
+  type: date
+  date_format: 'yyyy-MM-dd'
+  runtime: >-
+    emit(doc['@timestamp'].value.toEpochMilli())
+- name: runtime.epoch_milli
+  type: long
+  runtime: >-
+    emit(doc['@timestamp'].value.toEpochMilli())
+- name: labels.*
+  type: long
+  object_type_mapping_type: double
+  runtime: true
+- name: responses
+  type: group
+  fields:
+    - name: runtime_group_boolean
+      type: boolean
+      runtime: true
+    - name: foo
+      type: boolean
--- a/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/data_stream/foo/manifest.yml
+++ b/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/data_stream/foo/manifest.yml
@ -0,0 +1,21 @@
+title: "Foo"
+type: logs
+streams:
+  - input: logfile
+    title: Sample logs
+    description: Collect sample logs
+    vars:
+      - name: paths
+        type: text
+        title: Paths
+        multi: true
+        default:
+          - /var/log/*.log
+elasticsearch:
+  index_template:
+    mappings:
+      runtime:
+        day_of_week_two:
+          type: keyword
+          script:
+            source: "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
--- a/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/docs/README.md
+++ b/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/docs/README.md
@ -0,0 +1,84 @@
+<!-- Use this template language as a starting point, replacing {placeholder text} with details about the integration. -->
+<!-- Find more detailed documentation guidelines in https://github.com/elastic/integrations/blob/main/docs/documentation_guidelines.md -->
+
+# Runtime fields
+
+<!-- The Runtime fields integration allows you to monitor {name of service}. {name of service} is {describe service}.
+
+Use the Runtime fields integration to {purpose}. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference {data stream type} when troubleshooting an issue.
+
+For example, if you wanted to {sample use case} you could {action}. Then you can {visualize|alert|troubleshoot} by {action}. -->
+
+## Data streams
+
+<!-- The Runtime fields integration collects {one|two} type{s} of data streams: {logs and/or metrics}. -->
+
+<!-- If applicable -->
+<!-- **Logs** help you keep a record of events happening in {service}.
+Log data streams collected by the {name} integration include {sample data stream(s)} and more. See more details in the [Logs](#logs-reference). -->
+
+<!-- If applicable -->
+<!-- **Metrics** give you insight into the state of {service}.
+Metric data streams collected by the {name} integration include {sample data stream(s)} and more. See more details in the [Metrics](#metrics-reference). -->
+
+<!-- Optional: Any additional notes on data streams -->
+
+## Requirements
+
+You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it.
+You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.
+
+<!--
+	Optional: Other requirements including:
+	* System compatibility
+	* Supported versions of third-party products
+	* Permissions needed
+	* Anything else that could block a user from successfully using the integration
+-->
+
+## Setup
+
+<!-- Any prerequisite instructions -->
+
+For step-by-step instructions on how to set up an integration, see the
+[Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide.
+
+<!-- Additional set up instructions -->
+
+<!-- If applicable -->
+<!-- ## Logs reference -->
+
+<!-- Repeat for each data stream of the current type -->
+<!-- ### {Data stream name}
+
+The `{data stream name}` data stream provides events from {source} of the following types: {list types}. -->
+
+<!-- Optional -->
+<!-- #### Example
+
+An example event for `{data stream name}` looks as following:
+
+{code block with example} -->
+
+<!-- #### Exported fields
+
+{insert table} -->
+
+<!-- If applicable -->
+<!-- ## Metrics reference -->
+
+<!-- Repeat for each data stream of the current type -->
+<!-- ### {Data stream name}
+
+The `{data stream name}` data stream provides events from {source} of the following types: {list types}. -->
+
+<!-- Optional -->
+<!-- #### Example
+
+An example event for `{data stream name}` looks as following:
+
+{code block with example} -->
+
+<!-- #### Exported fields
+
+{insert table} -->
--- a/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/img/sample-logo.svg
+++ b/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/img/sample-logo.svg
@ -0,0 +1 @@
+<svg width="32" height="32" fill="none" viewBox="0 0 32 32" xmlns="http://www.w3.org/2000/svg" class="euiIcon euiIcon--xxLarge" focusable="false" role="img" aria-hidden="true"><path fill="#FFF" d="M32 16.77a6.334 6.334 0 00-1.14-3.641 6.298 6.298 0 00-3.02-2.32 9.098 9.098 0 00-.873-5.965A9.05 9.05 0 0022.56.746a9.007 9.007 0 00-5.994-.419 9.037 9.037 0 00-4.93 3.446 4.789 4.789 0 00-5.78-.07A4.833 4.833 0 004.198 9.26a6.384 6.384 0 00-3.035 2.33A6.42 6.42 0 000 15.242 6.341 6.341 0 001.145 18.9a6.305 6.305 0 003.039 2.321 9.334 9.334 0 00-.16 1.725 9.067 9.067 0 001.727 5.333 9.014 9.014 0 004.526 3.287 8.982 8.982 0 005.587-.023 9.016 9.016 0 004.5-3.322 4.789 4.789 0 005.77.074 4.833 4.833 0 001.672-5.542 6.383 6.383 0 003.032-2.331A6.419 6.419 0 0032 16.77z"></path><path fill="#FEC514" d="M12.58 13.787l7.002 3.211 7.066-6.213a7.854 7.854 0 00.152-1.557 7.944 7.944 0 00-1.54-4.704 7.897 7.897 0 00-4.02-2.869 7.87 7.87 0 00-4.932.086 7.9 7.9 0 00-3.92 3.007l-1.174 6.118 1.367 2.92z"></path><path fill="#00BFB3" d="M5.333 21.228A7.964 7.964 0 006.72 27.53a7.918 7.918 0 004.04 2.874 7.89 7.89 0 004.95-.097 7.921 7.921 0 003.926-3.03l1.166-6.102-1.555-2.985-7.03-3.211-6.885 6.248z"></path><path fill="#F04E98" d="M5.288 9.067l4.8 1.137L11.14 4.73a3.785 3.785 0 00-4.538-.023A3.82 3.82 0 005.29 9.065"></path><path fill="#1BA9F5" d="M4.872 10.214a5.294 5.294 0 00-2.595 1.882 5.324 5.324 0 00-.142 6.124 5.287 5.287 0 002.505 2l6.733-6.101-1.235-2.65-5.266-1.255z"></path><path fill="#93C90E" d="M20.873 27.277a3.737 3.737 0 002.285.785 3.783 3.783 0 003.101-1.63 3.813 3.813 0 00.451-3.484l-4.8-1.125-1.037 5.454z"></path><path fill="#07C" d="M21.848 20.563l5.28 1.238a5.34 5.34 0 002.622-1.938 5.37 5.37 0 001.013-3.106 5.312 5.312 0 00-.936-3.01 5.283 5.283 0 00-2.475-1.944l-6.904 6.07 1.4 2.69z"></path></svg>
--- a/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/img/sample-screenshot.png
+++ b/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/img/sample-screenshot.png
--- a/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/manifest.yml
+++ b/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/runtime_fields/0.0.1/manifest.yml
@ -0,0 +1,33 @@
+format_version: 2.8.0
+name: runtime_fields
+title: "Runtime fields"
+version: 0.0.1
+source:
+  license: "Elastic-2.0"
+description: "This is a new package."
+type: integration
+categories:
+  - custom
+conditions:
+  kibana.version: "^8.10.0"
+  elastic.subscription: "basic"
+screenshots:
+  - src: /img/sample-screenshot.png
+    title: Sample screenshot
+    size: 600x600
+    type: image/png
+icons:
+  - src: /img/sample-logo.svg
+    title: Sample logo
+    size: 32x32
+    type: image/svg+xml
+policy_templates:
+  - name: sample
+    title: Sample logs
+    description: Collect sample logs
+    inputs:
+      - type: logfile
+        title: Collect sample logs from instances
+        description: Collecting sample logs
+owner:
+  github: elastic/integrations
				`@ -0,0 +1 @@`
				<svg width="32" height="32" fill="none" viewBox="0 0 32 32" xmlns="http://www.w3.org/2000/svg" class="euiIcon euiIcon--xxLarge" focusable="false" role="img" aria-hidden="true"><path fill="#FFF" d="M32 16.77a6.334 6.334 0 00-1.14-3.641 6.298 6.298 0 00-3.02-2.32 9.098 9.098 0 00-.873-5.965A9.05 9.05 0 0022.56.746a9.007 9.007 0 00-5.994-.419 9.037 9.037 0 00-4.93 3.446 4.789 4.789 0 00-5.78-.07A4.833 4.833 0 004.198 9.26a6.384 6.384 0 00-3.035 2.33A6.42 6.42 0 000 15.242 6.341 6.341 0 001.145 18.9a6.305 6.305 0 003.039 2.321 9.334 9.334 0 00-.16 1.725 9.067 9.067 0 001.727 5.333 9.014 9.014 0 004.526 3.287 8.982 8.982 0 005.587-.023 9.016 9.016 0 004.5-3.322 4.789 4.789 0 005.77.074 4.833 4.833 0 001.672-5.542 6.383 6.383 0 003.032-2.331A6.419 6.419 0 0032 16.77z"></path><path fill="#FEC514" d="M12.58 13.787l7.002 3.211 7.066-6.213a7.854 7.854 0 00.152-1.557 7.944 7.944 0 00-1.54-4.704 7.897 7.897 0 00-4.02-2.869 7.87 7.87 0 00-4.932.086 7.9 7.9 0 00-3.92 3.007l-1.174 6.118 1.367 2.92z"></path><path fill="#00BFB3" d="M5.333 21.228A7.964 7.964 0 006.72 27.53a7.918 7.918 0 004.04 2.874 7.89 7.89 0 004.95-.097 7.921 7.921 0 003.926-3.03l1.166-6.102-1.555-2.985-7.03-3.211-6.885 6.248z"></path><path fill="#F04E98" d="M5.288 9.067l4.8 1.137L11.14 4.73a3.785 3.785 0 00-4.538-.023A3.82 3.82 0 005.29 9.065"></path><path fill="#1BA9F5" d="M4.872 10.214a5.294 5.294 0 00-2.595 1.882 5.324 5.324 0 00-.142 6.124 5.287 5.287 0 002.505 2l6.733-6.101-1.235-2.65-5.266-1.255z"></path><path fill="#93C90E" d="M20.873 27.277a3.737 3.737 0 002.285.785 3.783 3.783 0 003.101-1.63 3.813 3.813 0 00.451-3.484l-4.8-1.125-1.037 5.454z"></path><path fill="#07C" d="M21.848 20.563l5.28 1.238a5.34 5.34 0 002.622-1.938 5.37 5.37 0 001.013-3.106 5.312 5.312 0 00-.936-3.01 5.283 5.283 0 00-2.475-1.944l-6.904 6.07 1.4 2.69z"></path></svg>