[8.16] [APM] Fix query for transaction marks (#215819) (#216684)

# Backport

This will backport the following commits from `main` to `8.16`:
- [[APM] Fix query for transaction marks
(#215819)](https://github.com/elastic/kibana/pull/215819)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Sunghyun
Kim","email":"cqbqdd11519@gmail.com"},"sourceCommit":{"committedDate":"2025-04-01T15:05:42Z","message":"[APM]
Fix query for transaction marks (#215819)\n\n## Summary\n\nThere is a
bug in kibana 8.17, where no transaction marks are shown in\nthe APM's
transaction waterfall ui.\nThe marks are stored in the field
`transaction.marks.agent` of\ndocuments, but kibana apm server is
querying `transaction.agent.marks`.\n\nThis PR fixes the field
name.\n\n\nI also added `span.id` in the query source to include the
marks in the\nresponse, even if there is no `span.links` in the
transaction info.\n(I found the case from RUM data with
`transaction.marks.agent` but\nwithout `span.links`, so that the
response does not include marks\nbecause there is no `source` field in
the query result)\n\nI am not sure if it's the right way to fix it, as i
have no\nunderstanding about the relationsip between
`transaction.marks.agent`\nand `span.links`, so this PR is more like a
bug report.\n\n\n\n### Checklist\n\nCheck the PR satisfies following
conditions. \n\nReviewers should verify this PR satisfies this list as
well.\n\n- [ ] Any text added follows [EUI's
writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\nsentence case text and includes
[i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n-
[
]\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\nwas
added for features that require explanation or tutorials\n- [ ] [Unit or
functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere
updated or added to match the most common scenarios\n- [ ] If a plugin
configuration key changed, check if it needs to be\nallowlisted in the
cloud and added to the
[docker\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\n-
[ ] This was checked for breaking HTTP API changes, and any
breaking\nchanges have been approved by the breaking-change committee.
The\n`release_note:breaking` label should be applied in these
situations.\n- [ ] [Flaky
Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\nused on any tests changed\n- [ ] The PR description includes the
appropriate Release Notes section,\nand the correct `release_note:*`
label is applied per
the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n\n###
Identify risks\n\nDoes this PR introduce any risks? For example,
consider risks like hard\nto test bugs, performance regression,
potential of data loss.\n\nDescribe the risk, its severity, and
mitigation for each identified\nrisk. Invite stakeholders and evaluate
how to proceed before merging.\n\n- [ ] [See some
risk\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)\n-
[ ]
...\nNone","sha":"d4d1c2b6ddf98859e5d36654eab0ca71cefc8808","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","💝community","backport:all-open","ci:project-deploy-observability","Team:obs-ux-infra_services","v9.1.0"],"title":"[APM]
Fix query for transaction
marks","number":215819,"url":"https://github.com/elastic/kibana/pull/215819","mergeCommit":{"message":"[APM]
Fix query for transaction marks (#215819)\n\n## Summary\n\nThere is a
bug in kibana 8.17, where no transaction marks are shown in\nthe APM's
transaction waterfall ui.\nThe marks are stored in the field
`transaction.marks.agent` of\ndocuments, but kibana apm server is
querying `transaction.agent.marks`.\n\nThis PR fixes the field
name.\n\n\nI also added `span.id` in the query source to include the
marks in the\nresponse, even if there is no `span.links` in the
transaction info.\n(I found the case from RUM data with
`transaction.marks.agent` but\nwithout `span.links`, so that the
response does not include marks\nbecause there is no `source` field in
the query result)\n\nI am not sure if it's the right way to fix it, as i
have no\nunderstanding about the relationsip between
`transaction.marks.agent`\nand `span.links`, so this PR is more like a
bug report.\n\n\n\n### Checklist\n\nCheck the PR satisfies following
conditions. \n\nReviewers should verify this PR satisfies this list as
well.\n\n- [ ] Any text added follows [EUI's
writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\nsentence case text and includes
[i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n-
[
]\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\nwas
added for features that require explanation or tutorials\n- [ ] [Unit or
functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere
updated or added to match the most common scenarios\n- [ ] If a plugin
configuration key changed, check if it needs to be\nallowlisted in the
cloud and added to the
[docker\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\n-
[ ] This was checked for breaking HTTP API changes, and any
breaking\nchanges have been approved by the breaking-change committee.
The\n`release_note:breaking` label should be applied in these
situations.\n- [ ] [Flaky
Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\nused on any tests changed\n- [ ] The PR description includes the
appropriate Release Notes section,\nand the correct `release_note:*`
label is applied per
the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n\n###
Identify risks\n\nDoes this PR introduce any risks? For example,
consider risks like hard\nto test bugs, performance regression,
potential of data loss.\n\nDescribe the risk, its severity, and
mitigation for each identified\nrisk. Invite stakeholders and evaluate
how to proceed before merging.\n\n- [ ] [See some
risk\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)\n-
[ ]
...\nNone","sha":"d4d1c2b6ddf98859e5d36654eab0ca71cefc8808"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/215819","number":215819,"mergeCommit":{"message":"[APM]
Fix query for transaction marks (#215819)\n\n## Summary\n\nThere is a
bug in kibana 8.17, where no transaction marks are shown in\nthe APM's
transaction waterfall ui.\nThe marks are stored in the field
`transaction.marks.agent` of\ndocuments, but kibana apm server is
querying `transaction.agent.marks`.\n\nThis PR fixes the field
name.\n\n\nI also added `span.id` in the query source to include the
marks in the\nresponse, even if there is no `span.links` in the
transaction info.\n(I found the case from RUM data with
`transaction.marks.agent` but\nwithout `span.links`, so that the
response does not include marks\nbecause there is no `source` field in
the query result)\n\nI am not sure if it's the right way to fix it, as i
have no\nunderstanding about the relationsip between
`transaction.marks.agent`\nand `span.links`, so this PR is more like a
bug report.\n\n\n\n### Checklist\n\nCheck the PR satisfies following
conditions. \n\nReviewers should verify this PR satisfies this list as
well.\n\n- [ ] Any text added follows [EUI's
writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\nsentence case text and includes
[i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n-
[
]\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\nwas
added for features that require explanation or tutorials\n- [ ] [Unit or
functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere
updated or added to match the most common scenarios\n- [ ] If a plugin
configuration key changed, check if it needs to be\nallowlisted in the
cloud and added to the
[docker\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\n-
[ ] This was checked for breaking HTTP API changes, and any
breaking\nchanges have been approved by the breaking-change committee.
The\n`release_note:breaking` label should be applied in these
situations.\n- [ ] [Flaky
Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\nused on any tests changed\n- [ ] The PR description includes the
appropriate Release Notes section,\nand the correct `release_note:*`
label is applied per
the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n\n###
Identify risks\n\nDoes this PR introduce any risks? For example,
consider risks like hard\nto test bugs, performance regression,
potential of data loss.\n\nDescribe the risk, its severity, and
mitigation for each identified\nrisk. Invite stakeholders and evaluate
how to proceed before merging.\n\n- [ ] [See some
risk\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)\n-
[ ] ...\nNone","sha":"d4d1c2b6ddf98859e5d36654eab0ca71cefc8808"}}]}]
BACKPORT-->

---------

Co-authored-by: Sunghyun Kim <cqbqdd11519@gmail.com>

api_docs/kbn_apm_types.devdocs.json

@@ -4782,6 +4782,21 @@
         "trackAdoption": false,
         "initialIsOpen": false
       },
+      {
+        "parentPluginId": "@kbn/apm-types",
+        "id": "def-common.TRANSACTION_MARKS_AGENT",
+        "type": "string",
+        "tags": [],
+        "label": "TRANSACTION_MARKS_AGENT",
+        "description": [],
+        "signature": [
+          "\"transaction.marks.agent\""
+        ],
+        "path": "packages/kbn-apm-types/src/es_fields/apm.ts",
+        "deprecated": false,
+        "trackAdoption": false,
+        "initialIsOpen": false
+      },
       {
         "parentPluginId": "@kbn/apm-types",
         "id": "def-common.TRANSACTION_DURATION",

packages/kbn-apm-types/src/es_fields/apm.ts

@@ -59,7 +59,7 @@ export const OBSERVER_LISTENING = 'observer.listening';
 export const PROCESSOR_EVENT = 'processor.event';
 export const PROCESSOR_NAME = 'processor.name';
 
-export const TRANSACTION_AGENT_MARKS = 'transaction.agent.marks';
+export const TRANSACTION_MARKS_AGENT = 'transaction.marks.agent';
 export const TRANSACTION_DURATION = 'transaction.duration.us';
 export const TRANSACTION_DURATION_HISTOGRAM = 'transaction.duration.histogram';
 export const TRANSACTION_DURATION_SUMMARY = 'transaction.duration.summary';

x-pack/plugins/observability_solution/apm/common/es_fields/__snapshots__/es_fields.test.ts.snap

@@ -344,7 +344,7 @@ exports[`Error TIMESTAMP_US 1`] = `1337`;
 
 exports[`Error TRACE_ID 1`] = `"trace id"`;
 
-exports[`Error TRANSACTION_AGENT_MARKS 1`] = `undefined`;
+exports[`Error TRANSACTION_MARKS_AGENT 1`] = `undefined`;
 
 exports[`Error TRANSACTION_DURATION 1`] = `undefined`;
 
@@ -719,7 +719,7 @@ exports[`Span TIMESTAMP_US 1`] = `1337`;
 
 exports[`Span TRACE_ID 1`] = `"trace id"`;
 
-exports[`Span TRANSACTION_AGENT_MARKS 1`] = `undefined`;
+exports[`Span TRANSACTION_MARKS_AGENT 1`] = `undefined`;
 
 exports[`Span TRANSACTION_DURATION 1`] = `undefined`;
 
@@ -1112,7 +1112,7 @@ exports[`Transaction TIMESTAMP_US 1`] = `1337`;
 
 exports[`Transaction TRACE_ID 1`] = `"trace id"`;
 
-exports[`Transaction TRANSACTION_AGENT_MARKS 1`] = `undefined`;
+exports[`Transaction TRANSACTION_MARKS_AGENT 1`] = `undefined`;
 
 exports[`Transaction TRANSACTION_DURATION 1`] = `1337`;
 

x-pack/plugins/observability_solution/apm/server/routes/transactions/__snapshots__/queries.test.ts.snap

@@ -13,7 +13,7 @@ Object {
   "body": Object {
     "_source": Array [
       "span.links",
-      "transaction.agent.marks",
+      "transaction.marks.agent",
     ],
     "fields": Array [
       "trace.id",

x-pack/plugins/observability_solution/apm/server/routes/transactions/get_transaction/index.ts

@@ -23,7 +23,7 @@ import {
   AT_TIMESTAMP,
   PROCESSOR_NAME,
   SPAN_LINKS,
-  TRANSACTION_AGENT_MARKS,
+  TRANSACTION_MARKS_AGENT,
   SERVICE_LANGUAGE_NAME,
   URL_FULL,
   HTTP_REQUEST_METHOD,
@@ -106,7 +106,7 @@ export async function getTransaction({
         },
       },
       fields: [...requiredFields, ...optionalFields],
-      _source: [SPAN_LINKS, TRANSACTION_AGENT_MARKS],
+      _source: [SPAN_LINKS, TRANSACTION_MARKS_AGENT],
     },
   });
 
@@ -119,9 +119,9 @@ export async function getTransaction({
   const event = unflattenKnownApmEventFields(hit.fields, requiredFields);
 
   const source =
-    'span' in hit._source && 'transaction' in hit._source
+    'span' in hit._source || 'transaction' in hit._source
       ? (hit._source as {
-          transaction: Pick<Required<Transaction>['transaction'], 'marks'>;
+          transaction?: Pick<Required<Transaction>['transaction'], 'marks'>;
           span?: Pick<Required<Transaction>['span'], 'links'>;
         })
       : undefined;
@@ -130,7 +130,7 @@ export async function getTransaction({
     ...event,
     transaction: {
       ...event.transaction,
-      marks: source?.transaction.marks,
+      marks: source?.transaction?.marks,
     },
     processor: {
       name: 'transaction',