[8.15] [Playground] [Bug] Continue to use text_expansion when rank_features field (#188232) (#188303)

# Backport

This will backport the following commits from `main` to `8.15`:
- [[Playground] [Bug] Continue to use text_expansion when rank_features
field (#188232)](https://github.com/elastic/kibana/pull/188232)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Joe
McElroy","email":"joseph.mcelroy@elastic.co"},"sourceCommit":{"committedDate":"2024-07-15T11:26:46Z","message":"[Playground]
[Bug] Continue to use text_expansion when rank_features field
(#188232)\n\n## Summary\r\n\r\nwe switched over to use `sparse_vector`
query clause for both\r\n`sparse_vector` and `rank_features`. Due to a
recent design\r\ndecision,`rank_features` fields no longer supports
`sparse_vector` query\r\nclause and we therefore need to fallback to the
`text_expansion` query\r\ninstead.\r\n\r\n### Checklist\r\n\r\nDelete
any items that are not applicable to this PR.\r\n\r\n- [ ] Any text
added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [x] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n- [ ] Any UI touched in this PR is
usable by keyboard only (learn more\r\nabout [keyboard
accessibility](https://webaim.org/techniques/keyboard/))\r\n- [ ] Any UI
touched in this PR does not create any new axe failures\r\n(run axe in
browser:\r\n[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),\r\n[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))\r\n-
[ ] If a plugin configuration key changed, check if it needs to
be\r\nallowlisted in the cloud and added to the
[docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n-
[ ] This renders correctly on smaller devices using a
responsive\r\nlayout. (You can test this [in
your\r\nbrowser](https://www.browserstack.com/guide/responsive-testing-on-local-server))\r\n-
[ ] This was checked for
[cross-browser\r\ncompatibility](https://www.elastic.co/support/matrix#matrix_browsers)\r\n\r\n\r\n###
Risk Matrix\r\n\r\nDelete this section if it is not applicable to this
PR.\r\n\r\nBefore closing this PR, invite QA, stakeholders, and other
developers to\r\nidentify risks that should be tested prior to the
change/feature\r\nrelease.\r\n\r\nWhen forming the risk matrix, consider
some of the following examples\r\nand how they may potentially impact
the change:\r\n\r\n| Risk | Probability | Severity | Mitigation/Notes
|\r\n\r\n|---------------------------|-------------|----------|-------------------------|\r\n|
Multiple Spaces&mdash;unexpected behavior in non-default Kibana
Space.\r\n| Low | High | Integration tests will verify that all features
are still\r\nsupported in non-default Kibana Space and when user
switches between\r\nspaces. |\r\n| Multiple nodes&mdash;Elasticsearch
polling might have race conditions\r\nwhen multiple Kibana nodes are
polling for the same tasks. | High | Low\r\n| Tasks are idempotent, so
executing them multiple times will not result\r\nin logical error, but
will degrade performance. To test for this case we\r\nadd plenty of unit
tests around this logic and document manual testing\r\nprocedure. |\r\n|
Code should gracefully handle cases when feature X or plugin Y
are\r\ndisabled. | Medium | High | Unit tests will verify that any
feature flag\r\nor plugin combination still results in our service
operational. |\r\n| [See more potential
risk\r\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
|\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for
breaking API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"dc577c0bf1e02116287e2c5c617223094d34af45","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:EnterpriseSearch","v8.15.0","v8.16.0"],"title":"[Playground]
[Bug] Continue to use text_expansion when rank_features
field","number":188232,"url":"https://github.com/elastic/kibana/pull/188232","mergeCommit":{"message":"[Playground]
[Bug] Continue to use text_expansion when rank_features field
(#188232)\n\n## Summary\r\n\r\nwe switched over to use `sparse_vector`
query clause for both\r\n`sparse_vector` and `rank_features`. Due to a
recent design\r\ndecision,`rank_features` fields no longer supports
`sparse_vector` query\r\nclause and we therefore need to fallback to the
`text_expansion` query\r\ninstead.\r\n\r\n### Checklist\r\n\r\nDelete
any items that are not applicable to this PR.\r\n\r\n- [ ] Any text
added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [x] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n- [ ] Any UI touched in this PR is
usable by keyboard only (learn more\r\nabout [keyboard
accessibility](https://webaim.org/techniques/keyboard/))\r\n- [ ] Any UI
touched in this PR does not create any new axe failures\r\n(run axe in
browser:\r\n[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),\r\n[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))\r\n-
[ ] If a plugin configuration key changed, check if it needs to
be\r\nallowlisted in the cloud and added to the
[docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n-
[ ] This renders correctly on smaller devices using a
responsive\r\nlayout. (You can test this [in
your\r\nbrowser](https://www.browserstack.com/guide/responsive-testing-on-local-server))\r\n-
[ ] This was checked for
[cross-browser\r\ncompatibility](https://www.elastic.co/support/matrix#matrix_browsers)\r\n\r\n\r\n###
Risk Matrix\r\n\r\nDelete this section if it is not applicable to this
PR.\r\n\r\nBefore closing this PR, invite QA, stakeholders, and other
developers to\r\nidentify risks that should be tested prior to the
change/feature\r\nrelease.\r\n\r\nWhen forming the risk matrix, consider
some of the following examples\r\nand how they may potentially impact
the change:\r\n\r\n| Risk | Probability | Severity | Mitigation/Notes
|\r\n\r\n|---------------------------|-------------|----------|-------------------------|\r\n|
Multiple Spaces&mdash;unexpected behavior in non-default Kibana
Space.\r\n| Low | High | Integration tests will verify that all features
are still\r\nsupported in non-default Kibana Space and when user
switches between\r\nspaces. |\r\n| Multiple nodes&mdash;Elasticsearch
polling might have race conditions\r\nwhen multiple Kibana nodes are
polling for the same tasks. | High | Low\r\n| Tasks are idempotent, so
executing them multiple times will not result\r\nin logical error, but
will degrade performance. To test for this case we\r\nadd plenty of unit
tests around this logic and document manual testing\r\nprocedure. |\r\n|
Code should gracefully handle cases when feature X or plugin Y
are\r\ndisabled. | Medium | High | Unit tests will verify that any
feature flag\r\nor plugin combination still results in our service
operational. |\r\n| [See more potential
risk\r\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
|\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for
breaking API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"dc577c0bf1e02116287e2c5c617223094d34af45"}},"sourceBranch":"main","suggestedTargetBranches":["8.15"],"targetPullRequestStates":[{"branch":"8.15","label":"v8.15.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/188232","number":188232,"mergeCommit":{"message":"[Playground]
[Bug] Continue to use text_expansion when rank_features field
(#188232)\n\n## Summary\r\n\r\nwe switched over to use `sparse_vector`
query clause for both\r\n`sparse_vector` and `rank_features`. Due to a
recent design\r\ndecision,`rank_features` fields no longer supports
`sparse_vector` query\r\nclause and we therefore need to fallback to the
`text_expansion` query\r\ninstead.\r\n\r\n### Checklist\r\n\r\nDelete
any items that are not applicable to this PR.\r\n\r\n- [ ] Any text
added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [x] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n- [ ] Any UI touched in this PR is
usable by keyboard only (learn more\r\nabout [keyboard
accessibility](https://webaim.org/techniques/keyboard/))\r\n- [ ] Any UI
touched in this PR does not create any new axe failures\r\n(run axe in
browser:\r\n[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),\r\n[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))\r\n-
[ ] If a plugin configuration key changed, check if it needs to
be\r\nallowlisted in the cloud and added to the
[docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n-
[ ] This renders correctly on smaller devices using a
responsive\r\nlayout. (You can test this [in
your\r\nbrowser](https://www.browserstack.com/guide/responsive-testing-on-local-server))\r\n-
[ ] This was checked for
[cross-browser\r\ncompatibility](https://www.elastic.co/support/matrix#matrix_browsers)\r\n\r\n\r\n###
Risk Matrix\r\n\r\nDelete this section if it is not applicable to this
PR.\r\n\r\nBefore closing this PR, invite QA, stakeholders, and other
developers to\r\nidentify risks that should be tested prior to the
change/feature\r\nrelease.\r\n\r\nWhen forming the risk matrix, consider
some of the following examples\r\nand how they may potentially impact
the change:\r\n\r\n| Risk | Probability | Severity | Mitigation/Notes
|\r\n\r\n|---------------------------|-------------|----------|-------------------------|\r\n|
Multiple Spaces&mdash;unexpected behavior in non-default Kibana
Space.\r\n| Low | High | Integration tests will verify that all features
are still\r\nsupported in non-default Kibana Space and when user
switches between\r\nspaces. |\r\n| Multiple nodes&mdash;Elasticsearch
polling might have race conditions\r\nwhen multiple Kibana nodes are
polling for the same tasks. | High | Low\r\n| Tasks are idempotent, so
executing them multiple times will not result\r\nin logical error, but
will degrade performance. To test for this case we\r\nadd plenty of unit
tests around this logic and document manual testing\r\nprocedure. |\r\n|
Code should gracefully handle cases when feature X or plugin Y
are\r\ndisabled. | Medium | High | Unit tests will verify that any
feature flag\r\nor plugin combination still results in our service
operational. |\r\n| [See more potential
risk\r\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
|\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for
breaking API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"dc577c0bf1e02116287e2c5c617223094d34af45"}}]}]
BACKPORT-->

Co-authored-by: Joe McElroy <joseph.mcelroy@elastic.co>

x-pack/plugins/search_playground/common/types.ts

@@ -19,6 +19,10 @@ interface ModelField {
   indices: string[];
 }
 
+interface ELSERQueryFields extends ModelField {
+  sparse_vector: boolean;
+}
+
 export interface ChatMessage {
   id: string;
   role: MessageRole;
@@ -33,7 +37,7 @@ interface SemanticField {
 }
 
 export interface QuerySourceFields {
-  elser_query_fields: ModelField[];
+  elser_query_fields: ELSERQueryFields[];
   dense_vector_query_fields: ModelField[];
   bm25_query_fields: string[];
   source_fields: string[];

x-pack/plugins/search_playground/public/hooks/use_source_indices_fields.test.tsx

@@ -37,6 +37,7 @@ describe.skip('useSourceIndicesFields Hook', () => {
             field: 'field1',
             model_id: 'model1',
             indices: ['newIndex'],
+            sparse_vector: true,
           },
         ],
         dense_vector_query_fields: [],

x-pack/plugins/search_playground/public/utils/create_query.test.ts

@@ -19,7 +19,9 @@ describe('create_query', () => {
 
       const fieldDescriptors: IndicesQuerySourceFields = {
         index1: {
-          elser_query_fields: [{ field: 'field1', model_id: 'model1', indices: ['index1'] }],
+          elser_query_fields: [
+            { field: 'field1', model_id: 'model1', indices: ['index1'], sparse_vector: true },
+          ],
           dense_vector_query_fields: [],
           bm25_query_fields: [],
           source_fields: [],
@@ -43,6 +45,40 @@ describe('create_query', () => {
       });
     });
 
+    it('should return a text_expansion single query', () => {
+      const fields = {
+        index1: ['field1'],
+      };
+
+      const fieldDescriptors: IndicesQuerySourceFields = {
+        index1: {
+          elser_query_fields: [
+            { field: 'field1', model_id: 'model1', indices: ['index1'], sparse_vector: false },
+          ],
+          dense_vector_query_fields: [],
+          bm25_query_fields: [],
+          source_fields: [],
+          skipped_fields: 0,
+          semantic_fields: [],
+        },
+      };
+
+      expect(createQuery(fields, sourceFields, fieldDescriptors)).toEqual({
+        retriever: {
+          standard: {
+            query: {
+              text_expansion: {
+                field1: {
+                  model_id: 'model1',
+                  model_text: '{query}',
+                },
+              },
+            },
+          },
+        },
+      });
+    });
+
     it('should return a knn query single', () => {
       const fields = {
         index1: ['field1'],
@@ -88,7 +124,12 @@ describe('create_query', () => {
       const fieldDescriptors: IndicesQuerySourceFields = {
         index1: {
           elser_query_fields: [
-            { field: 'field1', model_id: 'model1', indices: ['index1', 'index2'] },
+            {
+              field: 'field1',
+              model_id: 'model1',
+              indices: ['index1', 'index2'],
+              sparse_vector: true,
+            },
           ],
           dense_vector_query_fields: [],
           bm25_query_fields: [],
@@ -98,7 +139,12 @@ describe('create_query', () => {
         },
         index2: {
           elser_query_fields: [
-            { field: 'field1', model_id: 'model1', indices: ['index1', 'index2'] },
+            {
+              field: 'field1',
+              model_id: 'model1',
+              indices: ['index1', 'index2'],
+              sparse_vector: true,
+            },
           ],
           dense_vector_query_fields: [],
           bm25_query_fields: [],
@@ -131,7 +177,9 @@ describe('create_query', () => {
 
       const fieldDescriptors: IndicesQuerySourceFields = {
         index1: {
-          elser_query_fields: [{ field: 'field1', model_id: 'model1', indices: ['index1'] }],
+          elser_query_fields: [
+            { field: 'field1', model_id: 'model1', indices: ['index1'], sparse_vector: true },
+          ],
           dense_vector_query_fields: [],
           bm25_query_fields: [],
           source_fields: [],
@@ -139,7 +187,9 @@ describe('create_query', () => {
           semantic_fields: [],
         },
         index2: {
-          elser_query_fields: [{ field: 'field2', model_id: 'model1', indices: ['index2'] }],
+          elser_query_fields: [
+            { field: 'field2', model_id: 'model1', indices: ['index2'], sparse_vector: true },
+          ],
           dense_vector_query_fields: [],
           bm25_query_fields: [],
           source_fields: [],
@@ -189,7 +239,9 @@ describe('create_query', () => {
 
         const fieldDescriptors: IndicesQuerySourceFields = {
           index1: {
-            elser_query_fields: [{ field: 'field1', model_id: 'model1', indices: ['index1'] }],
+            elser_query_fields: [
+              { field: 'field1', model_id: 'model1', indices: ['index1'], sparse_vector: true },
+            ],
             dense_vector_query_fields: [],
             bm25_query_fields: ['content', 'title'],
             source_fields: [],
@@ -197,7 +249,9 @@ describe('create_query', () => {
             semantic_fields: [],
           },
           index2: {
-            elser_query_fields: [{ field: 'field2', model_id: 'model1', indices: ['index2'] }],
+            elser_query_fields: [
+              { field: 'field2', model_id: 'model1', indices: ['index2'], sparse_vector: true },
+            ],
             dense_vector_query_fields: [],
             bm25_query_fields: [],
             source_fields: [],
@@ -251,7 +305,9 @@ describe('create_query', () => {
 
         const fieldDescriptors: IndicesQuerySourceFields = {
           index1: {
-            elser_query_fields: [{ field: 'field1', model_id: 'model1', indices: ['index1'] }],
+            elser_query_fields: [
+              { field: 'field1', model_id: 'model1', indices: ['index1'], sparse_vector: true },
+            ],
             dense_vector_query_fields: [],
             bm25_query_fields: ['content', 'title'],
             source_fields: [],
@@ -259,7 +315,9 @@ describe('create_query', () => {
             semantic_fields: [],
           },
           index2: {
-            elser_query_fields: [{ field: 'field2', model_id: 'model1', indices: ['index2'] }],
+            elser_query_fields: [
+              { field: 'field2', model_id: 'model1', indices: ['index2'], sparse_vector: true },
+            ],
             dense_vector_query_fields: [],
             bm25_query_fields: [],
             source_fields: [],
@@ -326,7 +384,9 @@ describe('create_query', () => {
           semantic_fields: [],
         },
         index2: {
-          elser_query_fields: [{ field: 'field2', model_id: 'model1', indices: ['index2'] }],
+          elser_query_fields: [
+            { field: 'field2', model_id: 'model1', indices: ['index2'], sparse_vector: true },
+          ],
           dense_vector_query_fields: [],
           bm25_query_fields: [],
           source_fields: [],
@@ -663,7 +723,9 @@ describe('create_query', () => {
     it('should return default ELSER query fields', () => {
       const fieldDescriptors: IndicesQuerySourceFields = {
         index1: {
-          elser_query_fields: [{ field: 'field1', model_id: 'model1', indices: ['index1'] }],
+          elser_query_fields: [
+            { field: 'field1', model_id: 'model1', indices: ['index1'], sparse_vector: true },
+          ],
           dense_vector_query_fields: [
             { field: 'field1', model_id: 'dense_model', indices: ['index1'] },
           ],
@@ -680,7 +742,9 @@ describe('create_query', () => {
     it('should return default elser query fields for multiple indices', () => {
       const fieldDescriptors: IndicesQuerySourceFields = {
         index1: {
-          elser_query_fields: [{ field: 'field1', model_id: 'model1', indices: ['index1'] }],
+          elser_query_fields: [
+            { field: 'field1', model_id: 'model1', indices: ['index1'], sparse_vector: true },
+          ],
           dense_vector_query_fields: [
             {
               field: 'dv_field1',
@@ -695,7 +759,9 @@ describe('create_query', () => {
           semantic_fields: [],
         },
         index2: {
-          elser_query_fields: [{ field: 'vector', model_id: 'model1', indices: ['index2'] }],
+          elser_query_fields: [
+            { field: 'vector', model_id: 'model1', indices: ['index2'], sparse_vector: true },
+          ],
           dense_vector_query_fields: [
             {
               field: 'dv_field1',
@@ -720,7 +786,9 @@ describe('create_query', () => {
     it('should return elser query fields for default fields', () => {
       const fieldDescriptors: IndicesQuerySourceFields = {
         index1: {
-          elser_query_fields: [{ field: 'field1', model_id: 'model1', indices: ['index1'] }],
+          elser_query_fields: [
+            { field: 'field1', model_id: 'model1', indices: ['index1'], sparse_vector: true },
+          ],
           dense_vector_query_fields: [
             {
               field: 'dv_field1',
@@ -735,7 +803,9 @@ describe('create_query', () => {
           semantic_fields: [],
         },
         index2: {
-          elser_query_fields: [{ field: 'vector', model_id: 'model1', indices: ['index2'] }],
+          elser_query_fields: [
+            { field: 'vector', model_id: 'model1', indices: ['index2'], sparse_vector: true },
+          ],
           dense_vector_query_fields: [
             {
               field: 'dv_field1',

x-pack/plugins/search_playground/public/utils/create_query.ts

@@ -134,10 +134,11 @@ export function createQuery(
             (x) => x.field === field
           );
 
-          if (elserField) {
+          if (elserField && elserField.sparse_vector) {
             // when another index has the same field, we don't want to duplicate the match rule
             const hasExistingSparseMatch = acc.queryMatches.find(
               (x) =>
+                // when the field is a sparse_vector field
                 x?.sparse_vector?.field === field &&
                 x?.sparse_vector?.inference_id === elserField?.model_id
             );
@@ -154,6 +155,28 @@ export function createQuery(
               },
             };
           }
+
+          if (elserField && !elserField.sparse_vector) {
+            // when the field is a rank_features field
+            const hasExistingSparseMatch = acc.queryMatches.find(
+              (x) =>
+                x?.text_expansion?.[elserField.field] &&
+                x?.sparse_vector?.inference_id === elserField?.model_id
+            );
+
+            if (hasExistingSparseMatch) {
+              return null;
+            }
+
+            return {
+              text_expansion: {
+                [elserField.field]: {
+                  model_id: elserField.model_id,
+                  model_text: '{query}',
+                },
+              },
+            };
+          }
           return null;
         }) || [];
 

x-pack/plugins/search_playground/server/lib/fetch_query_source_fields.test.ts

@@ -73,6 +73,7 @@ describe('fetch_query_source_fields', () => {
               field: 'vector.tokens',
               model_id: '.elser_model_2',
               indices: ['workplace_index'],
+              sparse_vector: false,
             },
           ],
           skipped_fields: 8,
@@ -94,6 +95,7 @@ describe('fetch_query_source_fields', () => {
               field: 'content_vector.tokens',
               model_id: '.elser_model_2',
               indices: ['workplace_index2'],
+              sparse_vector: false,
             },
           ],
           source_fields: [
@@ -188,6 +190,7 @@ describe('fetch_query_source_fields', () => {
               field: 'ml.inference.body_content_expanded.predicted_value',
               indices: ['search-nethys'],
               model_id: '.elser_model_2_linux-x86_64',
+              sparse_vector: true,
             },
           ],
           source_fields: ['body_content', 'headings', 'title'],
@@ -292,6 +295,7 @@ describe('fetch_query_source_fields', () => {
               field: 'text_embedding',
               indices: ['index'],
               model_id: '.elser_model_2',
+              sparse_vector: true,
             },
           ],
           dense_vector_query_fields: [],
@@ -347,6 +351,7 @@ describe('fetch_query_source_fields', () => {
               field: 'text_embedding',
               indices: ['elser_index'],
               model_id: 'my-elser-model',
+              sparse_vector: true,
             },
           ],
           skipped_fields: 2,

x-pack/plugins/search_playground/server/lib/fetch_query_source_fields.ts

@@ -259,12 +259,9 @@ export const parseFieldsCapabilities = (
           } else {
             acc[index].skipped_fields++;
           }
-        } else if (
-          isFieldInIndex(field, 'rank_features', index) ||
-          isFieldInIndex(field, 'sparse_vector', index)
-        ) {
+        } else if (isFieldInIndex(field, 'sparse_vector', index)) {
           const modelId = getModelField(fieldKey, modelIdFields);
-          const fieldCapabilities = field.rank_features || field.sparse_vector;
+          const fieldCapabilities = field.sparse_vector;
 
           // Check if the sparse vector field has a model_id associated with it
           // skip this field if has no model associated with it
@@ -274,6 +271,27 @@ export const parseFieldsCapabilities = (
               field: fieldKey,
               model_id: modelId,
               indices: (fieldCapabilities.indices as string[]) || indicesPresentIn,
+              // we must use sparse_vector query
+              sparse_vector: true,
+            };
+            acc[index].elser_query_fields.push(elserModelField);
+          } else {
+            acc[index].skipped_fields++;
+          }
+        } else if (isFieldInIndex(field, 'rank_features', index)) {
+          const modelId = getModelField(fieldKey, modelIdFields);
+          const fieldCapabilities = field.rank_features;
+
+          // Check if the sparse vector field has a model_id associated with it
+          // skip this field if has no model associated with it
+          // and the vectors were embedded outside of stack
+          if (modelId && !nestedField) {
+            const elserModelField = {
+              field: fieldKey,
+              model_id: modelId,
+              indices: (fieldCapabilities.indices as string[]) || indicesPresentIn,
+              // we must use text_expansion query
+              sparse_vector: false,
             };
             acc[index].elser_query_fields.push(elserModelField);
           } else {