[Security Solution] Disallow merging critical rule field values upon rule upgrade when base version is missing (#213757)

## Summary

This PR makes changes to force target version for critical rule fields upon upgrade when rule's base version isn't found.

## Details

In case of missing base version rule upgrade diff algorithm isn't able to detect rule customizations. The only thing it's able to determine is different current and target field values. It tries to merge mergeable fields whenever possible. In fact mergeable fields are only scalar arrays. There are 4 fields processed that way

- `tags`
- `references`
- `threat_index`
- `new_terms`

However, `threat_index` and `new_terms` are critical rule fields impacting what alerts rule is gonna generate. Auto-merged values have much higher chance to be improper.

This PR **changes** the current behavior for `threat_index` and `new_terms`. It forces target field version when base rule version isn't found.

## How to test

To test this fix an updated prebuilt rule asset should be created. Follow the next steps for that

- Start fresh ES and Kibana (clear any ES data before)
- Open Rules Management page to install the latest prebuilt rules package (`8.17.6` currently)
- Install a threat match rule (e.g. `Threat Intel Hash Indicator Match`) and a new terms rule (e.g. `AWS EC2 Route Table Modified or Deleted`)
- Create new prebuilt rule assets. You may copy and paste suggested queries below. Make sure to run them under `system_indicies_superuser`/`changeme` user.

    <details>
    <summary>PUT a new Threat Match prebuilt rule asset</summary>
    
    ```
    PUT  .kibana_security_solution/_doc/security-rule:aab184d3-72b3-4639-b242-6597c99d8bca_10
    {
        "security-rule": {
        "author": [
            "Elastic"
        ],
        "description": "This rule is triggered when a hash indicator from the Threat Intel Filebeat module or integrations has a match against an event that contains file hashes, such as antivirus alerts, process creation, library load, and file operation events.",
        "from": "now-65m",
        "index": [
            "auditbeat-*",
            "endgame-*",
            "filebeat-*",
            "logs-*",
            "winlogbeat-*"
        ],
        "interval": "1h",
        "language": "kuery",
        "license": "Elastic License v2",
        "name": "Threat Intel Hash Indicator Match",
        "note": """## Triage and analysis
    
    ### Investigating Threat Intel Hash Indicator Match
    
    Threat Intel indicator match rules allow matching from a local observation, such as an endpoint event that records a file hash with an entry of a file hash stored within the Threat Intel integrations index.
    
    Matches are based on threat intelligence data that's been ingested during the last 30 days. Some integrations don't place expiration dates on their threat indicators, so we strongly recommend validating ingested threat indicators and reviewing match results. When reviewing match results, check associated activity to determine whether the event requires additional investigation.
    
    This rule is triggered when a hash indicator from the Threat Intel Filebeat module or an indicator ingested from a threat intelligence integration matches against an event that contains file hashes, such as antivirus alerts, file operation events, etc.
    
    > **Note**:
    > This investigation guide uses the [Osquery Markdown Plugin](https://www.elastic.co/guide/en/security/current/invest-guide-run-osquery.html) introduced in Elastic Stack version 8.5.0. Older Elastic Stack versions will display unrendered Markdown in this guide.
    
    #### Possible investigation steps
    
    - Gain context about the field that matched the local observation. This information can be found in the `threat.indicator.matched.field` field.
    - Investigate the hash , which can be found in the `threat.indicator.matched.atomic` field:
    - Search for the existence and reputation of the hash in resources like VirusTotal, Hybrid-Analysis, CISCO Talos, Any.run, etc.
    - Scope other potentially compromised hosts in your environment by mapping hosts with file operations involving the same hash.
    - Identify the process that created the file.
    - Investigate the process execution chain (parent process tree) for unknown processes. Examine their executable files for prevalence, whether they are located in expected locations, and if they are signed with valid digital signatures.
    - Enrich the information that you have right now by determining how the file was dropped, where it was downloaded from, etc. This can help you determine if the event is part of an ongoing campaign against the organization.
    - Retrieve the involved file and examine the host for derived artifacts that indicate suspicious activities:
    - Analyze the process executable using a private sandboxed analysis system.
    - Observe and collect information about the following activities in both the sandbox and the alert subject host:
    - Attempts to contact external domains and addresses.
    - Use the Elastic Defend network events to determine domains and addresses contacted by the subject process by filtering by the process' `process.entity_id`.
    - Examine the DNS cache for suspicious or anomalous entries.
    - !{osquery{"label":"Osquery - Retrieve DNS Cache","query":"SELECT * FROM dns_cache"}}
    - Use the Elastic Defend registry events to examine registry keys accessed, modified, or created by the related processes in the process tree.
    - Examine the host services for suspicious or anomalous entries.
    - !{osquery{"label":"Osquery - Retrieve All Services","query":"SELECT description, display_name, name, path, pid, service_type, start_type, status, user_account FROM services"}}
    - !{osquery{"label":"Osquery - Retrieve Services Running on User Accounts","query":"SELECT description, display_name, name, path, pid, service_type, start_type, status, user_account FROM services WHERE\nNOT (user_account LIKE '%LocalSystem' OR user_account LIKE '%LocalService' OR user_account LIKE '%NetworkService' OR\nuser_account == null)\n"}}
    - !{osquery{"label":"Osquery - Retrieve Service Unsigned Executables with Virustotal Link","query":"SELECT concat('https://www.virustotal.com/gui/file/', sha1) AS VtLink, name, description, start_type, status, pid,\nservices.path FROM services JOIN authenticode ON services.path = authenticode.path OR services.module_path =\nauthenticode.path JOIN hash ON services.path = hash.path WHERE authenticode.result != 'trusted'\n"}}
    - Using the data collected through the analysis, scope users targeted and other machines infected in the environment.
    
    ### False Positive Analysis
    
    - Adversaries often use legitimate tools as network administrators, such as `PsExec` or `AdFind`. These tools are often included in indicator lists, which creates the potential for false positives.
    
    ### Response and Remediation
    
    - Initiate the incident response process based on the outcome of the triage.
    - Isolate the involved host to prevent further post-compromise behavior.
    - If the triage identified malware, search the environment for additional compromised hosts.
    - Implement temporary network rules, procedures, and segmentation to contain the malware.
    - Stop suspicious processes.
    - Immediately block the identified indicators of compromise (IoCs).
    - Inspect the affected systems for additional malware backdoors like reverse shells, reverse proxies, or droppers that attackers could use to reinfect the system.
    - Remove and block malicious artifacts identified during triage.
    - Run a full antimalware scan. This may reveal additional artifacts left in the system, persistence mechanisms, and malware components.
    - Determine the initial vector abused by the attacker and take action to prevent reinfection through the same vector.
    - Using the incident response data, update logging and audit policies to improve the mean time to detect (MTTD) and the mean time to respond (MTTR).
    """,
        "query": """file.hash.*:* or process.hash.*:* or dll.hash.*:*
    """,
        "references": [
            "https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-threatintel.html",
            "https://www.elastic.co/guide/en/security/current/es-threat-intel-integrations.html",
            "https://www.elastic.co/security/tip"
        ],
        "required_fields": [
            {
            "ecs": false,
            "name": "dll.hash.*",
            "type": "unknown"
            },
            {
            "ecs": false,
            "name": "file.hash.*",
            "type": "unknown"
            },
            {
            "ecs": false,
            "name": "process.hash.*",
            "type": "unknown"
            }
        ],
        "risk_score": 99,
        "rule_id": "aab184d3-72b3-4639-b242-6597c99d8bca",
        "setup": """## Setup
    
    This rule needs threat intelligence indicators to work.
    Threat intelligence indicators can be collected using an [Elastic Agent integration](https://www.elastic.co/guide/en/security/current/es-threat-intel-integrations.html#agent-ti-integration),
    the [Threat Intel module](https://www.elastic.co/guide/en/security/current/es-threat-intel-integrations.html#ti-mod-integration),
    or a [custom integration](https://www.elastic.co/guide/en/security/current/es-threat-intel-integrations.html#custom-ti-integration).
    
    More information can be found [here](https://www.elastic.co/guide/en/security/current/es-threat-intel-integrations.html).
    """,
        "severity": "critical",
        "tags": [
            "OS: Windows",
            "Data Source: Elastic Endgame",
            "Rule Type: Threat Match",
            "Resources: Investigation Guide"
        ],
        "threat_filters": [
            {
            "$state": {
                "store": "appState"
            },
            "meta": {
                "disabled": false,
                "key": "event.category",
                "negate": false,
                "params": {
                "query": "threat"
                },
                "type": "phrase"
            },
            "query": {
                "match_phrase": {
                "event.category": "threat"
                }
            }
            },
            {
            "$state": {
                "store": "appState"
            },
            "meta": {
                "disabled": false,
                "key": "event.kind",
                "negate": false,
                "params": {
                "query": "enrichment"
                },
                "type": "phrase"
            },
            "query": {
                "match_phrase": {
                "event.kind": "enrichment"
                }
            }
            },
            {
            "$state": {
                "store": "appState"
            },
            "meta": {
                "disabled": false,
                "key": "event.type",
                "negate": false,
                "params": {
                "query": "indicator"
                },
                "type": "phrase"
            },
            "query": {
                "match_phrase": {
                "event.type": "indicator"
                }
            }
            }
        ],
        "threat_index": [
            "logs-ti_*"
        ],
        "threat_indicator_path": "threat.indicator",
        "threat_language": "kuery",
        "threat_mapping": [
            {
            "entries": [
                {
                "field": "file.hash.md5",
                "type": "mapping",
                "value": "threat.indicator.file.hash.md5"
                }
            ]
            },
            {
            "entries": [
                {
                "field": "file.hash.sha1",
                "type": "mapping",
                "value": "threat.indicator.file.hash.sha1"
                }
            ]
            },
            {
            "entries": [
                {
                "field": "file.hash.sha256",
                "type": "mapping",
                "value": "threat.indicator.file.hash.sha256"
                }
            ]
            },
            {
            "entries": [
                {
                "field": "dll.hash.md5",
                "type": "mapping",
                "value": "threat.indicator.file.hash.md5"
                }
            ]
            },
            {
            "entries": [
                {
                "field": "dll.hash.sha1",
                "type": "mapping",
                "value": "threat.indicator.file.hash.sha1"
                }
            ]
            },
            {
            "entries": [
                {
                "field": "dll.hash.sha256",
                "type": "mapping",
                "value": "threat.indicator.file.hash.sha256"
                }
            ]
            },
            {
            "entries": [
                {
                "field": "process.hash.md5",
                "type": "mapping",
                "value": "threat.indicator.file.hash.md5"
                }
            ]
            },
            {
            "entries": [
                {
                "field": "process.hash.sha1",
                "type": "mapping",
                "value": "threat.indicator.file.hash.sha1"
                }
            ]
            },
            {
            "entries": [
                {
                "field": "process.hash.sha256",
                "type": "mapping",
                "value": "threat.indicator.file.hash.sha256"
                }
            ]
            }
        ],
        "threat_query": "@timestamp >= \"now-30d/d\" and event.module:(threatintel or ti_*) and (threat.indicator.file.hash.*:* or threat.indicator.file.pe.imphash:*) and not labels.is_ioc_transform_source:\"true\"",
        "timeline_id": "495ad7a7-316e-4544-8a0f-9c098daee76e",
        "timeline_title": "Generic Threat Match Timeline",
        "timestamp_override": "event.ingested",
        "type": "threat_match",
        "version": 10
        },
        "type": "security-rule",
        "references": [],
        "managed": true,
        "coreMigrationVersion": "8.8.0",
        "updated_at": "2025-03-11T07:21:44.100Z",
        "created_at": "2025-03-11T07:21:44.100Z"
    }
    ```
    </details>

     <details>
    <summary>PUT a new New Terms prebuilt rule asset</summary>
    
    ```
   PUT  .kibana_security_solution/_doc/security-rule:e7cd5982-17c8-4959-874c-633acde7d426_209
    {
        "security-rule": {
        "author": [
            "Elastic",
            "Austin Songer"
        ],
        "description": "Identifies AWS CloudTrail events where an EC2 route table or association has been modified or deleted. Route table or association modifications can be used by attackers to disrupt network traffic, reroute communications, or maintain persistence in a compromised environment. This is a [New Terms](https://www.elastic.co/guide/en/security/current/rules-ui-create.html#create-new-terms-rule) rule that detects the first instance of this behavior by the `aws.cloudtrail.user_identity.arn` field in the last 10 days.",
        "false_positives": [
            "Route Table could be modified or deleted by a system administrator. Verify whether the user identity, user agent, and/or hostname should be making changes in your environment. Route Table being modified from unfamiliar users should be investigated. If known behavior is causing false positives, it can be exempted from the rule. Also automated processes that use Terraform may lead to false positives."
        ],
        "from": "now-9m",
        "history_window_start": "now-10d",
        "index": [
            "filebeat-*",
            "logs-aws.cloudtrail-*"
        ],
        "language": "kuery",
        "license": "Elastic License v2",
        "name": "AWS EC2 Route Table Modified or Deleted",
        "new_terms_fields": [
            "test"
        ],
        "note": """## Triage and Analysis
    
    ### Investigating AWS EC2 Route Table Modified or Deleted
    
    This rule detects modifications or deletions of AWS route tables using actions such as `ReplaceRoute`, `ReplaceRouteTableAssociation`, `DeleteRouteTable`, `DeleteRoute`, or `DisassociateRouteTable`. These actions may indicate legitimate administrative activity, but they can also be abused by attackers to disrupt network traffic, reroute communications, or maintain persistence in a compromised environment.
    
    #### Possible Investigation Steps
    
    - **Review Request Parameters:**
    - Check the `aws.cloudtrail.flattened.request_parameters` field. The sub-fields may vary depending on the `event.action` (e.g., `routeTableId` for `DeleteRouteTable`, `destinationCidrBlock` for `ReplaceRoute`).
    - Validate the affected route table, routes, or associations based on the API call:
    - For `ReplaceRoute`: Look for changes in specific routes using `destinationCidrBlock`.
    - For `ReplaceRouteTableAssociation`: Review the new association details (e.g., subnet ID).
    - For `DeleteRouteTable`: Confirm the `routeTableId` of the deleted table.
    - For `DisassociateRouteTable`: Verify the disassociated resources.
    
    - **Review User Context**:
    - **User Identity**: Inspect the `aws.cloudtrail.user_identity.arn` field to determine the user or role initiating the action. Investigate whether this user is authorized to perform these operations.
    - **Access Key ID**: Check the `aws.cloudtrail.user_identity.access_key_id` field to identify if the access key used was expected or potentially compromised.
    - **Access Patterns**: Validate whether the user or role has a history of performing route table modifications and whether this aligns with their expected responsibilities.
    
    - **Analyze Request Details**:
    - **Action Type**: Verify the specific API call in the `event.action` field (e.g., `ReplaceRoute`, `DeleteRouteTable`) to understand the nature of the modification.
    - **Source IP and Geolocation**: Examine the `source.address` and `source.geo` fields to confirm whether the request originated from a trusted location. Suspicious geolocations or IPs may indicate adversarial activity.
    - **User Agent**: Review the `user_agent.original` field to determine the tool used for the request (e.g., AWS CLI, Terraform). Unusual or custom user agents may indicate malicious intent.
    
    - **Correlate with Other Activity**:
    - **Concurrent API Calls**: Look for related API calls (e.g., `CreateRoute`, `AuthorizeSecurityGroupIngress`, or `ModifyInstanceAttribute`) from the same user or IP to detect broader attack patterns.
    - **IAM Changes**: Investigate whether any IAM policy updates or privilege escalation attempts preceded this activity.
    - **Unusual Volume of Changes**: Check if the user has performed multiple route table modifications or deletions in a short timeframe.
    
    - **Validate the Intent**:
    - **Planned Changes**: Confirm with administrators whether the route table changes were part of a planned update or maintenance activity.
    - **Permissions and Justification**: Ensure that the user or role has the least privilege necessary for these actions and that there is a valid reason for modifying the route table.
    
    ### False Positive Analysis
    
    - **Routine Administration**: Route table modifications are often part of routine administrative tasks, such as creating new routes, updating associations, or removing unused resources.
    - **Automation Tools**: Automated workflows, such as those executed by Terraform or CloudFormation, may trigger these events. Verify whether the `user_agent.original` field or source IP matches known automation tools.
    - **Maintenance or Scaling**: Confirm whether these actions align with maintenance activities or scaling events (e.g., adding or removing subnets).
    
    ### Response and Remediation
    
    - **Revoke Unauthorized Permissions**: If unauthorized, remove permissions for `ec2:ReplaceRoute`, `ec2:DeleteRouteTable`, or other related actions from the user or role.
    - **Restore the Route Table**:
    - If critical networking was impacted, restore the route table or reapply previous configurations from backups or Terraform state files.
    - Verify connectivity to affected subnets or instances to ensure no disruptions to services.
    - **Audit IAM Policies**:
    - Limit route table modification permissions to specific trusted users, roles, or automation accounts.
    - Implement conditions in IAM policies, such as source IP restrictions, to reduce the risk of unauthorized access.
    - **Monitor and Alert**:
    - Set up additional alerts for unexpected route table modifications or deletions.
    - Use VPC flow logs and CloudTrail to monitor for related suspicious activity.
    - **Secure Automation**: Ensure automation tools, such as Terraform or CloudFormation, are configured securely and that their credentials are stored in secure locations like AWS Secrets Manager.
    """,
        "query": """event.dataset: "aws.cloudtrail"
    and event.provider: "ec2.amazonaws.com"
    and event.action:(
    "ReplaceRoute" or
    "ReplaceRouteTableAssociation" or
    "DeleteRouteTable" or
    "DeleteRoute" or
    "DisassociateRouteTable"
    )
    and event.outcome: "success"
    and not source.address: (
    "cloudformation.amazonaws.com" or
    "servicecatalog.amazonaws.com" or
    "fsx.amazonaws.com"
    )
    """,
        "references": [
            "https://github.com/easttimor/aws-incident-response#network-routing",
            "https://docs.datadoghq.com/security_platform/default_rules/aws-ec2-route-table-modified/",
            "https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_ReplaceRoute.html",
            "https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_ReplaceRouteTableAssociation",
            "https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DeleteRouteTable.html",
            "https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DeleteRoute.html",
            "https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DisassociateRouteTable.html"
        ],
        "related_integrations": [
            {
            "integration": "cloudtrail",
            "package": "aws",
            "version": "^2.0.0"
            }
        ],
        "required_fields": [
            {
            "ecs": true,
            "name": "event.action",
            "type": "keyword"
            },
            {
            "ecs": true,
            "name": "event.dataset",
            "type": "keyword"
            },
            {
            "ecs": true,
            "name": "event.outcome",
            "type": "keyword"
            },
            {
            "ecs": true,
            "name": "event.provider",
            "type": "keyword"
            },
            {
            "ecs": true,
            "name": "source.address",
            "type": "keyword"
            }
        ],
        "risk_score": 21,
        "rule_id": "e7cd5982-17c8-4959-874c-633acde7d426",
        "severity": "low",
        "tags": [
            "Domain: Cloud",
            "Data Source: AWS",
            "Data Source: Amazon Web Services",
            "Data Source: AWS EC2",
            "Use Case: Network Security Monitoring",
            "Resources: Investigation Guide",
            "Tactic: Persistence"
        ],
        "threat": [
            {
            "framework": "MITRE ATT&CK",
            "tactic": {
                "id": "TA0003",
                "name": "Persistence",
                "reference": "https://attack.mitre.org/tactics/TA0003/"
            },
            "technique": []
            }
        ],
        "timestamp_override": "event.ingested",
        "type": "new_terms",
        "version": 209
        },
        "type": "security-rule",
        "references": [],
        "managed": true,
        "coreMigrationVersion": "8.8.0",
        "updated_at": "2025-03-11T07:21:44.889Z",
        "created_at": "2025-03-11T07:21:44.889Z"
    }
    ```
    </details>

- Remove the base versions

  <details>
    <summary>Remove Threat Match rule's base version</summary>

    ```
    DELETE .kibana_security_solution/_doc/security-rule:aab184d3-72b3-4639-b242-6597c99d8bca_9
    ```
  </details>

    <details>
    <summary>Remove New Terms rule's base version</summary>

    ```
    DELETE .kibana_security_solution/_doc/security-rule:e7cd5982-17c8-4959-874c-633acde7d426_208
    ```
    </details>
 
- Customize rule data source
- Open Prebuilt Rule Upgrade flyout and notice there are fields with solvable conflict
- Bulk update rules
- Notice Threat Match Index and New Terms Fields got target values. Data source value was set to target as well. Rules aren't marked as customized (there is not a "Modified" badge).

x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/diff/calculation/algorithms/index.ts

@@ -7,7 +7,7 @@
 
 export { numberDiffAlgorithm } from './number_diff_algorithm';
 export { singleLineStringDiffAlgorithm } from './single_line_string_diff_algorithm';
-export { scalarArrayDiffAlgorithm } from './scalar_array_diff_algorithm';
+export { createScalarArrayDiffAlgorithm } from './scalar_array_diff_algorithm';
 export { simpleDiffAlgorithm } from './simple_diff_algorithm';
 export { multiLineStringDiffAlgorithm } from './multi_line_string_diff_algorithm';
 export { dataSourceDiffAlgorithm } from './data_source_diff_algorithm';

x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/diff/calculation/algorithms/scalar_array_diff_algorithm.test.ts

@@ -12,324 +12,45 @@ import {
   MissingVersion,
   ThreeWayDiffConflict,
 } from '../../../../../../../../common/api/detection_engine';
-import { scalarArrayDiffAlgorithm } from './scalar_array_diff_algorithm';
+import {
+  ScalarArrayDiffMissingBaseVersionStrategy,
+  createScalarArrayDiffAlgorithm,
+} from './scalar_array_diff_algorithm';
 
 describe('scalarArrayDiffAlgorithm', () => {
-  it('returns current_version as merged output if there is no update - scenario AAA', () => {
-    const mockVersions: ThreeVersionsOf<string[]> = {
-      base_version: ['one', 'two', 'three'],
-      current_version: ['one', 'two', 'three'],
-      target_version: ['one', 'two', 'three'],
-    };
+  describe.each([
+    [ScalarArrayDiffMissingBaseVersionStrategy.Merge],
+    [ScalarArrayDiffMissingBaseVersionStrategy.UseTarget],
+  ])('with missingBaseCanUpdateMergeStrategy = %s', (mergeStrategy) => {
+    const scalarArrayDiffAlgorithm = createScalarArrayDiffAlgorithm({
+      missingBaseVersionStrategy: mergeStrategy,
+    });
 
-    const result = scalarArrayDiffAlgorithm(mockVersions, false);
-
-    expect(result).toEqual(
-      expect.objectContaining({
-        merged_version: mockVersions.current_version,
-        diff_outcome: ThreeWayDiffOutcome.StockValueNoUpdate,
-        merge_outcome: ThreeWayMergeOutcome.Current,
-        conflict: ThreeWayDiffConflict.NONE,
-      })
-    );
-  });
-
-  it('returns current_version as merged output if current_version is different and there is no update - scenario ABA', () => {
-    const mockVersions: ThreeVersionsOf<string[]> = {
-      base_version: ['one', 'two', 'three'],
-      current_version: ['one', 'three', 'four'],
-      target_version: ['one', 'two', 'three'],
-    };
-
-    const result = scalarArrayDiffAlgorithm(mockVersions, false);
-
-    expect(result).toEqual(
-      expect.objectContaining({
-        merged_version: mockVersions.current_version,
-        diff_outcome: ThreeWayDiffOutcome.CustomizedValueNoUpdate,
-        merge_outcome: ThreeWayMergeOutcome.Current,
-        conflict: ThreeWayDiffConflict.NONE,
-      })
-    );
-  });
-
-  it('returns target_version as merged output if current_version is the same and there is an update - scenario AAB', () => {
-    const mockVersions: ThreeVersionsOf<string[]> = {
-      base_version: ['one', 'two', 'three'],
-      current_version: ['one', 'two', 'three'],
-      target_version: ['one', 'four', 'three'],
-    };
-
-    const result = scalarArrayDiffAlgorithm(mockVersions, false);
-
-    expect(result).toEqual(
-      expect.objectContaining({
-        merged_version: mockVersions.target_version,
-        diff_outcome: ThreeWayDiffOutcome.StockValueCanUpdate,
-        merge_outcome: ThreeWayMergeOutcome.Target,
-        conflict: ThreeWayDiffConflict.NONE,
-      })
-    );
-  });
-
-  it('returns current_version as merged output if current version is different but it matches the update - scenario ABB', () => {
-    const mockVersions: ThreeVersionsOf<string[]> = {
-      base_version: ['one', 'two', 'three'],
-      current_version: ['one', 'three', 'four'],
-      target_version: ['one', 'four', 'three'],
-    };
-
-    const result = scalarArrayDiffAlgorithm(mockVersions, false);
-
-    expect(result).toEqual(
-      expect.objectContaining({
-        merged_version: mockVersions.current_version,
-        diff_outcome: ThreeWayDiffOutcome.CustomizedValueSameUpdate,
-        merge_outcome: ThreeWayMergeOutcome.Current,
-        conflict: ThreeWayDiffConflict.NONE,
-      })
-    );
-  });
-
-  it('returns custom merged version as merged output if all three versions are different - scenario ABC', () => {
-    const mockVersions: ThreeVersionsOf<string[]> = {
-      base_version: ['one', 'two', 'three'],
-      current_version: ['two', 'three', 'four', 'five'],
-      target_version: ['one', 'three', 'four', 'six'],
-    };
-    const expectedMergedVersion = ['three', 'four', 'five', 'six'];
-
-    const result = scalarArrayDiffAlgorithm(mockVersions, false);
-
-    expect(result).toEqual(
-      expect.objectContaining({
-        merged_version: expectedMergedVersion,
-        diff_outcome: ThreeWayDiffOutcome.CustomizedValueCanUpdate,
-        merge_outcome: ThreeWayMergeOutcome.Merged,
-        conflict: ThreeWayDiffConflict.SOLVABLE,
-      })
-    );
-  });
-
-  describe('if base_version is missing', () => {
-    describe('returns target_version as merged output if current_version and target_version are the same - scenario -AA', () => {
-      it('returns NONE conflict if rule is not customized', () => {
+    describe('base cases', () => {
+      it('returns current_version as merged output if there is no update - scenario AAA', () => {
         const mockVersions: ThreeVersionsOf<string[]> = {
-          base_version: MissingVersion,
+          base_version: ['one', 'two', 'three'],
           current_version: ['one', 'two', 'three'],
           target_version: ['one', 'two', 'three'],
         };
 
         const result = scalarArrayDiffAlgorithm(mockVersions, false);
 
-        expect(result).toEqual(
-          expect.objectContaining({
-            has_base_version: false,
-            base_version: undefined,
-            merged_version: mockVersions.target_version,
-            diff_outcome: ThreeWayDiffOutcome.MissingBaseNoUpdate,
-            merge_outcome: ThreeWayMergeOutcome.Target,
-            conflict: ThreeWayDiffConflict.NONE,
-          })
-        );
-      });
-
-      it('returns NONE conflict if rule is customized', () => {
-        const mockVersions: ThreeVersionsOf<string[]> = {
-          base_version: MissingVersion,
-          current_version: ['one', 'two', 'three'],
-          target_version: ['one', 'two', 'three'],
-        };
-
-        const result = scalarArrayDiffAlgorithm(mockVersions, true);
-
-        expect(result).toEqual(
-          expect.objectContaining({
-            has_base_version: false,
-            base_version: undefined,
-            merged_version: mockVersions.target_version,
-            diff_outcome: ThreeWayDiffOutcome.MissingBaseNoUpdate,
-            merge_outcome: ThreeWayMergeOutcome.Target,
-            conflict: ThreeWayDiffConflict.NONE,
-          })
-        );
-      });
-    });
-
-    describe('if current_version and target_version are different - scenario -AB', () => {
-      it('returns target_version as merged output and NONE conflict if rule is not customized', () => {
-        const mockVersions: ThreeVersionsOf<string[]> = {
-          base_version: MissingVersion,
-          current_version: ['one', 'two', 'three'],
-          target_version: ['one', 'four', 'three'],
-        };
-
-        const result = scalarArrayDiffAlgorithm(mockVersions, false);
-
-        expect(result).toEqual(
-          expect.objectContaining({
-            has_base_version: false,
-            base_version: undefined,
-            merged_version: mockVersions.target_version,
-            diff_outcome: ThreeWayDiffOutcome.MissingBaseCanUpdate,
-            merge_outcome: ThreeWayMergeOutcome.Target,
-            conflict: ThreeWayDiffConflict.NONE,
-          })
-        );
-      });
-
-      it('returns merged version of current and target as merged output if rule is customized', () => {
-        const mockVersions: ThreeVersionsOf<string[]> = {
-          base_version: MissingVersion,
-          current_version: ['one', 'two', 'three'],
-          target_version: ['one', 'four', 'three'],
-        };
-
-        const expectedMergedVersion = ['one', 'two', 'three', 'four'];
-
-        const result = scalarArrayDiffAlgorithm(mockVersions, true);
-
-        expect(result).toEqual(
-          expect.objectContaining({
-            has_base_version: false,
-            base_version: undefined,
-            merged_version: expectedMergedVersion,
-            diff_outcome: ThreeWayDiffOutcome.MissingBaseCanUpdate,
-            merge_outcome: ThreeWayMergeOutcome.Merged,
-            conflict: ThreeWayDiffConflict.SOLVABLE,
-          })
-        );
-      });
-    });
-  });
-
-  describe('edge cases', () => {
-    it('compares arrays agnostic of order', () => {
-      const mockVersions: ThreeVersionsOf<string[]> = {
-        base_version: ['one', 'two', 'three'],
-        current_version: ['one', 'three', 'two'],
-        target_version: ['three', 'one', 'two'],
-      };
-
-      const result = scalarArrayDiffAlgorithm(mockVersions, false);
-
-      expect(result).toEqual(
-        expect.objectContaining({
-          merged_version: mockVersions.current_version,
-          diff_outcome: ThreeWayDiffOutcome.StockValueNoUpdate,
-          merge_outcome: ThreeWayMergeOutcome.Current,
-          conflict: ThreeWayDiffConflict.NONE,
-        })
-      );
-    });
-
-    describe('compares arrays deduplicated', () => {
-      it('when values duplicated in base version', () => {
-        const mockVersions: ThreeVersionsOf<string[]> = {
-          base_version: ['one', 'two', 'two'],
-          current_version: ['one', 'two'],
-          target_version: ['one', 'two'],
-        };
-        const expectedMergedVersion = ['one', 'two'];
-
-        const result = scalarArrayDiffAlgorithm(mockVersions, false);
-
-        expect(result).toEqual(
-          expect.objectContaining({
-            merged_version: expectedMergedVersion,
-            diff_outcome: ThreeWayDiffOutcome.StockValueNoUpdate,
-            merge_outcome: ThreeWayMergeOutcome.Current,
-            conflict: ThreeWayDiffConflict.NONE,
-          })
-        );
-      });
-
-      it('when values are duplicated in current version', () => {
-        const mockVersions: ThreeVersionsOf<string[]> = {
-          base_version: ['one', 'two'],
-          current_version: ['one', 'two', 'two'],
-          target_version: ['one', 'two'],
-        };
-        const expectedMergedVersion = ['one', 'two'];
-
-        const result = scalarArrayDiffAlgorithm(mockVersions, false);
-
-        expect(result).toEqual(
-          expect.objectContaining({
-            merged_version: expectedMergedVersion,
-            diff_outcome: ThreeWayDiffOutcome.StockValueNoUpdate,
-            merge_outcome: ThreeWayMergeOutcome.Current,
-            conflict: ThreeWayDiffConflict.NONE,
-          })
-        );
-      });
-
-      it('when values are duplicated in target version', () => {
-        const mockVersions: ThreeVersionsOf<string[]> = {
-          base_version: ['one', 'two'],
-          current_version: ['one', 'two'],
-          target_version: ['one', 'two', 'two'],
-        };
-        const expectedMergedVersion = ['one', 'two'];
-
-        const result = scalarArrayDiffAlgorithm(mockVersions, false);
-
-        expect(result).toEqual(
-          expect.objectContaining({
-            merged_version: expectedMergedVersion,
-            diff_outcome: ThreeWayDiffOutcome.StockValueNoUpdate,
-            merge_outcome: ThreeWayMergeOutcome.Current,
-            conflict: ThreeWayDiffConflict.NONE,
-          })
-        );
-      });
-
-      it('when values are duplicated in all versions', () => {
-        const mockVersions: ThreeVersionsOf<string[]> = {
-          base_version: ['one', 'two', 'two'],
-          current_version: ['two', 'two', 'three'],
-          target_version: ['one', 'one', 'three', 'three'],
-        };
-        const expectedMergedVersion = ['three'];
-
-        const result = scalarArrayDiffAlgorithm(mockVersions, false);
-
-        expect(result).toEqual(
-          expect.objectContaining({
-            merged_version: expectedMergedVersion,
-            diff_outcome: ThreeWayDiffOutcome.CustomizedValueCanUpdate,
-            merge_outcome: ThreeWayMergeOutcome.Merged,
-            conflict: ThreeWayDiffConflict.SOLVABLE,
-          })
-        );
-      });
-    });
-
-    describe('compares empty arrays', () => {
-      it('when base version is empty', () => {
-        const mockVersions: ThreeVersionsOf<string[]> = {
-          base_version: [],
-          current_version: ['one', 'two'],
-          target_version: ['one', 'two'],
-        };
-
-        const result = scalarArrayDiffAlgorithm(mockVersions, false);
-
         expect(result).toEqual(
           expect.objectContaining({
             merged_version: mockVersions.current_version,
-            diff_outcome: ThreeWayDiffOutcome.CustomizedValueSameUpdate,
+            diff_outcome: ThreeWayDiffOutcome.StockValueNoUpdate,
             merge_outcome: ThreeWayMergeOutcome.Current,
             conflict: ThreeWayDiffConflict.NONE,
           })
         );
       });
 
-      it('when current version is empty', () => {
+      it('returns current_version as merged output if current_version is different and there is no update - scenario ABA', () => {
         const mockVersions: ThreeVersionsOf<string[]> = {
-          base_version: ['one', 'two'],
-          current_version: [],
-          target_version: ['one', 'two'],
+          base_version: ['one', 'two', 'three'],
+          current_version: ['one', 'three', 'four'],
+          target_version: ['one', 'two', 'three'],
         };
 
         const result = scalarArrayDiffAlgorithm(mockVersions, false);
@@ -344,11 +65,11 @@ describe('scalarArrayDiffAlgorithm', () => {
         );
       });
 
-      it('when target version is empty', () => {
+      it('returns target_version as merged output if current_version is the same and there is an update - scenario AAB', () => {
         const mockVersions: ThreeVersionsOf<string[]> = {
-          base_version: ['one', 'two'],
-          current_version: ['one', 'two'],
-          target_version: [],
+          base_version: ['one', 'two', 'three'],
+          current_version: ['one', 'two', 'three'],
+          target_version: ['one', 'four', 'three'],
         };
 
         const result = scalarArrayDiffAlgorithm(mockVersions, false);
@@ -363,24 +84,348 @@ describe('scalarArrayDiffAlgorithm', () => {
         );
       });
 
-      it('when all versions are empty', () => {
+      it('returns current_version as merged output if current version is different but it matches the update - scenario ABB', () => {
         const mockVersions: ThreeVersionsOf<string[]> = {
-          base_version: [],
-          current_version: [],
-          target_version: [],
+          base_version: ['one', 'two', 'three'],
+          current_version: ['one', 'three', 'four'],
+          target_version: ['one', 'four', 'three'],
         };
 
         const result = scalarArrayDiffAlgorithm(mockVersions, false);
 
         expect(result).toEqual(
           expect.objectContaining({
-            merged_version: [],
+            merged_version: mockVersions.current_version,
+            diff_outcome: ThreeWayDiffOutcome.CustomizedValueSameUpdate,
+            merge_outcome: ThreeWayMergeOutcome.Current,
+            conflict: ThreeWayDiffConflict.NONE,
+          })
+        );
+      });
+
+      it('returns custom merged version as merged output if all three versions are different - scenario ABC', () => {
+        const mockVersions: ThreeVersionsOf<string[]> = {
+          base_version: ['one', 'two', 'three'],
+          current_version: ['two', 'three', 'four', 'five'],
+          target_version: ['one', 'three', 'four', 'six'],
+        };
+        const expectedMergedVersion = ['three', 'four', 'five', 'six'];
+
+        const result = scalarArrayDiffAlgorithm(mockVersions, false);
+
+        expect(result).toEqual(
+          expect.objectContaining({
+            merged_version: expectedMergedVersion,
+            diff_outcome: ThreeWayDiffOutcome.CustomizedValueCanUpdate,
+            merge_outcome: ThreeWayMergeOutcome.Merged,
+            conflict: ThreeWayDiffConflict.SOLVABLE,
+          })
+        );
+      });
+    });
+
+    describe('edge cases', () => {
+      it('compares arrays agnostic of order', () => {
+        const mockVersions: ThreeVersionsOf<string[]> = {
+          base_version: ['one', 'two', 'three'],
+          current_version: ['one', 'three', 'two'],
+          target_version: ['three', 'one', 'two'],
+        };
+
+        const result = scalarArrayDiffAlgorithm(mockVersions, false);
+
+        expect(result).toEqual(
+          expect.objectContaining({
+            merged_version: mockVersions.current_version,
             diff_outcome: ThreeWayDiffOutcome.StockValueNoUpdate,
             merge_outcome: ThreeWayMergeOutcome.Current,
             conflict: ThreeWayDiffConflict.NONE,
           })
         );
       });
+
+      describe('compares arrays deduplicated', () => {
+        it('when values duplicated in base version', () => {
+          const mockVersions: ThreeVersionsOf<string[]> = {
+            base_version: ['one', 'two', 'two'],
+            current_version: ['one', 'two'],
+            target_version: ['one', 'two'],
+          };
+          const expectedMergedVersion = ['one', 'two'];
+
+          const result = scalarArrayDiffAlgorithm(mockVersions, false);
+
+          expect(result).toEqual(
+            expect.objectContaining({
+              merged_version: expectedMergedVersion,
+              diff_outcome: ThreeWayDiffOutcome.StockValueNoUpdate,
+              merge_outcome: ThreeWayMergeOutcome.Current,
+              conflict: ThreeWayDiffConflict.NONE,
+            })
+          );
+        });
+
+        it('when values are duplicated in current version', () => {
+          const mockVersions: ThreeVersionsOf<string[]> = {
+            base_version: ['one', 'two'],
+            current_version: ['one', 'two', 'two'],
+            target_version: ['one', 'two'],
+          };
+          const expectedMergedVersion = ['one', 'two'];
+
+          const result = scalarArrayDiffAlgorithm(mockVersions, false);
+
+          expect(result).toEqual(
+            expect.objectContaining({
+              merged_version: expectedMergedVersion,
+              diff_outcome: ThreeWayDiffOutcome.StockValueNoUpdate,
+              merge_outcome: ThreeWayMergeOutcome.Current,
+              conflict: ThreeWayDiffConflict.NONE,
+            })
+          );
+        });
+
+        it('when values are duplicated in target version', () => {
+          const mockVersions: ThreeVersionsOf<string[]> = {
+            base_version: ['one', 'two'],
+            current_version: ['one', 'two'],
+            target_version: ['one', 'two', 'two'],
+          };
+          const expectedMergedVersion = ['one', 'two'];
+
+          const result = scalarArrayDiffAlgorithm(mockVersions, false);
+
+          expect(result).toEqual(
+            expect.objectContaining({
+              merged_version: expectedMergedVersion,
+              diff_outcome: ThreeWayDiffOutcome.StockValueNoUpdate,
+              merge_outcome: ThreeWayMergeOutcome.Current,
+              conflict: ThreeWayDiffConflict.NONE,
+            })
+          );
+        });
+
+        it('when values are duplicated in all versions', () => {
+          const mockVersions: ThreeVersionsOf<string[]> = {
+            base_version: ['one', 'two', 'two'],
+            current_version: ['two', 'two', 'three'],
+            target_version: ['one', 'one', 'three', 'three'],
+          };
+          const expectedMergedVersion = ['three'];
+
+          const result = scalarArrayDiffAlgorithm(mockVersions, false);
+
+          expect(result).toEqual(
+            expect.objectContaining({
+              merged_version: expectedMergedVersion,
+              diff_outcome: ThreeWayDiffOutcome.CustomizedValueCanUpdate,
+              merge_outcome: ThreeWayMergeOutcome.Merged,
+              conflict: ThreeWayDiffConflict.SOLVABLE,
+            })
+          );
+        });
+      });
+
+      describe('compares empty arrays', () => {
+        it('when base version is empty', () => {
+          const mockVersions: ThreeVersionsOf<string[]> = {
+            base_version: [],
+            current_version: ['one', 'two'],
+            target_version: ['one', 'two'],
+          };
+
+          const result = scalarArrayDiffAlgorithm(mockVersions, false);
+
+          expect(result).toEqual(
+            expect.objectContaining({
+              merged_version: mockVersions.current_version,
+              diff_outcome: ThreeWayDiffOutcome.CustomizedValueSameUpdate,
+              merge_outcome: ThreeWayMergeOutcome.Current,
+              conflict: ThreeWayDiffConflict.NONE,
+            })
+          );
+        });
+
+        it('when current version is empty', () => {
+          const mockVersions: ThreeVersionsOf<string[]> = {
+            base_version: ['one', 'two'],
+            current_version: [],
+            target_version: ['one', 'two'],
+          };
+
+          const result = scalarArrayDiffAlgorithm(mockVersions, false);
+
+          expect(result).toEqual(
+            expect.objectContaining({
+              merged_version: mockVersions.current_version,
+              diff_outcome: ThreeWayDiffOutcome.CustomizedValueNoUpdate,
+              merge_outcome: ThreeWayMergeOutcome.Current,
+              conflict: ThreeWayDiffConflict.NONE,
+            })
+          );
+        });
+
+        it('when target version is empty', () => {
+          const mockVersions: ThreeVersionsOf<string[]> = {
+            base_version: ['one', 'two'],
+            current_version: ['one', 'two'],
+            target_version: [],
+          };
+
+          const result = scalarArrayDiffAlgorithm(mockVersions, false);
+
+          expect(result).toEqual(
+            expect.objectContaining({
+              merged_version: mockVersions.target_version,
+              diff_outcome: ThreeWayDiffOutcome.StockValueCanUpdate,
+              merge_outcome: ThreeWayMergeOutcome.Target,
+              conflict: ThreeWayDiffConflict.NONE,
+            })
+          );
+        });
+
+        it('when all versions are empty', () => {
+          const mockVersions: ThreeVersionsOf<string[]> = {
+            base_version: [],
+            current_version: [],
+            target_version: [],
+          };
+
+          const result = scalarArrayDiffAlgorithm(mockVersions, false);
+
+          expect(result).toEqual(
+            expect.objectContaining({
+              merged_version: [],
+              diff_outcome: ThreeWayDiffOutcome.StockValueNoUpdate,
+              merge_outcome: ThreeWayMergeOutcome.Current,
+              conflict: ThreeWayDiffConflict.NONE,
+            })
+          );
+        });
+      });
+    });
+
+    describe('and base_version is missing', () => {
+      describe('returns target_version as merged output if current_version and target_version are the same - scenario -AA', () => {
+        it('returns NONE conflict if rule is not customized', () => {
+          const mockVersions: ThreeVersionsOf<string[]> = {
+            base_version: MissingVersion,
+            current_version: ['one', 'two', 'three'],
+            target_version: ['one', 'two', 'three'],
+          };
+
+          const result = scalarArrayDiffAlgorithm(mockVersions, false);
+
+          expect(result).toEqual(
+            expect.objectContaining({
+              has_base_version: false,
+              base_version: undefined,
+              merged_version: mockVersions.target_version,
+              diff_outcome: ThreeWayDiffOutcome.MissingBaseNoUpdate,
+              merge_outcome: ThreeWayMergeOutcome.Target,
+              conflict: ThreeWayDiffConflict.NONE,
+            })
+          );
+        });
+
+        it('returns NONE conflict if rule is customized', () => {
+          const mockVersions: ThreeVersionsOf<string[]> = {
+            base_version: MissingVersion,
+            current_version: ['one', 'two', 'three'],
+            target_version: ['one', 'two', 'three'],
+          };
+
+          const result = scalarArrayDiffAlgorithm(mockVersions, true);
+
+          expect(result).toEqual(
+            expect.objectContaining({
+              has_base_version: false,
+              base_version: undefined,
+              merged_version: mockVersions.target_version,
+              diff_outcome: ThreeWayDiffOutcome.MissingBaseNoUpdate,
+              merge_outcome: ThreeWayMergeOutcome.Target,
+              conflict: ThreeWayDiffConflict.NONE,
+            })
+          );
+        });
+      });
+
+      describe('if current_version and target_version are different - scenario -AB', () => {
+        it('returns target_version as merged output and NONE conflict if rule is not customized', () => {
+          const mockVersions: ThreeVersionsOf<string[]> = {
+            base_version: MissingVersion,
+            current_version: ['one', 'two', 'three'],
+            target_version: ['one', 'four', 'three'],
+          };
+
+          const result = scalarArrayDiffAlgorithm(mockVersions, false);
+
+          expect(result).toEqual(
+            expect.objectContaining({
+              has_base_version: false,
+              base_version: undefined,
+              merged_version: mockVersions.target_version,
+              diff_outcome: ThreeWayDiffOutcome.MissingBaseCanUpdate,
+              merge_outcome: ThreeWayMergeOutcome.Target,
+              conflict: ThreeWayDiffConflict.NONE,
+            })
+          );
+        });
+      });
+    });
+  });
+
+  describe('when base_version is missing', () => {
+    it('returns merged version of current and target as merged output if rule is customized', () => {
+      const mockVersions: ThreeVersionsOf<string[]> = {
+        base_version: MissingVersion,
+        current_version: ['one', 'two', 'three'],
+        target_version: ['one', 'four', 'three'],
+      };
+
+      const expectedMergedVersion = ['one', 'two', 'three', 'four'];
+
+      const scalarArrayDiffAlgorithm = createScalarArrayDiffAlgorithm({
+        missingBaseVersionStrategy: ScalarArrayDiffMissingBaseVersionStrategy.Merge,
+      });
+      const result = scalarArrayDiffAlgorithm(mockVersions, true);
+
+      expect(result).toEqual(
+        expect.objectContaining({
+          has_base_version: false,
+          base_version: undefined,
+          merged_version: expectedMergedVersion,
+          diff_outcome: ThreeWayDiffOutcome.MissingBaseCanUpdate,
+          merge_outcome: ThreeWayMergeOutcome.Merged,
+          conflict: ThreeWayDiffConflict.SOLVABLE,
+        })
+      );
+    });
+
+    it('returns target version of current and target as merged output if rule is customized', () => {
+      const targetVersion = ['one', 'four', 'three'];
+
+      const mockVersions: ThreeVersionsOf<string[]> = {
+        base_version: MissingVersion,
+        current_version: ['one', 'two', 'three'],
+        target_version: targetVersion,
+      };
+
+      const scalarArrayDiffAlgorithm = createScalarArrayDiffAlgorithm({
+        missingBaseVersionStrategy: ScalarArrayDiffMissingBaseVersionStrategy.UseTarget,
+      });
+      const result = scalarArrayDiffAlgorithm(mockVersions, true);
+
+      expect(result).toEqual(
+        expect.objectContaining({
+          has_base_version: false,
+          base_version: undefined,
+          merged_version: targetVersion,
+          diff_outcome: ThreeWayDiffOutcome.MissingBaseCanUpdate,
+          merge_outcome: ThreeWayMergeOutcome.Target,
+          conflict: ThreeWayDiffConflict.SOLVABLE,
+        })
+      );
     });
   });
 });

x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/diff/calculation/algorithms/scalar_array_diff_algorithm.ts

@@ -21,47 +21,90 @@ import {
 } from '../../../../../../../../common/api/detection_engine/prebuilt_rules';
 import { mergeDedupedArrays } from './helpers';
 
+type ScalarArrayDiffAlgorithm<TValue> = (
+  versions: ThreeVersionsOf<TValue[]>,
+  isRuleCustomized: boolean
+) => ThreeWayDiff<TValue[]>;
+
+/**
+ * This strategy applies when all these conditions are met:
+ * 1) when the base version is missing;
+ * 2) and there is an update from Elastic (current version != target version);
+ * 3) and the rule IS marked as customized.
+ *
+ * When all that is true, the scalar array diff algorithm uses this strategy
+ * to determine what to do, exactly.
+ */
+export enum ScalarArrayDiffMissingBaseVersionStrategy {
+  /**
+   * Merge the current and target versions and return the result as the merged version.
+   */
+  Merge = 'Merge',
+
+  /**
+   * Return the target version as the merged version.
+   */
+  UseTarget = 'UseTarget',
+}
+
+interface ScalarArrayDiffAlgorithmOptions {
+  /**
+   * Algorithm's behavior when the base version is missing and current field's
+   * value differs from the target value.
+   */
+  missingBaseVersionStrategy: ScalarArrayDiffMissingBaseVersionStrategy;
+}
+
 /**
  * Diff algorithm used for arrays of scalar values (eg. numbers, strings, booleans, etc.)
  *
  * NOTE: Diffing logic will be agnostic to array order
  */
-export const scalarArrayDiffAlgorithm = <TValue>(
-  versions: ThreeVersionsOf<TValue[]>,
-  isRuleCustomized: boolean
-): ThreeWayDiff<TValue[]> => {
-  const {
-    base_version: baseVersion,
-    current_version: currentVersion,
-    target_version: targetVersion,
-  } = versions;
+export function createScalarArrayDiffAlgorithm<TValue>(
+  options: ScalarArrayDiffAlgorithmOptions
+): ScalarArrayDiffAlgorithm<TValue> {
+  return function scalarArrayDiffAlgorithm(
+    versions: ThreeVersionsOf<TValue[]>,
+    isRuleCustomized: boolean
+  ) {
+    const {
+      base_version: baseVersion,
+      current_version: currentVersion,
+      target_version: targetVersion,
+    } = versions;
 
-  const diffOutcome = determineOrderAgnosticDiffOutcome(baseVersion, currentVersion, targetVersion);
-  const valueCanUpdate = determineIfValueCanUpdate(diffOutcome);
+    const diffOutcome = determineOrderAgnosticDiffOutcome(
+      baseVersion,
+      currentVersion,
+      targetVersion
+    );
+    const valueCanUpdate = determineIfValueCanUpdate(diffOutcome);
 
-  const hasBaseVersion = baseVersion !== MissingVersion;
+    const hasBaseVersion = baseVersion !== MissingVersion;
 
-  const { mergeOutcome, conflict, mergedVersion } = mergeVersions({
-    baseVersion: hasBaseVersion ? baseVersion : undefined,
-    currentVersion,
-    targetVersion,
-    diffOutcome,
-    isRuleCustomized,
-  });
+    const { mergeOutcome, conflict, mergedVersion } = mergeVersions({
+      baseVersion: hasBaseVersion ? baseVersion : undefined,
+      currentVersion,
+      targetVersion,
+      diffOutcome,
+      isRuleCustomized,
+      options,
+    });
 
-  return {
-    has_base_version: hasBaseVersion,
-    base_version: hasBaseVersion ? baseVersion : undefined,
-    current_version: currentVersion,
-    target_version: targetVersion,
-    merged_version: mergedVersion,
-    merge_outcome: mergeOutcome,
+    return {
+      has_base_version: hasBaseVersion,
+      base_version: hasBaseVersion ? baseVersion : undefined,
+      current_version: currentVersion,
+      target_version: targetVersion,
+      merged_version: mergedVersion,
+      merge_outcome: mergeOutcome,
 
-    diff_outcome: diffOutcome,
-    conflict,
-    has_update: valueCanUpdate,
+      diff_outcome: diffOutcome,
+      conflict,
+      has_update: valueCanUpdate,
+    };
   };
-};
+}
 
 interface MergeResult<TValue> {
   mergeOutcome: ThreeWayMergeOutcome;
@@ -75,6 +118,7 @@ interface MergeArgs<TValue> {
   targetVersion: TValue[];
   diffOutcome: ThreeWayDiffOutcome;
   isRuleCustomized: boolean;
+  options: ScalarArrayDiffAlgorithmOptions;
 }
 
 const mergeVersions = <TValue>({
@@ -83,6 +127,7 @@ const mergeVersions = <TValue>({
   targetVersion,
   diffOutcome,
   isRuleCustomized,
+  options,
 }: MergeArgs<TValue>): MergeResult<TValue> => {
   const dedupedBaseVersion = uniq(baseVersion);
   const dedupedCurrentVersion = uniq(currentVersion);
@@ -135,17 +180,35 @@ const mergeVersions = <TValue>({
     // Otherwise we treat scenario -AB as AAB
     // https://github.com/elastic/kibana/issues/210358#issuecomment-2654492854
     case ThreeWayDiffOutcome.MissingBaseCanUpdate: {
-      return isRuleCustomized
-        ? {
+      if (!isRuleCustomized) {
+        return {
+          mergedVersion: targetVersion,
+          mergeOutcome: ThreeWayMergeOutcome.Target,
+          conflict: ThreeWayDiffConflict.NONE,
+        };
+      }
+
+      switch (options.missingBaseVersionStrategy) {
+        case ScalarArrayDiffMissingBaseVersionStrategy.Merge: {
+          return {
             mergedVersion: union(dedupedCurrentVersion, dedupedTargetVersion),
             mergeOutcome: ThreeWayMergeOutcome.Merged,
             conflict: ThreeWayDiffConflict.SOLVABLE,
-          }
-        : {
+          };
+        }
+
+        case ScalarArrayDiffMissingBaseVersionStrategy.UseTarget: {
+          return {
             mergedVersion: targetVersion,
             mergeOutcome: ThreeWayMergeOutcome.Target,
-            conflict: ThreeWayDiffConflict.NONE,
+            conflict: ThreeWayDiffConflict.SOLVABLE,
           };
+        }
+
+        default: {
+          return assertUnreachable(options.missingBaseVersionStrategy);
+        }
+      }
     }
     default:
       return assertUnreachable(diffOutcome);

x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/diff/calculation/calculate_rule_fields_diff.ts

@@ -41,7 +41,6 @@ import {
   dataSourceDiffAlgorithm,
   multiLineStringDiffAlgorithm,
   numberDiffAlgorithm,
-  scalarArrayDiffAlgorithm,
   simpleDiffAlgorithm,
   singleLineStringDiffAlgorithm,
   kqlQueryDiffAlgorithm,
@@ -50,6 +49,10 @@ import {
   ruleTypeDiffAlgorithm,
   forceTargetVersionDiffAlgorithm,
 } from './algorithms';
+import {
+  ScalarArrayDiffMissingBaseVersionStrategy,
+  createScalarArrayDiffAlgorithm,
+} from './algorithms/scalar_array_diff_algorithm';
 
 const BASE_TYPE_ERROR = `Base version can't be of different rule type`;
 const TARGET_TYPE_ERROR = `Target version can't be of different rule type`;
@@ -215,13 +218,17 @@ const commonFieldsDiffAlgorithms: FieldsDiffAlgorithmsFor<DiffableCommonFields>
    */
   version: forceTargetVersionDiffAlgorithm,
   name: singleLineStringDiffAlgorithm,
-  tags: scalarArrayDiffAlgorithm,
+  tags: createScalarArrayDiffAlgorithm({
+    missingBaseVersionStrategy: ScalarArrayDiffMissingBaseVersionStrategy.Merge,
+  }),
   description: multiLineStringDiffAlgorithm,
   severity: singleLineStringDiffAlgorithm,
   severity_mapping: simpleDiffAlgorithm,
   risk_score: numberDiffAlgorithm,
   risk_score_mapping: simpleDiffAlgorithm,
-  references: scalarArrayDiffAlgorithm,
+  references: createScalarArrayDiffAlgorithm({
+    missingBaseVersionStrategy: ScalarArrayDiffMissingBaseVersionStrategy.Merge,
+  }),
   false_positives: simpleDiffAlgorithm,
   threat: simpleDiffAlgorithm,
   note: multiLineStringDiffAlgorithm,
@@ -304,7 +311,9 @@ const threatMatchFieldsDiffAlgorithms: FieldsDiffAlgorithmsFor<DiffableThreatMat
   kql_query: kqlQueryDiffAlgorithm,
   data_source: dataSourceDiffAlgorithm,
   threat_query: kqlQueryDiffAlgorithm,
-  threat_index: scalarArrayDiffAlgorithm,
+  threat_index: createScalarArrayDiffAlgorithm({
+    missingBaseVersionStrategy: ScalarArrayDiffMissingBaseVersionStrategy.UseTarget,
+  }),
   threat_mapping: simpleDiffAlgorithm,
   threat_indicator_path: singleLineStringDiffAlgorithm,
   alert_suppression: simpleDiffAlgorithm,
@@ -355,7 +364,9 @@ const newTermsFieldsDiffAlgorithms: FieldsDiffAlgorithmsFor<DiffableNewTermsFiel
   type: ruleTypeDiffAlgorithm,
   kql_query: kqlQueryDiffAlgorithm,
   data_source: dataSourceDiffAlgorithm,
-  new_terms_fields: scalarArrayDiffAlgorithm,
+  new_terms_fields: createScalarArrayDiffAlgorithm({
+    missingBaseVersionStrategy: ScalarArrayDiffMissingBaseVersionStrategy.UseTarget,
+  }),
   history_window_start: singleLineStringDiffAlgorithm,
   alert_suppression: simpleDiffAlgorithm,
 };

x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/prebuilt_rule_customization/customization_enabled/diffable_rule_fields/test_helpers.ts

@@ -10,7 +10,9 @@ import { isUndefined, omitBy } from 'lodash';
 import type {
   PartialRuleDiff,
   RuleResponse,
+  UpgradeConflictResolution,
 } from '@kbn/security-solution-plugin/common/api/detection_engine';
+import { UpgradeConflictResolutionEnum } from '@kbn/security-solution-plugin/common/api/detection_engine';
 import { ModeEnum } from '@kbn/security-solution-plugin/common/api/detection_engine';
 import {
   ThreeWayDiffConflict,
@@ -170,6 +172,7 @@ export function testFieldUpgradeReview(
 
 interface TestFieldUpgradesToMergedValueParams {
   ruleUpgradeAssets: TestFieldRuleUpgradeAssets;
+  onConflict?: UpgradeConflictResolution;
   diffableRuleFieldName: string;
   expectedFieldsAfterUpgrade: Partial<RuleResponse>;
 }
@@ -187,6 +190,7 @@ interface TestFieldUpgradesToMergedValueParams {
 export function testFieldUpgradesToMergedValue(
   {
     ruleUpgradeAssets,
+    onConflict = UpgradeConflictResolutionEnum.SKIP,
     diffableRuleFieldName,
     expectedFieldsAfterUpgrade,
   }: TestFieldUpgradesToMergedValueParams,
@@ -215,6 +219,7 @@ export function testFieldUpgradesToMergedValue(
 
     const response = await performUpgradePrebuiltRules(es, supertest, {
       mode: ModeEnum.SPECIFIC_RULES,
+      on_conflict: onConflict,
       rules: [
         {
           rule_id: ruleUpgradeAssets.upgrade.rule_id ?? DEFAULT_TEST_RULE_ID,

x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/prebuilt_rule_customization/customization_enabled/diffable_rule_fields/type_specific_fields/new_terms_fields.ts

@@ -5,7 +5,10 @@
  * 2.0.
  */
 
-import { ThreeWayDiffOutcome } from '@kbn/security-solution-plugin/common/api/detection_engine';
+import {
+  ThreeWayDiffOutcome,
+  UpgradeConflictResolutionEnum,
+} from '@kbn/security-solution-plugin/common/api/detection_engine';
 import { FtrProviderContext } from '../../../../../../../../ftr_provider_context';
 import type { TestFieldRuleUpgradeAssets } from '../test_helpers';
 import {
@@ -302,16 +305,26 @@ export function newTermsFieldsField({ getService }: FtrProviderContext): void {
             ruleUpgradeAssets,
             diffableRuleFieldName: 'new_terms_fields',
             expectedDiffOutcome: ThreeWayDiffOutcome.MissingBaseCanUpdate,
-            isMergableField: true,
+            isMergableField: false,
             expectedFieldDiffValues: {
               current: ['fieldB'],
               target: ['fieldA', 'fieldC'],
-              merged: ['fieldB', 'fieldA', 'fieldC'],
+              merged: ['fieldA', 'fieldC'],
             },
           },
           getService
         );
 
+        testFieldUpgradesToMergedValue(
+          {
+            ruleUpgradeAssets,
+            onConflict: UpgradeConflictResolutionEnum.UPGRADE_SOLVABLE,
+            diffableRuleFieldName: 'new_terms_fields',
+            expectedFieldsAfterUpgrade: { new_terms_fields: ['fieldA', 'fieldC'] },
+          },
+          getService
+        );
+
         testFieldUpgradesToResolvedValue(
           {
             ruleUpgradeAssets,

x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/prebuilt_rule_customization/customization_enabled/diffable_rule_fields/type_specific_fields/threat_index.ts

@@ -5,7 +5,10 @@
  * 2.0.
  */
 
-import { ThreeWayDiffOutcome } from '@kbn/security-solution-plugin/common/api/detection_engine';
+import {
+  ThreeWayDiffOutcome,
+  UpgradeConflictResolutionEnum,
+} from '@kbn/security-solution-plugin/common/api/detection_engine';
 import { FtrProviderContext } from '../../../../../../../../ftr_provider_context';
 import type { TestFieldRuleUpgradeAssets } from '../test_helpers';
 import {
@@ -302,16 +305,26 @@ export function threatIndexField({ getService }: FtrProviderContext): void {
             ruleUpgradeAssets,
             diffableRuleFieldName: 'threat_index',
             expectedDiffOutcome: ThreeWayDiffOutcome.MissingBaseCanUpdate,
-            isMergableField: true,
+            isMergableField: false,
             expectedFieldDiffValues: {
               current: ['indexD'],
               target: ['indexB', 'indexC'],
-              merged: ['indexD', 'indexB', 'indexC'],
+              merged: ['indexB', 'indexC'],
             },
           },
           getService
         );
 
+        testFieldUpgradesToMergedValue(
+          {
+            ruleUpgradeAssets,
+            onConflict: UpgradeConflictResolutionEnum.UPGRADE_SOLVABLE,
+            diffableRuleFieldName: 'threat_index',
+            expectedFieldsAfterUpgrade: { threat_index: ['indexB', 'indexC'] },
+          },
+          getService
+        );
+
         testFieldUpgradesToResolvedValue(
           {
             ruleUpgradeAssets,