[8.6] [RAM] Add strict_date_optional_time to excludeHitsFromPreviousRun time range query for ES query rule (#151958) (#152254)

# Backport

This will backport the following commits from `main` to `8.6`:
- [[RAM] Add strict_date_optional_time to excludeHitsFromPreviousRun
time range query for ES query rule
(#151958)](https://github.com/elastic/kibana/pull/151958)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Jiawei
Wu","email":"74562234+JiaweiWu@users.noreply.github.com"},"sourceCommit":{"committedDate":"2023-02-27T18:14:12Z","message":"[RAM]
Add strict_date_optional_time to excludeHitsFromPreviousRun time range
query for ES query rule (#151958)\n\n## Summary\r\nResolves:
https://github.com/elastic/kibana/issues/150448\r\n\r\nAdds
`strict_date_optional_time` to the `excludeHitsFromPreviousRun`\r\nlast
timestamp range query to allow querying data views with
non-ISO\r\n`timefields`. This fixes the crash that occurred when we
didn't have\r\nthis field when the ES query ran consecutively to exclude
previous hits\r\nfor data views with non-iso time fields.\r\n\r\n### To
test:\r\n\r\n##### 1. Create mapping with date with non-ISO
format\r\n```\r\nPUT test-index\r\n{\r\n \"mappings\": {\r\n
\"properties\": {\r\n \"date\": {\r\n \"type\": \"date\",\r\n
\"format\": \"epoch_second\"\r\n },\r\n \"test_field\": {\r\n \"type\":
\"keyword\"\r\n }\r\n }\r\n }\r\n}\r\n```\r\n##### 2. Insert test
data\r\n```\r\nPUT test-index/_bulk?refresh\r\n{ \"index\" : { \"_id\" :
\"1\" } }\r\n{ \"date\": 1677105176, \"test_field\": \"hi\"}\r\n{
\"index\" : { \"_id\" : \"2\" } }\r\n{ \"date\": 1677108776,
\"test_field\": \"bye\"}\r\n```\r\n\r\n##### 3. Create data view with
the `date` as the time field.\r\n\r\n##### 4. Create a Elasticsearch
query rule with KQL, making sure\r\n`Exclude matches from previous runs`
is checked. Let rule run twice to\r\ngenerate alerts\r\n\r\n##### 5. The
rule should run without errors (especially after the first\r\nrun).
Before this fix, the rule would error after 1 run.\r\n\r\n###
Checklist\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"cfd0e8731a5921651a8a578afe634a54a225fdf1","branchLabelMapping":{"^v8.8.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:skip","Team:ResponseOps","Feature:Alerting/RulesManagement","backport:prev-minor","v8.7.0","v8.6.2","v8.8.0"],"number":151958,"url":"https://github.com/elastic/kibana/pull/151958","mergeCommit":{"message":"[RAM]
Add strict_date_optional_time to excludeHitsFromPreviousRun time range
query for ES query rule (#151958)\n\n## Summary\r\nResolves:
https://github.com/elastic/kibana/issues/150448\r\n\r\nAdds
`strict_date_optional_time` to the `excludeHitsFromPreviousRun`\r\nlast
timestamp range query to allow querying data views with
non-ISO\r\n`timefields`. This fixes the crash that occurred when we
didn't have\r\nthis field when the ES query ran consecutively to exclude
previous hits\r\nfor data views with non-iso time fields.\r\n\r\n### To
test:\r\n\r\n##### 1. Create mapping with date with non-ISO
format\r\n```\r\nPUT test-index\r\n{\r\n \"mappings\": {\r\n
\"properties\": {\r\n \"date\": {\r\n \"type\": \"date\",\r\n
\"format\": \"epoch_second\"\r\n },\r\n \"test_field\": {\r\n \"type\":
\"keyword\"\r\n }\r\n }\r\n }\r\n}\r\n```\r\n##### 2. Insert test
data\r\n```\r\nPUT test-index/_bulk?refresh\r\n{ \"index\" : { \"_id\" :
\"1\" } }\r\n{ \"date\": 1677105176, \"test_field\": \"hi\"}\r\n{
\"index\" : { \"_id\" : \"2\" } }\r\n{ \"date\": 1677108776,
\"test_field\": \"bye\"}\r\n```\r\n\r\n##### 3. Create data view with
the `date` as the time field.\r\n\r\n##### 4. Create a Elasticsearch
query rule with KQL, making sure\r\n`Exclude matches from previous runs`
is checked. Let rule run twice to\r\ngenerate alerts\r\n\r\n##### 5. The
rule should run without errors (especially after the first\r\nrun).
Before this fix, the rule would error after 1 run.\r\n\r\n###
Checklist\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"cfd0e8731a5921651a8a578afe634a54a225fdf1"}},"sourceBranch":"main","suggestedTargetBranches":["8.7","8.6"],"targetPullRequestStates":[{"branch":"8.7","label":"v8.7.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.6","label":"v8.6.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.8.0","labelRegex":"^v8.8.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/151958","number":151958,"mergeCommit":{"message":"[RAM]
Add strict_date_optional_time to excludeHitsFromPreviousRun time range
query for ES query rule (#151958)\n\n## Summary\r\nResolves:
https://github.com/elastic/kibana/issues/150448\r\n\r\nAdds
`strict_date_optional_time` to the `excludeHitsFromPreviousRun`\r\nlast
timestamp range query to allow querying data views with
non-ISO\r\n`timefields`. This fixes the crash that occurred when we
didn't have\r\nthis field when the ES query ran consecutively to exclude
previous hits\r\nfor data views with non-iso time fields.\r\n\r\n### To
test:\r\n\r\n##### 1. Create mapping with date with non-ISO
format\r\n```\r\nPUT test-index\r\n{\r\n \"mappings\": {\r\n
\"properties\": {\r\n \"date\": {\r\n \"type\": \"date\",\r\n
\"format\": \"epoch_second\"\r\n },\r\n \"test_field\": {\r\n \"type\":
\"keyword\"\r\n }\r\n }\r\n }\r\n}\r\n```\r\n##### 2. Insert test
data\r\n```\r\nPUT test-index/_bulk?refresh\r\n{ \"index\" : { \"_id\" :
\"1\" } }\r\n{ \"date\": 1677105176, \"test_field\": \"hi\"}\r\n{
\"index\" : { \"_id\" : \"2\" } }\r\n{ \"date\": 1677108776,
\"test_field\": \"bye\"}\r\n```\r\n\r\n##### 3. Create data view with
the `date` as the time field.\r\n\r\n##### 4. Create a Elasticsearch
query rule with KQL, making sure\r\n`Exclude matches from previous runs`
is checked. Let rule run twice to\r\ngenerate alerts\r\n\r\n##### 5. The
rule should run without errors (especially after the first\r\nrun).
Before this fix, the rule would error after 1 run.\r\n\r\n###
Checklist\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"cfd0e8731a5921651a8a578afe634a54a225fdf1"}}]}]
BACKPORT-->

Co-authored-by: Jiawei Wu <74562234+JiaweiWu@users.noreply.github.com>

x-pack/plugins/stack_alerts/server/rule_types/es_query/lib/fetch_search_source_query.test.ts

@@ -116,6 +116,7 @@ describe('fetchSearchSourceQuery', () => {
               Object {
                 "range": Object {
                   "time": Object {
+                    "format": "strict_date_optional_time",
                     "gt": "2020-02-09T23:12:41.941Z",
                   },
                 },

x-pack/plugins/stack_alerts/server/rule_types/es_query/lib/fetch_search_source_query.ts

@@ -76,7 +76,11 @@ export function updateSearchSource(
       // add additional filter for documents with a timestamp greater then
       // the timestamp of the previous run, so that those documents are not counted twice
       const field = index.fields.find((f) => f.name === timeFieldName);
-      const addTimeRangeField = buildRangeFilter(field!, { gt: latestTimestamp }, index);
+      const addTimeRangeField = buildRangeFilter(
+        field!,
+        { gt: latestTimestamp, format: 'strict_date_optional_time' },
+        index
+      );
       filters.push(addTimeRangeField);
     }
   }