github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-23 09:19:04 -04:00
Code Issues Projects Releases Packages Wiki Activity

[8.15] [Security Gen AI] [ Integration Assistant ] Modify pipeline template to handle dates as array of string in the logs (#187643) (#187726)

Browse source 
# Backport

This will backport the following commits from `main` to `8.15`:
- [[Security Gen AI] [ Integration Assistant ] Modify pipeline template
to handle dates as array of string in the logs
(#187643)](https://github.com/elastic/kibana/pull/187643)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Bharat
Pasupula","email":"123897612+bhapas@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-07-08T07:08:14Z","message":"[Security
Gen AI] [ Integration Assistant ] Modify pipeline template to handle
dates as array of string in the logs (#187643)\n\n##
Summary\r\n\r\nThere could
be\r\n[scenario](https://docs.paloaltonetworks.com/iot/iot-security-api-reference/iot-security-api/get-vulnerability-instances)\r\nwhere
a date can come in as an array -\r\n\r\n`\"detected_date\": [\r\n
\"2021-04-19T23:59:59\"\r\n ],`\r\n\r\nThe `date` processor fails to
handle an array with a string.\r\nThis PR adds a `script` processor as a
pre processor to date processor\r\nto convert the array of string into a
string.\r\n\r\nPipeline after the changes\r\n\r\n```json\r\n{\r\n
\"script\": {\r\n \"tag\": \"script_convert_array_to_string\",\r\n
\"description\": \"Ensures the date processor does not receive an array
value.\",\r\n \"lang\": \"painless\",\r\n \"source\": \"if
(ctx.palo_alto_iot.vulnerability.detected_date instanceof ArrayList){\\n
ctx.palo_alto_iot.vulnerability.detected_date =
ctx.palo_alto_iot.vulnerability.detected_date[0];\\n}\\n\"\r\n }\r\n
},\r\n {\r\n \"date\": {\r\n \"if\":
\"ctx.palo_alto_iot?.vulnerability?.detected_date != null\",\r\n
\"tag\":
\"date_processor_palo_alto_iot.vulnerability.detected_date\",\r\n
\"field\": \"palo_alto_iot.vulnerability.detected_date\",\r\n
\"target_field\": \"event.start\",\r\n \"formats\": [\r\n
\"ISO8601\"\r\n ]\r\n }\r\n
}\r\n```\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"a347d7b5f1f7b43add64ec9e5751a41888f9afde","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:skip","Team:
SecuritySolution","v8.15.0","v8.16.0"],"title":"[Security Gen AI] [
Integration Assistant ] Modify pipeline template to handle dates as
array of string in the
logs","number":187643,"url":"https://github.com/elastic/kibana/pull/187643","mergeCommit":{"message":"[Security
Gen AI] [ Integration Assistant ] Modify pipeline template to handle
dates as array of string in the logs (#187643)\n\n##
Summary\r\n\r\nThere could
be\r\n[scenario](https://docs.paloaltonetworks.com/iot/iot-security-api-reference/iot-security-api/get-vulnerability-instances)\r\nwhere
a date can come in as an array -\r\n\r\n`\"detected_date\": [\r\n
\"2021-04-19T23:59:59\"\r\n ],`\r\n\r\nThe `date` processor fails to
handle an array with a string.\r\nThis PR adds a `script` processor as a
pre processor to date processor\r\nto convert the array of string into a
string.\r\n\r\nPipeline after the changes\r\n\r\n```json\r\n{\r\n
\"script\": {\r\n \"tag\": \"script_convert_array_to_string\",\r\n
\"description\": \"Ensures the date processor does not receive an array
value.\",\r\n \"lang\": \"painless\",\r\n \"source\": \"if
(ctx.palo_alto_iot.vulnerability.detected_date instanceof ArrayList){\\n
ctx.palo_alto_iot.vulnerability.detected_date =
ctx.palo_alto_iot.vulnerability.detected_date[0];\\n}\\n\"\r\n }\r\n
},\r\n {\r\n \"date\": {\r\n \"if\":
\"ctx.palo_alto_iot?.vulnerability?.detected_date != null\",\r\n
\"tag\":
\"date_processor_palo_alto_iot.vulnerability.detected_date\",\r\n
\"field\": \"palo_alto_iot.vulnerability.detected_date\",\r\n
\"target_field\": \"event.start\",\r\n \"formats\": [\r\n
\"ISO8601\"\r\n ]\r\n }\r\n
}\r\n```\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"a347d7b5f1f7b43add64ec9e5751a41888f9afde"}},"sourceBranch":"main","suggestedTargetBranches":["8.15"],"targetPullRequestStates":[{"branch":"8.15","label":"v8.15.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/187643","number":187643,"mergeCommit":{"message":"[Security
Gen AI] [ Integration Assistant ] Modify pipeline template to handle
dates as array of string in the logs (#187643)\n\n##
Summary\r\n\r\nThere could
be\r\n[scenario](https://docs.paloaltonetworks.com/iot/iot-security-api-reference/iot-security-api/get-vulnerability-instances)\r\nwhere
a date can come in as an array -\r\n\r\n`\"detected_date\": [\r\n
\"2021-04-19T23:59:59\"\r\n ],`\r\n\r\nThe `date` processor fails to
handle an array with a string.\r\nThis PR adds a `script` processor as a
pre processor to date processor\r\nto convert the array of string into a
string.\r\n\r\nPipeline after the changes\r\n\r\n```json\r\n{\r\n
\"script\": {\r\n \"tag\": \"script_convert_array_to_string\",\r\n
\"description\": \"Ensures the date processor does not receive an array
value.\",\r\n \"lang\": \"painless\",\r\n \"source\": \"if
(ctx.palo_alto_iot.vulnerability.detected_date instanceof ArrayList){\\n
ctx.palo_alto_iot.vulnerability.detected_date =
ctx.palo_alto_iot.vulnerability.detected_date[0];\\n}\\n\"\r\n }\r\n
},\r\n {\r\n \"date\": {\r\n \"if\":
\"ctx.palo_alto_iot?.vulnerability?.detected_date != null\",\r\n
\"tag\":
\"date_processor_palo_alto_iot.vulnerability.detected_date\",\r\n
\"field\": \"palo_alto_iot.vulnerability.detected_date\",\r\n
\"target_field\": \"event.start\",\r\n \"formats\": [\r\n
\"ISO8601\"\r\n ]\r\n }\r\n
}\r\n```\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"a347d7b5f1f7b43add64ec9e5751a41888f9afde"}}]}]
BACKPORT-->

Co-authored-by: Bharat Pasupula <123897612+bhapas@users.noreply.github.com>
This commit is contained in:
Kibana Machine 2024-07-08 10:45:39 +02:00 • committed by GitHub
parent 2490c13086
commit 7e0626c9de
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 19 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

10
x-pack/plugins/integration_assistant/__jest__/fixtures/ecs_mapping.ts
View file

@ -86,9 +86,19 @@ export const ecsMappingExpectedResults = {
target_field: 'mysql_enterprise.audit',
},
},
{
script: {
description: 'Ensures the date processor does not receive an array value.',
lang: 'painless',
source:
'if (ctx.mysql_enterprise.audit.timestamp instanceof ArrayList){\n ctx.mysql_enterprise.audit.timestamp = ctx.mysql_enterprise.audit.timestamp[0];\n}\n',
tag: 'script_convert_array_to_string',
},
},
{
date: {
field: 'mysql_enterprise.audit.timestamp',
tag: 'date_processor_mysql_enterprise.audit.timestamp',
target_field: '@timestamp',
formats: ['yyyy-MM-dd HH:mm:ss'],
if: 'ctx.mysql_enterprise?.audit?.timestamp != null',

9
x-pack/plugins/integration_assistant/server/templates/pipeline.yml.njk
View file

@ -27,6 +27,14 @@ processors:
target_field: {% if value.target_field | startswith('@') %}"{{ value.target_field }}"{% else %}{{ value.target_field }}{% endif %}
ignore_missing: true{% endif %}
{% if key == 'date' %}
- script:
description: Ensures the date processor does not receive an array value.
tag: script_convert_array_to_string
lang: painless
{% raw %}source: |
if (ctx.{% endraw %}{{ value.field }}{% raw %} instanceof ArrayList){
ctx.{% endraw %}{{ value.field }}{% raw %} = ctx.{% endraw %}{{ value.field }}{% raw %}[0];
}{% endraw %}
- {{ key }}:
field: {{ value.field }}
target_field: {% if value.target_field | startswith('@') %}"{{ value.target_field }}"{% else %}{{ value.target_field }}{% endif %}
@ -34,6 +42,7 @@ processors:
{% for format in value.formats %}
- {{ format }}
{% endfor %}
tag: date_processor_{{ value.field}}
if: "ctx.{{ value.if }} != null"{% endif %}
{% if key == 'convert' %}
- {{ key }}: