mirror of
https://github.com/elastic/kibana.git
synced 2025-04-24 17:59:23 -04:00
# Backport This will backport the following commits from `main` to `8.17`: - [[Automatic Import] Fix Structured log flow to handle different type of structured syslogs (#212611)](https://github.com/elastic/kibana/pull/212611) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Bharat Pasupula","email":"123897612+bhapas@users.noreply.github.com"},"sourceCommit":{"committedDate":"2025-02-27T12:32:17Z","message":"[Automatic Import] Fix Structured log flow to handle different type of structured syslogs (#212611)\n\n## Release note\nFix structured log flow to handle multiple types of structured logs.\n\n## Summary\nThe structured log flow has some issues where the KV header validation\nfails for some type of logs. This PR fixes the flow to match variety of\nstructured syslog messages.\n\nA variety of logs are tested.\n\n```\n[2025-01-03T07:48:58.989821Z] [DEBUG] AuthService - EventID=361a5289eaf8e42b4c195b9b | Message=\"Session expired\" | UserID=2882 | Duration=376ms\n[2025-01-29T17:34:18.989830Z] [ERROR] InventoryService - EventID=acbb20d3c955edf718e691d9 | Message=\"Item restocked\" | UserID=9656 | Duration=421ms\n[2025-01-11T21:51:54.989839Z] [ERROR] APIGateway - EventID=9c273d43b946020d5fdbe36c | Message=\"Response sent\" | UserID=1468 | Duration=409ms\n[2025-01-20T08:40:22.989848Z] [WARN] PaymentService - EventID=ae8c1425079119b848fa451cb7a | Message=\"3D Secure required\" | UserID=9353 | Duration=270ms\n```\n\n```\n2021-10-22 22:11:32,131 DEBUG [org.keycloak.events] (default task-3) type=CODE_TO_TOKEN, realmId=test, clientId=security-admin-console, userId=ce637d23--4fca-9088-1aea1d053e19, ipAddress=10.1.2.1, token_id=561459c0-75f1-46d4-986d, grant_type=authorization_code, refresh_token_type=Refresh, scope=openid, refresh_token_id=07434488-ca99-412a-c2e47c93d6d1, code_id=bae6e56e-368f-4809-48cfb6279f5e, client_auth_method=client-secret\n2021-10-22 22:12:09,871 DEBUG [org.keycloak.events] (default task-3) operationType=CREATE, realmId=test, clientId=7bcaf1cb-820a-40f1-75ced03ef03b, userId=ce637d23-b89c-4fca-1aea1d053e19, ipAddress=10.1.2.6, resourceType=USER, resourcePath=users/07972d16-b173-803d-90f211080f40\n```\n\n```\n[18/Feb/2025:22:39:18 +0000] CONNECT conn=730729 from=10.2.2.9:56518 to=10.2.1.14:4389 protocol=LDAP\n[18/Feb/2025:22:39:16 +0000] CONNECT conn=207223 from=10.2.1.24:55730 to=10.1.3.7:4389 protocol=LDAP\n```\n\n```\n<134>1 1647479580.487048774 MX84_2 airmarshal_events type=rogue_ssid_detected ssid='' bssid='AA:17:C8:D8:51' src='AA:17:C8:D8:51' dst='FF:FF:FF:FF:FF' wired_mac='AC:17:C7:D8:51' vlan_id='0' channel='6' rssi='35' fc_type='0' fc_subtype='8'\n<134>1 1647479604.334549372 MX84_5 airmarshal_events type=rogue_ssid_detected ssid='' bssid='92:17:C7:D8:51' src='92:17:C8:D8:51' dst='6A:3A:3E:85:F6' wired_mac='AC:17:C7:D8:51' vlan_id='0' channel='6' rssi='23' fc_type='0' fc_subtype='5'\n```\n\n### Checklist\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"f579f2d637ce6e7e51f15e32ba8c5d8ba554478e","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","backport:prev-minor","backport:prev-major","Team:Security-Scalability","backport:version","Feature:AutomaticImport","v9.1.0","backport:8.18"],"title":"[Automatic Import] Fix Structured log flow to handle different type of structured syslogs","number":212611,"url":"https://github.com/elastic/kibana/pull/212611","mergeCommit":{"message":"[Automatic Import] Fix Structured log flow to handle different type of structured syslogs (#212611)\n\n## Release note\nFix structured log flow to handle multiple types of structured logs.\n\n## Summary\nThe structured log flow has some issues where the KV header validation\nfails for some type of logs. This PR fixes the flow to match variety of\nstructured syslog messages.\n\nA variety of logs are tested.\n\n```\n[2025-01-03T07:48:58.989821Z] [DEBUG] AuthService - EventID=361a5289eaf8e42b4c195b9b | Message=\"Session expired\" | UserID=2882 | Duration=376ms\n[2025-01-29T17:34:18.989830Z] [ERROR] InventoryService - EventID=acbb20d3c955edf718e691d9 | Message=\"Item restocked\" | UserID=9656 | Duration=421ms\n[2025-01-11T21:51:54.989839Z] [ERROR] APIGateway - EventID=9c273d43b946020d5fdbe36c | Message=\"Response sent\" | UserID=1468 | Duration=409ms\n[2025-01-20T08:40:22.989848Z] [WARN] PaymentService - EventID=ae8c1425079119b848fa451cb7a | Message=\"3D Secure required\" | UserID=9353 | Duration=270ms\n```\n\n```\n2021-10-22 22:11:32,131 DEBUG [org.keycloak.events] (default task-3) type=CODE_TO_TOKEN, realmId=test, clientId=security-admin-console, userId=ce637d23--4fca-9088-1aea1d053e19, ipAddress=10.1.2.1, token_id=561459c0-75f1-46d4-986d, grant_type=authorization_code, refresh_token_type=Refresh, scope=openid, refresh_token_id=07434488-ca99-412a-c2e47c93d6d1, code_id=bae6e56e-368f-4809-48cfb6279f5e, client_auth_method=client-secret\n2021-10-22 22:12:09,871 DEBUG [org.keycloak.events] (default task-3) operationType=CREATE, realmId=test, clientId=7bcaf1cb-820a-40f1-75ced03ef03b, userId=ce637d23-b89c-4fca-1aea1d053e19, ipAddress=10.1.2.6, resourceType=USER, resourcePath=users/07972d16-b173-803d-90f211080f40\n```\n\n```\n[18/Feb/2025:22:39:18 +0000] CONNECT conn=730729 from=10.2.2.9:56518 to=10.2.1.14:4389 protocol=LDAP\n[18/Feb/2025:22:39:16 +0000] CONNECT conn=207223 from=10.2.1.24:55730 to=10.1.3.7:4389 protocol=LDAP\n```\n\n```\n<134>1 1647479580.487048774 MX84_2 airmarshal_events type=rogue_ssid_detected ssid='' bssid='AA:17:C8:D8:51' src='AA:17:C8:D8:51' dst='FF:FF:FF:FF:FF' wired_mac='AC:17:C7:D8:51' vlan_id='0' channel='6' rssi='35' fc_type='0' fc_subtype='8'\n<134>1 1647479604.334549372 MX84_5 airmarshal_events type=rogue_ssid_detected ssid='' bssid='92:17:C7:D8:51' src='92:17:C8:D8:51' dst='6A:3A:3E:85:F6' wired_mac='AC:17:C7:D8:51' vlan_id='0' channel='6' rssi='23' fc_type='0' fc_subtype='5'\n```\n\n### Checklist\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"f579f2d637ce6e7e51f15e32ba8c5d8ba554478e"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/212611","number":212611,"mergeCommit":{"message":"[Automatic Import] Fix Structured log flow to handle different type of structured syslogs (#212611)\n\n## Release note\nFix structured log flow to handle multiple types of structured logs.\n\n## Summary\nThe structured log flow has some issues where the KV header validation\nfails for some type of logs. This PR fixes the flow to match variety of\nstructured syslog messages.\n\nA variety of logs are tested.\n\n```\n[2025-01-03T07:48:58.989821Z] [DEBUG] AuthService - EventID=361a5289eaf8e42b4c195b9b | Message=\"Session expired\" | UserID=2882 | Duration=376ms\n[2025-01-29T17:34:18.989830Z] [ERROR] InventoryService - EventID=acbb20d3c955edf718e691d9 | Message=\"Item restocked\" | UserID=9656 | Duration=421ms\n[2025-01-11T21:51:54.989839Z] [ERROR] APIGateway - EventID=9c273d43b946020d5fdbe36c | Message=\"Response sent\" | UserID=1468 | Duration=409ms\n[2025-01-20T08:40:22.989848Z] [WARN] PaymentService - EventID=ae8c1425079119b848fa451cb7a | Message=\"3D Secure required\" | UserID=9353 | Duration=270ms\n```\n\n```\n2021-10-22 22:11:32,131 DEBUG [org.keycloak.events] (default task-3) type=CODE_TO_TOKEN, realmId=test, clientId=security-admin-console, userId=ce637d23--4fca-9088-1aea1d053e19, ipAddress=10.1.2.1, token_id=561459c0-75f1-46d4-986d, grant_type=authorization_code, refresh_token_type=Refresh, scope=openid, refresh_token_id=07434488-ca99-412a-c2e47c93d6d1, code_id=bae6e56e-368f-4809-48cfb6279f5e, client_auth_method=client-secret\n2021-10-22 22:12:09,871 DEBUG [org.keycloak.events] (default task-3) operationType=CREATE, realmId=test, clientId=7bcaf1cb-820a-40f1-75ced03ef03b, userId=ce637d23-b89c-4fca-1aea1d053e19, ipAddress=10.1.2.6, resourceType=USER, resourcePath=users/07972d16-b173-803d-90f211080f40\n```\n\n```\n[18/Feb/2025:22:39:18 +0000] CONNECT conn=730729 from=10.2.2.9:56518 to=10.2.1.14:4389 protocol=LDAP\n[18/Feb/2025:22:39:16 +0000] CONNECT conn=207223 from=10.2.1.24:55730 to=10.1.3.7:4389 protocol=LDAP\n```\n\n```\n<134>1 1647479580.487048774 MX84_2 airmarshal_events type=rogue_ssid_detected ssid='' bssid='AA:17:C8:D8:51' src='AA:17:C8:D8:51' dst='FF:FF:FF:FF:FF' wired_mac='AC:17:C7:D8:51' vlan_id='0' channel='6' rssi='35' fc_type='0' fc_subtype='8'\n<134>1 1647479604.334549372 MX84_5 airmarshal_events type=rogue_ssid_detected ssid='' bssid='92:17:C7:D8:51' src='92:17:C8:D8:51' dst='6A:3A:3E:85:F6' wired_mac='AC:17:C7:D8:51' vlan_id='0' channel='6' rssi='23' fc_type='0' fc_subtype='5'\n```\n\n### Checklist\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"f579f2d637ce6e7e51f15e32ba8c5d8ba554478e"}}]}] BACKPORT--> Co-authored-by: Bharat Pasupula <123897612+bhapas@users.noreply.github.com> |
||
---|---|---|
.. | ||
actions | ||
ai_infra | ||
aiops | ||
alerting | ||
banners | ||
canvas | ||
cases | ||
cloud | ||
cloud_defend | ||
cloud_integrations | ||
cloud_security_posture | ||
cross_cluster_replication | ||
custom_branding | ||
dashboard_enhanced | ||
data_quality | ||
data_usage | ||
data_visualizer | ||
discover_enhanced | ||
drilldowns | ||
ecs_data_quality_dashboard | ||
elastic_assistant | ||
embeddable_enhanced | ||
encrypted_saved_objects | ||
enterprise_search | ||
entity_manager | ||
event_log | ||
features | ||
fields_metadata | ||
file_upload | ||
fleet | ||
global_search | ||
global_search_bar | ||
global_search_providers | ||
graph | ||
grokdebugger | ||
index_lifecycle_management | ||
index_management | ||
inference | ||
ingest_pipelines | ||
integration_assistant | ||
kubernetes_security | ||
lens | ||
license_api_guard | ||
license_management | ||
licensing | ||
lists | ||
logstash | ||
maps | ||
ml | ||
monitoring | ||
monitoring_collection | ||
notifications | ||
observability_solution | ||
osquery | ||
painless_lab | ||
remote_clusters | ||
reporting | ||
rollup | ||
rule_registry | ||
runtime_fields | ||
saved_objects_tagging | ||
screenshotting | ||
search_assistant | ||
search_connectors | ||
search_homepage | ||
search_indices | ||
search_inference_endpoints | ||
search_notebooks | ||
search_playground | ||
searchprofiler | ||
security | ||
security_solution | ||
security_solution_ess | ||
security_solution_serverless | ||
serverless | ||
serverless_observability | ||
serverless_search | ||
session_view | ||
snapshot_restore | ||
spaces | ||
stack_alerts | ||
stack_connectors | ||
task_manager | ||
telemetry_collection_xpack | ||
threat_intelligence | ||
timelines | ||
transform | ||
translations | ||
triggers_actions_ui | ||
upgrade_assistant | ||
watcher |