mirror of
https://github.com/elastic/kibana.git
synced 2025-04-24 01:38:56 -04:00
2933ad6278
# Backport This will backport the following commits from `main` to `8.x`: - [[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist (#193205)](https://github.com/elastic/kibana/pull/193205) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Ying Mao","email":"ying.mao@elastic.co"},"sourceCommit":{"committedDate":"2024-09-20T13:55:48Z","message":"[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist (#193205)\n\nResolves https://github.com/elastic/kibana/issues/192682\r\n\r\n## Summary\r\n\r\nAs of 8.8, we started writing all event log documents to the\r\n`.kibana-event-log-ds` index. Prior to this, we created a new index\r\ntemplate and data stream for every version (`.kibana-event-log-8.7` for\r\nexample) so any mapping updates that were added for the version were\r\ncreated in the new index on upgrade.\r\n\r\nWith the static index name and serverless, we need a way to update\r\nmappings in existing indices. This PR uses the same mechanism that we\r\nuse for the alerts index to update the index template mappings and the\r\nmappings for the concrete backing indices of a datastream.\r\n\r\n## To Verify\r\n\r\nRun ES and Kibana in `main` to test the upgrade path for serverless \r\na. Check out `main`, run ES: `yarn es snapshot --license trial --ssl -E\r\npath.data=../test_el_upgrade` and Kibana `yarn start --ssl`\r\n b. Create a rule and let it run to populate the event log index\r\nc. Switch to this PR branch. Make a mapping update to the event log\r\nindex:\r\n\r\n```\r\n--- a/x-pack/plugins/event_log/generated/mappings.json\r\n+++ b/x-pack/plugins/event_log/generated/mappings.json\r\n@@ -172,6 +172,9 @@\r\n },\r\n \"rule\": {\r\n \"properties\": {\r\n+ \"test\": {\r\n+ \"type\": \"keyword\"\r\n+ },\r\n \"author\": {\r\n \"ignore_above\": 1024,\r\n \"type\": \"keyword\",\r\n```\r\n d. Start ES and Kibana with the same commands as above\r\ne. Verify that the `.kibana-event-log-ds` index is created and has the\r\nupdated mapping:\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/templates/.kibana-event-log-template\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/indices/index_details?indexName=.ds-.kibana-event-log-ds-2024.09.17-000001&filter=.kibana-&includeHiddenIndices=true&tab=mappings\r\n\r\nI also verified the following:\r\n1. Run ES and Kibana in 8.7 to test the upgrade path from 8.7 (when\r\nevent log indices were versioned) to now\r\n2. Run ES and Kibana in 8.15 to test the upgrade path from the previous\r\nrelease to now\r\n\r\nHowever, I had to create an 8.x branch and cherry pick this commit\r\nbecause `main` is now on 9.0 and we can't upgrade directly from older\r\n8.x version to 9.0!\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"e2798def07d50595806748dd64cccaa216c5e234","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:ResponseOps","v9.0.0","Feature:EventLog","backport:prev-minor","v8.16.0"],"title":"[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist","number":193205,"url":"https://github.com/elastic/kibana/pull/193205","mergeCommit":{"message":"[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist (#193205)\n\nResolves https://github.com/elastic/kibana/issues/192682\r\n\r\n## Summary\r\n\r\nAs of 8.8, we started writing all event log documents to the\r\n`.kibana-event-log-ds` index. Prior to this, we created a new index\r\ntemplate and data stream for every version (`.kibana-event-log-8.7` for\r\nexample) so any mapping updates that were added for the version were\r\ncreated in the new index on upgrade.\r\n\r\nWith the static index name and serverless, we need a way to update\r\nmappings in existing indices. This PR uses the same mechanism that we\r\nuse for the alerts index to update the index template mappings and the\r\nmappings for the concrete backing indices of a datastream.\r\n\r\n## To Verify\r\n\r\nRun ES and Kibana in `main` to test the upgrade path for serverless \r\na. Check out `main`, run ES: `yarn es snapshot --license trial --ssl -E\r\npath.data=../test_el_upgrade` and Kibana `yarn start --ssl`\r\n b. Create a rule and let it run to populate the event log index\r\nc. Switch to this PR branch. Make a mapping update to the event log\r\nindex:\r\n\r\n```\r\n--- a/x-pack/plugins/event_log/generated/mappings.json\r\n+++ b/x-pack/plugins/event_log/generated/mappings.json\r\n@@ -172,6 +172,9 @@\r\n },\r\n \"rule\": {\r\n \"properties\": {\r\n+ \"test\": {\r\n+ \"type\": \"keyword\"\r\n+ },\r\n \"author\": {\r\n \"ignore_above\": 1024,\r\n \"type\": \"keyword\",\r\n```\r\n d. Start ES and Kibana with the same commands as above\r\ne. Verify that the `.kibana-event-log-ds` index is created and has the\r\nupdated mapping:\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/templates/.kibana-event-log-template\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/indices/index_details?indexName=.ds-.kibana-event-log-ds-2024.09.17-000001&filter=.kibana-&includeHiddenIndices=true&tab=mappings\r\n\r\nI also verified the following:\r\n1. Run ES and Kibana in 8.7 to test the upgrade path from 8.7 (when\r\nevent log indices were versioned) to now\r\n2. Run ES and Kibana in 8.15 to test the upgrade path from the previous\r\nrelease to now\r\n\r\nHowever, I had to create an 8.x branch and cherry pick this commit\r\nbecause `main` is now on 9.0 and we can't upgrade directly from older\r\n8.x version to 9.0!\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"e2798def07d50595806748dd64cccaa216c5e234"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/193205","number":193205,"mergeCommit":{"message":"[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist (#193205)\n\nResolves https://github.com/elastic/kibana/issues/192682\r\n\r\n## Summary\r\n\r\nAs of 8.8, we started writing all event log documents to the\r\n`.kibana-event-log-ds` index. Prior to this, we created a new index\r\ntemplate and data stream for every version (`.kibana-event-log-8.7` for\r\nexample) so any mapping updates that were added for the version were\r\ncreated in the new index on upgrade.\r\n\r\nWith the static index name and serverless, we need a way to update\r\nmappings in existing indices. This PR uses the same mechanism that we\r\nuse for the alerts index to update the index template mappings and the\r\nmappings for the concrete backing indices of a datastream.\r\n\r\n## To Verify\r\n\r\nRun ES and Kibana in `main` to test the upgrade path for serverless \r\na. Check out `main`, run ES: `yarn es snapshot --license trial --ssl -E\r\npath.data=../test_el_upgrade` and Kibana `yarn start --ssl`\r\n b. Create a rule and let it run to populate the event log index\r\nc. Switch to this PR branch. Make a mapping update to the event log\r\nindex:\r\n\r\n```\r\n--- a/x-pack/plugins/event_log/generated/mappings.json\r\n+++ b/x-pack/plugins/event_log/generated/mappings.json\r\n@@ -172,6 +172,9 @@\r\n },\r\n \"rule\": {\r\n \"properties\": {\r\n+ \"test\": {\r\n+ \"type\": \"keyword\"\r\n+ },\r\n \"author\": {\r\n \"ignore_above\": 1024,\r\n \"type\": \"keyword\",\r\n```\r\n d. Start ES and Kibana with the same commands as above\r\ne. Verify that the `.kibana-event-log-ds` index is created and has the\r\nupdated mapping:\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/templates/.kibana-event-log-template\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/indices/index_details?indexName=.ds-.kibana-event-log-ds-2024.09.17-000001&filter=.kibana-&includeHiddenIndices=true&tab=mappings\r\n\r\nI also verified the following:\r\n1. Run ES and Kibana in 8.7 to test the upgrade path from 8.7 (when\r\nevent log indices were versioned) to now\r\n2. Run ES and Kibana in 8.15 to test the upgrade path from the previous\r\nrelease to now\r\n\r\nHowever, I had to create an 8.x branch and cherry pick this commit\r\nbecause `main` is now on 9.0 and we can't upgrade directly from older\r\n8.x version to 9.0!\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"e2798def07d50595806748dd64cccaa216c5e234"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Ying Mao <ying.mao@elastic.co> |
||
|---|---|---|
| .. | ||
| es | ||
| integration_tests | ||
| lib | ||
| event_log_client.mock.ts | ||
| event_log_client.test.ts | ||
| event_log_client.ts | ||
| event_log_service.mock.ts | ||
| event_log_service.test.ts | ||
| event_log_service.ts | ||
| event_log_start_service.mock.ts | ||
| event_log_start_service.test.ts | ||
| event_log_start_service.ts | ||
| event_logger.mock.ts | ||
| event_logger.test.ts | ||
| event_logger.ts | ||
| index.ts | ||
| mocks.ts | ||
| plugin.test.ts | ||
| plugin.ts | ||
| saved_object_provider_registry.mock.ts | ||
| saved_object_provider_registry.test.ts | ||
| saved_object_provider_registry.ts | ||
| types.ts | ||