mirror of
https://github.com/elastic/kibana.git
synced 2025-04-24 09:48:58 -04:00
[8.5] [SecuritySolution] Ingest pipelines conflict when upgrading host risk scores (#145232) (#145352)
# Backport This will backport the following commits from `main` to `8.5`: - [[SecuritySolution] Ingest pipelines conflict when upgrading host risk scores (#145232)](https://github.com/elastic/kibana/pull/145232) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Angela Chuang","email":"6295984+angorayc@users.noreply.github.com"},"sourceCommit":{"committedDate":"2022-11-16T12:26:56Z","message":"[SecuritySolution] Ingest pipelines conflict when upgrading host risk scores (#145232)\n\n## Summary\r\n\r\nOriginal issue: https://github.com/elastic/kibana/issues/144916\r\nUsers installed via\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md\r\nand\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/user-risk-score.md\r\ncouldn't upgrade successfully.\r\n\r\n**Fixes**:\r\n1. Remove all the legacy scripts and ingest pipelines with or without\r\nspace name\r\n2. Add version history to\r\nx-pack/plugins/security_solution/server/lib/risk_score/readme.md\r\n<img width=\"1459\" alt=\"Screenshot 2022-11-15 at 13 49 43\"\r\nsrc=\"https://user-images.githubusercontent.com/6295984/201936206-e73ab61c-9a0f-4cfe-8a01-9666217bb863.png\">\r\n\r\n<img width=\"1429\" alt=\"Screenshot 2022-11-15 at 13 53 54\"\r\nsrc=\"https://user-images.githubusercontent.com/6295984/201936751-c3a65f46-1f6e-4b2f-a04a-58f1f32a546f.png\">\r\n\r\n\r\n**Steps to reproduce**:\r\n\r\nOption 1: **Cypress**: Run `upgrade_risk_score.cy.ts`\r\n\r\nOption 2: **Manually**: \r\n1. Follow the steps of\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md\r\nand\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/user-risk-score.md\r\nto install the module.\r\n4. Back to `/app/security/entity_analytics` and click the upgrade\r\nbuttons.\r\n5. Observe if the installation success.\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\nCo-authored-by: Steph Milovic <stephanie.milovic@elastic.co>","sha":"b6693bd9260c1620ec5ad8f09141b534c3b02e81","branchLabelMapping":{"^v8.6.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:skip","Team:Threat Hunting","Team: SecuritySolution","Team:Threat Hunting:Explore","v8.6.0","v8.5.2"],"number":145232,"url":"https://github.com/elastic/kibana/pull/145232","mergeCommit":{"message":"[SecuritySolution] Ingest pipelines conflict when upgrading host risk scores (#145232)\n\n## Summary\r\n\r\nOriginal issue: https://github.com/elastic/kibana/issues/144916\r\nUsers installed via\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md\r\nand\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/user-risk-score.md\r\ncouldn't upgrade successfully.\r\n\r\n**Fixes**:\r\n1. Remove all the legacy scripts and ingest pipelines with or without\r\nspace name\r\n2. Add version history to\r\nx-pack/plugins/security_solution/server/lib/risk_score/readme.md\r\n<img width=\"1459\" alt=\"Screenshot 2022-11-15 at 13 49 43\"\r\nsrc=\"https://user-images.githubusercontent.com/6295984/201936206-e73ab61c-9a0f-4cfe-8a01-9666217bb863.png\">\r\n\r\n<img width=\"1429\" alt=\"Screenshot 2022-11-15 at 13 53 54\"\r\nsrc=\"https://user-images.githubusercontent.com/6295984/201936751-c3a65f46-1f6e-4b2f-a04a-58f1f32a546f.png\">\r\n\r\n\r\n**Steps to reproduce**:\r\n\r\nOption 1: **Cypress**: Run `upgrade_risk_score.cy.ts`\r\n\r\nOption 2: **Manually**: \r\n1. Follow the steps of\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md\r\nand\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/user-risk-score.md\r\nto install the module.\r\n4. Back to `/app/security/entity_analytics` and click the upgrade\r\nbuttons.\r\n5. Observe if the installation success.\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\nCo-authored-by: Steph Milovic <stephanie.milovic@elastic.co>","sha":"b6693bd9260c1620ec5ad8f09141b534c3b02e81"}},"sourceBranch":"main","suggestedTargetBranches":["8.5"],"targetPullRequestStates":[{"branch":"main","label":"v8.6.0","labelRegex":"^v8.6.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/145232","number":145232,"mergeCommit":{"message":"[SecuritySolution] Ingest pipelines conflict when upgrading host risk scores (#145232)\n\n## Summary\r\n\r\nOriginal issue: https://github.com/elastic/kibana/issues/144916\r\nUsers installed via\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md\r\nand\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/user-risk-score.md\r\ncouldn't upgrade successfully.\r\n\r\n**Fixes**:\r\n1. Remove all the legacy scripts and ingest pipelines with or without\r\nspace name\r\n2. Add version history to\r\nx-pack/plugins/security_solution/server/lib/risk_score/readme.md\r\n<img width=\"1459\" alt=\"Screenshot 2022-11-15 at 13 49 43\"\r\nsrc=\"https://user-images.githubusercontent.com/6295984/201936206-e73ab61c-9a0f-4cfe-8a01-9666217bb863.png\">\r\n\r\n<img width=\"1429\" alt=\"Screenshot 2022-11-15 at 13 53 54\"\r\nsrc=\"https://user-images.githubusercontent.com/6295984/201936751-c3a65f46-1f6e-4b2f-a04a-58f1f32a546f.png\">\r\n\r\n\r\n**Steps to reproduce**:\r\n\r\nOption 1: **Cypress**: Run `upgrade_risk_score.cy.ts`\r\n\r\nOption 2: **Manually**: \r\n1. Follow the steps of\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md\r\nand\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/user-risk-score.md\r\nto install the module.\r\n4. Back to `/app/security/entity_analytics` and click the upgrade\r\nbuttons.\r\n5. Observe if the installation success.\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\nCo-authored-by: Steph Milovic <stephanie.milovic@elastic.co>","sha":"b6693bd9260c1620ec5ad8f09141b534c3b02e81"}},{"branch":"8.5","label":"v8.5.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
This commit is contained in:
Angela Chuang
GitHub
parent
c1818d3454
commit
ed806a896e
4 changed files with 51 additions and 13 deletions
|
|
@ -9,7 +9,7 @@ import { DEFAULT_ALERTS_INDEX } from '../constants';
|
|||
import { RiskScoreEntity, RiskScoreFields } from '../search_strategy';
|
||||
|
||||
/**
|
||||
* * Since 8.5, all the transforms, scripts,
|
||||
* Aside from 8.4, all the transforms, scripts,
|
||||
* and ingest pipelines (and dashboard saved objects) are created with spaceId
|
||||
* so they won't affect each other across different spaces.
|
||||
*/
|
||||
|
|
@ -44,7 +44,7 @@ export const getRiskScoreReduceScriptId = (riskScoreEntity: RiskScoreEntity, spa
|
|||
`ml_${riskScoreEntity}riskscore_reduce_script_${spaceId}`;
|
||||
|
||||
/**
|
||||
* These scripts and Ingest pipeline were not space awared before 8.5.
|
||||
* These scripts and Ingest pipeline were not space aware in 8.4
|
||||
* They were shared across spaces and therefore affected each other.
|
||||
* New scripts and ingest pipeline are all independent in each space, so these ids
|
||||
* are Deprecated.
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@
|
|||
import type { HttpSetup } from '@kbn/core/public';
|
||||
import { RiskScoreEntity } from '../../../../../common/search_strategy';
|
||||
import {
|
||||
getIngestPipelineName,
|
||||
getLegacyIngestPipelineName,
|
||||
getRiskScoreLatestTransformId,
|
||||
getRiskScorePivotTransformId,
|
||||
|
|
@ -20,7 +21,7 @@ import * as api from '../../../../risk_score/containers/onboarding/api';
|
|||
import {
|
||||
installRiskScoreModule,
|
||||
restartRiskScoreTransforms,
|
||||
uninstallLegacyRiskScoreModule,
|
||||
uninstallRiskScoreModule,
|
||||
} from './utils';
|
||||
|
||||
jest.mock('../../../../risk_score/containers/onboarding/api');
|
||||
|
|
@ -163,12 +164,12 @@ describe(`installRiskScoreModule - ${RiskScoreEntity.user}`, () => {
|
|||
});
|
||||
|
||||
describe.each([[RiskScoreEntity.host], [RiskScoreEntity.user]])(
|
||||
'uninstallLegacyRiskScoreModule - %s',
|
||||
'uninstallRiskScoreModule - %s',
|
||||
(riskScoreEntity) => {
|
||||
beforeAll(async () => {
|
||||
await uninstallLegacyRiskScoreModule({
|
||||
await uninstallRiskScoreModule({
|
||||
http: mockHttp,
|
||||
spaceId: 'customSpace',
|
||||
spaceId: mockSpaceId,
|
||||
riskScoreEntity,
|
||||
});
|
||||
});
|
||||
|
|
@ -192,7 +193,10 @@ describe.each([[RiskScoreEntity.host], [RiskScoreEntity.user]])(
|
|||
|
||||
it('Delete legacy ingest pipelines', () => {
|
||||
expect((api.deleteIngestPipelines as jest.Mock).mock.calls[0][0].names).toEqual(
|
||||
getLegacyIngestPipelineName(riskScoreEntity)
|
||||
[
|
||||
getLegacyIngestPipelineName(riskScoreEntity),
|
||||
getIngestPipelineName(riskScoreEntity, mockSpaceId),
|
||||
].join(',')
|
||||
);
|
||||
});
|
||||
|
||||
|
|
@ -203,6 +207,9 @@ describe.each([[RiskScoreEntity.host], [RiskScoreEntity.user]])(
|
|||
"ml_userriskscore_levels_script",
|
||||
"ml_userriskscore_map_script",
|
||||
"ml_userriskscore_reduce_script",
|
||||
"ml_userriskscore_levels_script_customSpace",
|
||||
"ml_userriskscore_map_script_customSpace",
|
||||
"ml_userriskscore_reduce_script_customSpace",
|
||||
]
|
||||
`);
|
||||
} else {
|
||||
|
|
@ -212,6 +219,10 @@ describe.each([[RiskScoreEntity.host], [RiskScoreEntity.user]])(
|
|||
"ml_hostriskscore_init_script",
|
||||
"ml_hostriskscore_map_script",
|
||||
"ml_hostriskscore_reduce_script",
|
||||
"ml_hostriskscore_levels_script_customSpace",
|
||||
"ml_hostriskscore_init_script_customSpace",
|
||||
"ml_hostriskscore_map_script_customSpace",
|
||||
"ml_hostriskscore_reduce_script_customSpace",
|
||||
]
|
||||
`);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -406,7 +406,7 @@ export const installRiskScoreModule = async (settings: InstallRiskyScoreModule)
|
|||
}
|
||||
};
|
||||
|
||||
export const uninstallLegacyRiskScoreModule = async ({
|
||||
export const uninstallRiskScoreModule = async ({
|
||||
http,
|
||||
notifications,
|
||||
refetch,
|
||||
|
|
@ -425,22 +425,39 @@ export const uninstallLegacyRiskScoreModule = async ({
|
|||
deleteAll?: boolean;
|
||||
}) => {
|
||||
const legacyTransformIds = [
|
||||
// transform Ids never changed since 8.3
|
||||
utils.getRiskScorePivotTransformId(riskScoreEntity, spaceId),
|
||||
utils.getRiskScoreLatestTransformId(riskScoreEntity, spaceId),
|
||||
];
|
||||
const legacyRiskScoreHostsScriptIds = [
|
||||
// 8.4
|
||||
utils.getLegacyRiskScoreLevelScriptId(RiskScoreEntity.host),
|
||||
utils.getLegacyRiskScoreInitScriptId(RiskScoreEntity.host),
|
||||
utils.getLegacyRiskScoreMapScriptId(RiskScoreEntity.host),
|
||||
utils.getLegacyRiskScoreReduceScriptId(RiskScoreEntity.host),
|
||||
// 8.3 and after 8.5
|
||||
utils.getRiskScoreLevelScriptId(RiskScoreEntity.host, spaceId),
|
||||
utils.getRiskScoreInitScriptId(RiskScoreEntity.host, spaceId),
|
||||
utils.getRiskScoreMapScriptId(RiskScoreEntity.host, spaceId),
|
||||
utils.getRiskScoreReduceScriptId(RiskScoreEntity.host, spaceId),
|
||||
];
|
||||
const legacyRiskScoreUsersScriptIds = [
|
||||
// 8.4
|
||||
utils.getLegacyRiskScoreLevelScriptId(RiskScoreEntity.user),
|
||||
utils.getLegacyRiskScoreMapScriptId(RiskScoreEntity.user),
|
||||
utils.getLegacyRiskScoreReduceScriptId(RiskScoreEntity.user),
|
||||
// 8.3 and after 8.5
|
||||
utils.getRiskScoreLevelScriptId(RiskScoreEntity.user, spaceId),
|
||||
utils.getRiskScoreMapScriptId(RiskScoreEntity.user, spaceId),
|
||||
utils.getRiskScoreReduceScriptId(RiskScoreEntity.user, spaceId),
|
||||
];
|
||||
|
||||
const legacyIngestPipelineNames = [utils.getLegacyIngestPipelineName(riskScoreEntity)];
|
||||
const legacyIngestPipelineNames = [
|
||||
// 8.4
|
||||
utils.getLegacyIngestPipelineName(riskScoreEntity),
|
||||
// 8.3 and 8.5
|
||||
utils.getIngestPipelineName(riskScoreEntity, spaceId),
|
||||
];
|
||||
|
||||
/**
|
||||
* Intended not to pass notification to bulkDeletePrebuiltSavedObjects.
|
||||
|
|
@ -477,7 +494,7 @@ export const uninstallLegacyRiskScoreModule = async ({
|
|||
* Intended not to pass notification to deleteIngestPipelines.
|
||||
* As the only error it can happen is ingest pipeline not found, and
|
||||
* that is what deleteIngestPipelines wants.
|
||||
* (Before 8.5 once an ingest pipeline was created, it was shared across different spaces.
|
||||
* (In 8.4 once an ingest pipeline was created, it was shared across different spaces.
|
||||
* If it has been upgrade in one space, "ingest pipeline not found" will happen when upgrading other spaces.
|
||||
* Or it could be users manually deleted the ingest pipeline.)
|
||||
*/
|
||||
|
|
@ -493,7 +510,7 @@ export const uninstallLegacyRiskScoreModule = async ({
|
|||
* Intended not to pass notification to deleteStoredScripts.
|
||||
* As the only error it can happen is script not found, and
|
||||
* that is what deleteStoredScripts wants.
|
||||
* (Before 8.5 once a script was created, it was shared across different spaces.
|
||||
* (In 8.4 once a script was created, it was shared across different spaces.
|
||||
* If it has been upgrade in one space, "script not found" will happen when upgrading other spaces.
|
||||
* Or it could be users manually deleted the script.)
|
||||
*/
|
||||
|
|
@ -521,7 +538,7 @@ export const upgradeHostRiskScoreModule = async ({
|
|||
theme,
|
||||
timerange,
|
||||
}: UpgradeRiskyScoreModule) => {
|
||||
await uninstallLegacyRiskScoreModule({
|
||||
await uninstallRiskScoreModule({
|
||||
http,
|
||||
notifications,
|
||||
renderDocLink,
|
||||
|
|
@ -554,7 +571,7 @@ export const upgradeUserRiskScoreModule = async ({
|
|||
theme,
|
||||
timerange,
|
||||
}: UpgradeRiskyScoreModule) => {
|
||||
await uninstallLegacyRiskScoreModule({
|
||||
await uninstallRiskScoreModule({
|
||||
http,
|
||||
notifications,
|
||||
renderDocLink,
|
||||
|
|
|
|||
|
|
@ -1,3 +1,13 @@
|
|||
|
||||
# Version
|
||||
|Version|Risk Score Entity|Scripts created|Ingest pipelines created|Transforms created|Behind feature flag|Notes|
|
||||
|-------|------|-------|----------------|----------|----|----|
|
||||
|8.3`deprecated`|host|1.ml_hostriskscore_levels_script_{spacename} 2.ml_hostriskscore_map_script_{spacename} 3.ml_hostriskscore_reduce_script_{spacename} 4.ml_hostriskscore_init_script_{spacename}|ml_hostriskscore_ingest_pipeline_{spacename}|1.ml_hostriskscore_pivot_transform_{spacename} Destination Index: `ml_host_risk_score_{spacename}` 2.ml_hostriskscore_latest_transform_{spacename} Destination Index: `ml_host_risk_score_latest_{spacename}`| Yes|https://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md|
|
||||
|8.3`deprecated`|user|1.ml_userriskscore_levels_script_{spacename} 2.ml_userriskscore_map_script_{spacename} 3.ml_userriskscore_reduce_script_{spacename}|ml_userriskscore_ingest_pipeline_{spacename}|1.ml_userriskscore_pivot_transform_{spacename} Destination index: `ml_user_risk_score_{spacename}` 2.ml_userriskscore_latest_transform_{spacename} Destination index: `ml_user_risk_score_latest_{spacename}`|Yes|https://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/user-risk-score.md|
|
||||
|8.4`deprecated`|host|1.ml_hostriskscore_levels_script 2.ml_hostriskscore_map_script 3.ml_hostriskscore_reduce_script 4.ml_hostriskscore_init_script|ml_hostriskscore_ingest_pipeline|1.ml_hostriskscore_pivot_transform_{spacename} Destination Index: `ml_host_risk_score_{spacename}` 2.ml_hostriskscore_latest_transform_{spacename} Destination Index: `ml_host_risk_score_latest_{spacename}`|Yes|Installation via dev tools releasesd. https://github.com/elastic/kibana/blob/8.4/x-pack/plugins/security_solution/server/lib/prebuilt_dev_tool_content/console_templates/enable_host_risk_score.console|
|
||||
|8.4`deprecated`|user|1.ml_userriskscore_levels_script_{spacename} 2.ml_userriskscore_map_script_{spacename} 3.ml_userriskscore_reduce_script_{spacename}|ml_userriskscore_ingest_pipeline_{spacename}|1.ml_userriskscore_pivot_transform_{spacename} Destination index: `ml_user_risk_score_{spacename}` 2.ml_userriskscore_latest_transform_{spacename} Destination index: `ml_user_risk_score_latest_{spacename}`|Yes|Installation via dev tools not available yet (Installation via dev tools is availble in 8.5).
|
||||
|8.5+|host|1.ml_hostriskscore_levels_script_{spacename} 2.ml_hostriskscore_map_script_{spacename} 3.ml_hostriskscore_reduce_script_{spacename} 4.ml_hostriskscore_init_script_{spacename}|ml_hostriskscore_ingest_pipeline_{spacename}|1.ml_hostriskscore_pivot_transform_{spacename} Destination Index: `ml_host_risk_score_{spacename}` 2.ml_hostriskscore_latest_transform_{spacename} Destination Index: `ml_host_risk_score_latest_{spacename}`| No|`Breaking Chang`: New schema for Destination indices|
|
||||
|8.5+|user|1.ml_userriskscore_levels_script_{spacename} 2.ml_userriskscore_map_script_{spacename} 3.ml_userriskscore_reduce_script_{spacename}|ml_userriskscore_ingest_pipeline_{spacename}|1.ml_userriskscore_pivot_transform_{spacename} Destination index: `ml_user_risk_score_{spacename}` 2.ml_userriskscore_latest_transform_{spacename} Destination index: `ml_user_risk_score_latest_{spacename}`|No|`Breaking Chang`: New schema for Destination indices|
|
||||
# Risk Score API
|
||||
|
||||
### API usage
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue