mirror of
https://github.com/elastic/kibana.git
synced 2025-04-20 16:03:20 -04:00
53c706932e
# Backport This will backport the following commits from `main` to `8.15`: - [[BK] Migrate es-forward (+add versions.json dependent triggering) (#184018)](https://github.com/elastic/kibana/pull/184018) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Alex Szabo","email":"alex.szabo@elastic.co"},"sourceCommit":{"committedDate":"2024-07-10T10:08:16Z","message":"[BK] Migrate es-forward (+add versions.json dependent triggering) (#184018)\n\n## Goal\r\nWe'd like to introduce a way to run pipelines that have a dependency on\r\nthe currently active branch set (managed in\r\n[versions.json](./versions.json)).\r\n\r\nWith this, we'd like to migrate over the `es-forward` pipelines\r\n(currently:\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility),\r\nand\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-14-forward-compatibility))\r\nto the new buildkite infra.\r\n\r\n## Summary\r\nThis PR introduces a new pipeline:\r\nhttps://buildkite.com/elastic/kibana-trigger-version-dependent-jobs\r\n(through\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml)).\r\n\r\nThe purpose of this new pipeline is to take the name of a \"pipelineSet\"\r\nthat refers to a pipeline, and based on the `versions.json` file, work\r\nout what are the branches on which the referred pipeline should be\r\ntriggered.\r\n\r\n### Example: `Trigger ES forward compatibility tests`\r\n- a scheduled run on\r\n[kibana-trigger-version-dependent-jobs](https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs)\r\nwith the env var `TRIGGER_PIPELINE_SET=es-forward` runs\r\n- the pipeline implementation for\r\n`kibana-trigger-version-dependent-jobs` works out (looking at\r\n`versions.json`), that the `es-forward` set should trigger\r\nhttps://buildkite.com/elastic/kibana-es-forward (doesn't exist prior to\r\nthe PR) for (7.17+8.14) and (7.17+8.15)\r\n- the pipeline implementation uploads two trigger steps, running\r\nhttps://buildkite.com/elastic/kibana-es-forward in two instances with\r\nthe relevant parameterization.\r\n\r\nSince the trigger parameters are derived from the `versions.json` file,\r\nif we move on and close `8.14`, and open up `8.16`, this will follow,\r\nwithout having to update the pipeline resources or schedules.\r\n\r\n## Changes\r\n- 2 pipelines created:\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml),\r\n[kibana-es-forward.yml](.buildkite/pipeline-resource-definitions/kibana-es-forward.yml)\r\n - [x] add kibana-es-forward.yml\r\n - implementation for `trigger-version-dependent-jobs` added\r\n- branch configuration removed from pipelines (kibana-artifacts-staging,\r\nkibana-artifacts-snapshot, kibana-artifacts-trigger)\r\n - added a script for checking RREs validity (moved a few files)\r\n\r\n## Verification\r\nI've used the migration staging pipeline (*) to run this:\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/130\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-trigger\"`\r\n- Result:\r\n[(success):](https://buildkite.com/elastic/kibana-artifacts-trigger/builds/10806)\r\nit triggered for 8.14 only (as expected)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/131\r\n - Env: `TRIGGER_PIPELINE_SET=\"es-forward\"`\r\n- Result: (success): it generated 2 trigger steps, but since the\r\nes-forward pipeline doesn't exist, the upload step failed\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/132\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-snapshot\"`\r\n- Result: (success): it triggered jobs for all 3 open branches\r\n(main/8.14/7.17)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/134\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-staging\"`\r\n - Result: (success): it triggered 8.14 / 7.14, but not for main\r\n\r\n\r\n(*note: this migration staging pipeline will come in handy even after\r\nthe migration, to stage newly created pipelines without creating the\r\nresource up-front)","sha":"ea2509914f2238c646bb63eb5bbbb240365fd04d","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Operations","release_note:skip","backport:prev-minor","v8.15.0","v8.16.0"],"number":184018,"url":"https://github.com/elastic/kibana/pull/184018","mergeCommit":{"message":"[BK] Migrate es-forward (+add versions.json dependent triggering) (#184018)\n\n## Goal\r\nWe'd like to introduce a way to run pipelines that have a dependency on\r\nthe currently active branch set (managed in\r\n[versions.json](./versions.json)).\r\n\r\nWith this, we'd like to migrate over the `es-forward` pipelines\r\n(currently:\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility),\r\nand\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-14-forward-compatibility))\r\nto the new buildkite infra.\r\n\r\n## Summary\r\nThis PR introduces a new pipeline:\r\nhttps://buildkite.com/elastic/kibana-trigger-version-dependent-jobs\r\n(through\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml)).\r\n\r\nThe purpose of this new pipeline is to take the name of a \"pipelineSet\"\r\nthat refers to a pipeline, and based on the `versions.json` file, work\r\nout what are the branches on which the referred pipeline should be\r\ntriggered.\r\n\r\n### Example: `Trigger ES forward compatibility tests`\r\n- a scheduled run on\r\n[kibana-trigger-version-dependent-jobs](https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs)\r\nwith the env var `TRIGGER_PIPELINE_SET=es-forward` runs\r\n- the pipeline implementation for\r\n`kibana-trigger-version-dependent-jobs` works out (looking at\r\n`versions.json`), that the `es-forward` set should trigger\r\nhttps://buildkite.com/elastic/kibana-es-forward (doesn't exist prior to\r\nthe PR) for (7.17+8.14) and (7.17+8.15)\r\n- the pipeline implementation uploads two trigger steps, running\r\nhttps://buildkite.com/elastic/kibana-es-forward in two instances with\r\nthe relevant parameterization.\r\n\r\nSince the trigger parameters are derived from the `versions.json` file,\r\nif we move on and close `8.14`, and open up `8.16`, this will follow,\r\nwithout having to update the pipeline resources or schedules.\r\n\r\n## Changes\r\n- 2 pipelines created:\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml),\r\n[kibana-es-forward.yml](.buildkite/pipeline-resource-definitions/kibana-es-forward.yml)\r\n - [x] add kibana-es-forward.yml\r\n - implementation for `trigger-version-dependent-jobs` added\r\n- branch configuration removed from pipelines (kibana-artifacts-staging,\r\nkibana-artifacts-snapshot, kibana-artifacts-trigger)\r\n - added a script for checking RREs validity (moved a few files)\r\n\r\n## Verification\r\nI've used the migration staging pipeline (*) to run this:\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/130\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-trigger\"`\r\n- Result:\r\n[(success):](https://buildkite.com/elastic/kibana-artifacts-trigger/builds/10806)\r\nit triggered for 8.14 only (as expected)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/131\r\n - Env: `TRIGGER_PIPELINE_SET=\"es-forward\"`\r\n- Result: (success): it generated 2 trigger steps, but since the\r\nes-forward pipeline doesn't exist, the upload step failed\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/132\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-snapshot\"`\r\n- Result: (success): it triggered jobs for all 3 open branches\r\n(main/8.14/7.17)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/134\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-staging\"`\r\n - Result: (success): it triggered 8.14 / 7.14, but not for main\r\n\r\n\r\n(*note: this migration staging pipeline will come in handy even after\r\nthe migration, to stage newly created pipelines without creating the\r\nresource up-front)","sha":"ea2509914f2238c646bb63eb5bbbb240365fd04d"}},"sourceBranch":"main","suggestedTargetBranches":["8.15"],"targetPullRequestStates":[{"branch":"8.15","label":"v8.15.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.16.0","labelRegex":"^v8.16.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/184018","number":184018,"mergeCommit":{"message":"[BK] Migrate es-forward (+add versions.json dependent triggering) (#184018)\n\n## Goal\r\nWe'd like to introduce a way to run pipelines that have a dependency on\r\nthe currently active branch set (managed in\r\n[versions.json](./versions.json)).\r\n\r\nWith this, we'd like to migrate over the `es-forward` pipelines\r\n(currently:\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility),\r\nand\r\n[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-14-forward-compatibility))\r\nto the new buildkite infra.\r\n\r\n## Summary\r\nThis PR introduces a new pipeline:\r\nhttps://buildkite.com/elastic/kibana-trigger-version-dependent-jobs\r\n(through\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml)).\r\n\r\nThe purpose of this new pipeline is to take the name of a \"pipelineSet\"\r\nthat refers to a pipeline, and based on the `versions.json` file, work\r\nout what are the branches on which the referred pipeline should be\r\ntriggered.\r\n\r\n### Example: `Trigger ES forward compatibility tests`\r\n- a scheduled run on\r\n[kibana-trigger-version-dependent-jobs](https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs)\r\nwith the env var `TRIGGER_PIPELINE_SET=es-forward` runs\r\n- the pipeline implementation for\r\n`kibana-trigger-version-dependent-jobs` works out (looking at\r\n`versions.json`), that the `es-forward` set should trigger\r\nhttps://buildkite.com/elastic/kibana-es-forward (doesn't exist prior to\r\nthe PR) for (7.17+8.14) and (7.17+8.15)\r\n- the pipeline implementation uploads two trigger steps, running\r\nhttps://buildkite.com/elastic/kibana-es-forward in two instances with\r\nthe relevant parameterization.\r\n\r\nSince the trigger parameters are derived from the `versions.json` file,\r\nif we move on and close `8.14`, and open up `8.16`, this will follow,\r\nwithout having to update the pipeline resources or schedules.\r\n\r\n## Changes\r\n- 2 pipelines created:\r\n[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml),\r\n[kibana-es-forward.yml](.buildkite/pipeline-resource-definitions/kibana-es-forward.yml)\r\n - [x] add kibana-es-forward.yml\r\n - implementation for `trigger-version-dependent-jobs` added\r\n- branch configuration removed from pipelines (kibana-artifacts-staging,\r\nkibana-artifacts-snapshot, kibana-artifacts-trigger)\r\n - added a script for checking RREs validity (moved a few files)\r\n\r\n## Verification\r\nI've used the migration staging pipeline (*) to run this:\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/130\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-trigger\"`\r\n- Result:\r\n[(success):](https://buildkite.com/elastic/kibana-artifacts-trigger/builds/10806)\r\nit triggered for 8.14 only (as expected)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/131\r\n - Env: `TRIGGER_PIPELINE_SET=\"es-forward\"`\r\n- Result: (success): it generated 2 trigger steps, but since the\r\nes-forward pipeline doesn't exist, the upload step failed\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/132\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-snapshot\"`\r\n- Result: (success): it triggered jobs for all 3 open branches\r\n(main/8.14/7.17)\r\n-\r\nhttps://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/134\r\n - Env: `TRIGGER_PIPELINE_SET=\"artifacts-staging\"`\r\n - Result: (success): it triggered 8.14 / 7.14, but not for main\r\n\r\n\r\n(*note: this migration staging pipeline will come in handy even after\r\nthe migration, to stage newly created pipelines without creating the\r\nresource up-front)","sha":"ea2509914f2238c646bb63eb5bbbb240365fd04d"}}]}] BACKPORT--> |
||
|---|---|---|
| .. | ||
| hooks | ||
| pipeline-utils | ||
| pipelines | ||
| scripts | ||
| .mocharc.json | ||
| .npmrc | ||
| disabled_jest_configs.json | ||
| ftr_configs.yml | ||
| package-lock.json | ||
| package.json | ||
| pull_requests.json | ||
| README.md | ||
| tsconfig.json | ||
| tsconfig.test.json | ||
Kibana / Buildkite
Directory Structure
hooks- special directory used by Buildkite agents for hookspipeline-utils- Shared TypeScript utils for use in pipeline scriptspipelines- contains pipeline definitionsscripts/common- scripts that getsourced by other scripts to set environment variables or import shared functionsscripts/lifecycle- general scripts for tasks that run before or after individual steps or the entire buildscripts/steps- scripts that define something that will run for a step defined in a pipelinescripts/*- all other scripts are building blocks that make up the tasks in pipelines. They may be run by other scripts, but should not besourced