diff --git a/.gitlab-ci/README.md b/.gitlab-ci/README.md new file mode 100644 index 000000000..8f531175e --- /dev/null +++ b/.gitlab-ci/README.md @@ -0,0 +1,252 @@ + + +# GitLab CI for core-v-verif + CVA6 + +This document describes the different steps performed automatically when a branch is pushed to a repository. +It is not meant to be a complete description. +It is an entry point to help to understand the structure of the pipelines; to find the information your are looking for / the part of the CI you want to edit. +Please refer to the mentioned files for more details. + +Only the GitLab-related tasks are described here. + + +## Before the branch reaches GitLab + +CVA6 and core-v-verif repositories are mirrored into GitLab repositories owned by Thales, to perform regression tests on pull requests. + +> Note: in CVA6 regression tests are also run on the `master` branch, and in core-v-verif on the `cva6/dev` branch. + + +## Pipeline boot + +When a branch is pushed, the entry point of the CI is the `.gitlab-ci.yml` file at the repository root. + +> See [`core-v-verif/.gitlab-ci.yml`] and [`cva6/.gitlab-ci.yml`] + +[`core-v-verif/.gitlab-ci.yml`]: https://github.com/openhwgroup/core-v-verif/blob/cva6/dev/.gitlab-ci.yml +[`cva6/.gitlab-ci.yml`]: https://github.com/openhwgroup/cva6/blob/master/.gitlab-ci.yml + +Both source files from a `setup-ci` project (to locate tools etc.), define workflow rules and perform a small environment check. + +All pipelines need both CVA6 and core-v-verif to run tests. +By default the branches used are: + +- The one from the PR +- The main branch from the other repository. + The main branch is defined in `setup-ci` (`master` for CVA6 and `cva6/dev` for core-v-verif). + +However, the entry points also detect the `cvvdev/*` pattern in the branch name to run CVA6 and core-v-verif pipelines on branches with the same name. +It is useful to consistently test PRs impacting both repositories. + +In the CVA6 pipeline: + +1. The `core-v-verif-build` job gets the current commit hash of core-v-verif to set it as an environment variable. + It gets the list of tests to run [`core-v-verif/.gitlab-ci/cva6.yml`] (see next steps). +2. The `core-v-verif` job triggers a child pipeline using: + - [`core-v-verif/.gitlab-ci/cva6.yml`] fetched by `core-v-verif-build` + - [`cva6/.gitlab-ci/core-v-verif-cva6.yml`] which defines a `before_script` and an `after_script` to `cd` the core-v-verif repository with the hash defined by `core-v-verif-build` + +[`core-v-verif/.gitlab-ci/cva6.yml`]: https://github.com/openhwgroup/core-v-verif/blob/cva6/dev/.gitlab-ci/cva6.yml +[`cva6/.gitlab-ci/core-v-verif-cva6.yml`]: https://github.com/openhwgroup/cva6/blob/master/.gitlab-ci/core-v-verif-cva6.yml + +In core-v-verif pipelines, the `cva6` job triggers a child pipeline using: + +- [`core-v-verif/.gitlab-ci/cva6.yml`] (the list of tests) +- [`core-v-verif/.gitlab-ci/core-v-verif-cva6.yml`] (global `before_script` and `after_script`). + +[`core-v-verif/.gitlab-ci/core-v-verif-cva6.yml`]: https://github.com/openhwgroup/core-v-verif/blob/cva6/dev/.gitlab-ci/core-v-verif-cva6.yml + + +## Running the tests + +Thanks to the previous step, in pipelines from both CVA6 and core-v-verif, the current working directory is core-v-verif, with CVA6 checked out in `core-v-cores/cva6`. + +The tests are described in [`core-v-verif/.gitlab-ci/cva6.yml`]. + +Stages are defined as below (order matters): + +- `init env`: only contains `pub_initjob`, which sets a hash for CVA6 as an environment variable, so that it is the same one for all jobs of this pipeline. + It is only run in core-v-verif pipelines as CVA6 pipelines already have the CVA6 commit hash of the pipeline! +- `build tools`: `pub_build_tools` build Spike and `pub_check_env` prints some environment variable for debugging. +- `smoke tests`: `pub_smoke` runs smoke tests. +- `verif tests`: many jobs runs different verif tests. + The template for them is described later in this document. +- `backend tests`: jobs which use results of `verif tests`, often synthesis results. +- `report`: `merge reports` merges all reports into a single yaml. + + +### Adding a verif test + +A simple test looks like this: + +```yml +pub_: + extends: + - .verif_test + - .template_job_short_ci + variables: + DASHBOARD_JOB_TITLE: "" + DASHBOARD_JOB_DESCRIPTION: "<description for dashboard>" + DASHBOARD_SORT_INDEX: <index to sort jobs in dashboard> + DASHBOARD_JOB_CATEGORY: "<job category for dashboard>" + script: + - source cva6/regress/<my-script>.sh + - python3 .gitlab-ci/scripts/report_<kind>.py <args...> +``` + +- `.verif_test` tells that: + - The job goes in `verif tests` stage + - Before running the script part, additionally to the global `before_script`: + - Spike is got from `pub_build_tools` + - Artifacts are cleaned, `artifacts/reports` and `artifacts/logs` are created + - A "failure" report is created by default (in case the script exists early) + - `$SYN_VCS_BASHRC` is sourced + - All the contents of the `artifacts/` folder will be considered as artifacts (even if the job fails) +- `.template_job_short_ci` tells in which pipeline mode the job should run +- `variables` defines environment variables. + The 4 above are needed to generate the report for the dashboard. +- `script` defines the script to run: + 1. Run the test, for instance sourcing a script in `cva6/regress/` + 2. Generate a report running a script from `.gitlab-ci/scripts/reports_*.py` + +> Notes: +> +> You can add more environment variables such as: +> +> ```yml +> variables: +> DV_SIMULATORS: "veri-testharness,spike" +> DV_TESTLISTS: "../tests/testlist_riscv-tests-$DV_TARGET-p.yaml" +> ``` +> +> You can also have several jobs running in parallel with variables taking different values: +> +> ```yml +> parallel: +> matrix: +> - DV_TARGET: [cv64a6_imafdc_sv39, cv32a60x] +> ``` + + +### Adding a backend test + +```yml +pub_<name>: + needs: + - *initjob + - pub_<other_job> + - <...> + extends: + - .backend_test + - .template_job_always_manual + variables: + <same as for verif tests> + script: + - <mv spike from artifacts if you need it> + - <your script> + - python3 .gitlab-ci/scripts/report_<kind>.py <args...> +``` + +Backend tests are like verif tests, differences are: + +- `needs` list is needed to specify in which conditions the test is run (with `.template_job_*`). + It contains: + - `*initjob` to be sure the correct CVA6 commit is used. + Without a `needs` list, all jobs from all previous stages are considered as needed. + However, when a `needs` list is declared, all useful dependencies must be specified by hand, which is more complex. + - `pub_build_tools` if you need spike (don't forget to `mv` it from the artifacts!) + - The jobs you need artifacts from +- `.backend_test` indicates that: + - The job goes in `backend tests` stage + - It performs the same steps than `.backend_test`, except that: + - it does not source VCS (so you have to do it if you need it) + - it does not move spike (so you have to do it if you need it) + + +## Generating a report + +You might want to use `.gitlab-ci/scripts/report_simu.py`. + +If it does not suit your needs, below are snippets to help you write a report generator using our python library. + +```python +import report_builder as rb + +# Create metrics +metric = rb.TableMetric('Metric name') +metric.add_value('colomn 1', 'colomn 2', 'etc') + +# Gather them into a report +report = rb.Report('report label') +report.add_metric(metric) + +# Create the report file in the artifacts +report.dump() +``` + +There are 3 kinds of metric: + +```python +# A simple table +metric = rb.TableMetric('Metric name, actually not displayed yet') +metric.add_value('colomn 1', 'colomn 2', 'etc') + +# A table with a pass/fail label on each line +metric = rb.TableStatusMetric('Metric name, actually not displayed yet') +metric.add_pass('colomn 1', 'colomn 2', 'etc') +metric.add_fail('colomn 1', 'colomn 2', 'etc') + +# A log +metric = rb.LogMetric('Metric name, actually not displayed yet') +metric.add_value("one line (no need to add a backslash n)") +metric.values += ["one line (no need to add a backslash n)"] # same as above +metric.values = ["line1", "line2", "etc"] # also works + +# You can fail a metric of any kind at any moment +metric.fail() +``` + +Failures are propagated: + +- one fail in a `TableStatusMetric` fails the whole metric +- one failed metric fails the whole report +- one failed report fails the whole pipeline report + + +## Dashboard + +The `merge reports` job merges the report from all jobs of the pipeline into a single file. +It pushes this file to a repository. +This repository has a CI which produces HTML dashboard pages from the latest files. +These HTML pages are published on <https://riscv-ci.pages.thales-invia.fr/dashboard/> + +- Main pages [`dashboard_cva6_0.html`] and [`dashboard_core-v-verif_0.html`] gather results from all processed pipelines. +- Each page `dashboard_<project>_<PR id>.html` gathers results from all pipelines of one PR. + +[`dashboard_cva6_0.html`]: https://riscv-ci.pages.thales-invia.fr/dashboard/dashboard_cva6_0.html +[`dashboard_core-v-verif_0.html`]: https://riscv-ci.pages.thales-invia.fr/dashboard/dashboard_core-v-verif_0.html + + +## PR comment + +The `merge reports` job gets the list of open PRs. +It compares the name of the current branch with the name of each PR branch to find the PR. +If a PR matches, it triggers the GitHub workflow `dashboard-done.yml` in this repository, providing the PR number and the success/fail status. + +> See [`core-v-verif/.github/workflows/dashboard-done.yml`] and [`cva6/.github/workflows/dashboard-done.yml`] + +[`core-v-verif/.github/workflows/dashboard-done.yml`]: https://github.com/openhwgroup/core-v-verif/blob/cva6/dev/.github/workflows/dashboard-done.yml +[`cva6/.github/workflows/dashboard-done.yml`]: https://github.com/openhwgroup/cva6/blob/master/.github/workflows/dashboard-done.yml + +This GitHub workflow creates a comment in the PR with the success/fail status and a link to the dashboard page. + +However, the dashboard page may not be available right at this moment, as page generation, performed later, takes time. diff --git a/.gitlab-ci/cva6.yml b/.gitlab-ci/cva6.yml index 358441448..ef56869a2 100644 --- a/.gitlab-ci/cva6.yml +++ b/.gitlab-ci/cva6.yml @@ -14,6 +14,8 @@ # - In this pipeline, do not define before_script and after_script in the global section (avoid in job too). # - Please prefix all jobs in this file with "pub_" which stands for "public" job. +# Please refer to .gitlab-ci/README.md to add jobs + variables: GIT_STRATEGY: fetch