= Tandem-based verification
:toc:

This document describes the OpenHW Group functional verification environment based on lockstep ("tandem") simulation of a RISC-V reference model and an RTL implementation of a RISC-V core.

It is divided into two parts:

* a User Manual which discusses the principle, the architecture and the usage of the environment, and outlines future work;
* a Reference Manual which provides a reference of configuration parameters and programming interfaces to support hardware designers, software developers, and verification engineers.

== User Manual

=== Scope of verification

The verification is performed on the CVA6 core, with the exclusion of interrupt controllers, platform-related mechanisms (real-time timer), peripherals etc.

The behaviors that are verified are:

* instruction completion
* general-purpose register value changes
* CSR value changes
* memory operations (as seen by the CVA6 core)
* raising of exceptions
* raising of interrupts
* trap entry and return

=== Basic Architecture and Design Choices

Tandem-based functional verification of an RTL design consists in a lockstep comparison of the behavior of an RTL implementation against the behavior of a reference model (usually an Instruction Set Simulator, or ISS), both executing the same programs from the same initial state.  In opposition to post-execution trace comparison, the tandem approach enables immediate detection of differences in observable behavior of the RTL implementation and the reference model, thus greatly simplifying the investigation of the root causes of any discrepancies.

The OpenHW Group tandem verification environment relies on the use of RVFI (RISC-V Formal Interface, https://github.com/SymbioticEDA/riscv-formal/blob/master/docs/rvfi.md).  RVFI provides a well-defined abstraction of design state at each instruction commit or exception occurrence.

RVFI events are collected from both RTL and reference model side, and the RVFI states of both simulations are compared after every step.

The reference model is a slave to the RTL model; it performs an execution step only when the RTL model has completed the current step.  This enables partial verification of behaviors involving values that depend on micro-architecture features, e.g., clock cycle counts.  The RTL provides the actual values of such registers to the reference model; these values are trusted by the reference model and hence, the reads of CSR registers yield the same results in the reference model as they do in the RTL simulation.

The RTL simulation environment provides the testbench and the required abstractions of platform-level resources and behaviors.  To match the features of the RTL environment, the reference model needs to include a basic platform abstraction that at the very minimum contains a memory model.  For this reason Spike is a prime choice as it contains a superset of the minimum platform needed, cf. xref:fig-spike-platform[xrefstyle=short].

.High-level view of the platform supported by Spike
[#fig-spike-platform]
image::figures/Spike-simulation-scope.svg[High-level view of Spike platform,1024]

In order to ease the integration with RTL simulators, the tandem infrastructure does not depend on the availability of a fully-featured UVM implementation.  Instead, it uses a limited subset of UVM to provide a uniform SystemVerilog interface across all supported RTL simulators, whether commercial or open-source.

==== Operation

The principle of operation of CVA6 tandem verification is shown in xref:fig-overall-tandem-flow[xrefstyle=short].  The execution of the tandem simulation is controlled by the main RTL simulation loop that is managed either by the "run" phase of the UVM testbench, or explicitly in a test harness.  The RTL simulation runs through successive clock cycles until the RVFI state of the RTL model is marked as valid, indicating an instruction commit or an exception in the current RTL clock cycle.

.Overall tandem verification flow
[#fig-overall-tandem-flow]
image::figures/overall-tandem-architecture.svg[Tandem verification flow,1024]

In the CVA6 RTL implementation multiple commits or exceptions can occur in a single cycle.  If multiple simultaneous events are reported by RVFI, the corresponding entries in the RFVI interface are always ordered in the sequential execution order.   For each commit or exception reported by the RVFI interface of the RTL model, the tandem control loop performs a single step of the reference model and compares the RVFI state between the current RTL-side RVFI entry and the reference model RVFI output.  Per-commit entries in the RFVI interface are ordered in the sequential execution order,

Interrupt and debug events detected by the RTL model are injected into the reference model at the beginning of each step. Likewise, the most recent *previously* committed values of microarchitecture-dependent CSRs such as clock cycle counters (`cycle`/`cycleh`, `mcycle`/`mcycleh`) are injected before the reference model step since the RTL values of these registers already correspond to RTL state *after* the corresponding RTL event.

//.Stages of a single reference model step
//[#fig-refmodel-step]
//image::figures/refmodel-step-stages.svg[Stages of a reference model step,800,opts=inline]

If a discrepancy in RFVI state is detected, an appropriate human-readable error output is written into the log file and the error count of the simulation is incremented.  The tandem simulation stops once a preset number of errors is encountered.  By controlling the error count limit is it possible to accommodate transient differences between the RTL and reference model states.  A typical example of transient difference occurs when two instructions are committed in a single cycle by a superscalar RTL implementation, causing the counter of retired instructions `minstret` to be atomically incremented by two whereas the reference model will always commit instruction one by one, thus incrementing `minstret` by one each time.

At the end of a tandem simulation a Yaml report file is produced.  It contains information about the test setup (architecture, ISA, test name, etc.) and the list of mismatches found if any.

For the purpose of CVA6 core verification the core-level interrupt controller (CLIC) is **not** modeled; instead, interrupt management is performed by the testbench.  The RTL core model receives interrupt notifications from the testbench through the available interrupt input ports.  When the RTL model of the core detects an interrupt, the information about the interrupt is passed to the reference model by reconstructing the value of the `mip` (Machine Interrupt Pending) CSR from the value of the `mcause` register to ensure that the interrupt taken is the same as the one already taken by the RTL simulation.  The actual RTL value of the `mip` register may contain additional bits which are set; it will be injected into the reference model state at the end of the reference model step.

==== Implementation

The RTL testbenches use DPI functions to control Spike stepping, inject CSR values and query Spike's RVFI interface.

The selection of single/tandem operation mode is achieved using the SystemVerilog macro `SPIKE_TANDEM`.  If defined to a non-zero ("true") value, it enables the tandem configuration in the RTL simulation by instantiating and connecting the Spike wrapper, and activates the stepping and RVFI compare functions.

The selection of file locations for the Yaml configuration and the Yaml report file locations is performed using dedicated "plusarg" command-line options added to the testbenches:

* `+config_file=<NAME>` sets the name of the Yaml configuration file to `<NAME>`
* `+report_file=<NAME>` sets the name of the Yaml tandem report file to `<NAME>`.

Option `+config_file` should only be used when an appropriate configuration file exists, and should be omitted otherwise.

SystemVerilog code of the tandem infrastructure is maintained in OpenHW Group repositiory `core-v-verif` (https://github.com/openhwgroup/core-v-verif) which also contains the UVM components and agents that leverage the pure SystemVerilog layer.

A pure SystemVerilog tandem testbench for CVA6 is available in the OpenHW Group CVA6 repository (https://github.com/openhwgroup/cva6/).

To limit the impact on the Spike original code base, Spike modifications rely on "shim" (adapter) classes that add extra functionality in two ways:

* by adding a new base class below the original base class (used for CSRs)
* by specializing a class to provide additional interfaces (Processor and Simulation)

Spike execution model was modified to run a single-instruction step function instead of the performance-optimized "instruction batch" mode that executes hundreds to thousands of instructions without yielding control to the environment of the simulated design.

In order to provide fine-grained control over the configuration of the simulated design, Spike was extended with a parameter mechanism with the following characteristics:

* multiple ways of controlling Spike simulator behavior:
** configuration files (Yaml)
** DPI API
** C++ API
** additional command-line options
* ability to set platform-related values (memory map, boot addresses)
* ability to set implementation-defined values (number of PMP regions in total, vendor ID, architecture ID, implementation ID, ...)
* extended CSR control:
** availability or not of any CSR
** CSR initialization masks and values
** CSR write-enable masks with bit granularity
* ability to inject CSR values from the RTL model for architecture-dependent CSRs (`cycles`/`cyclesh`, `mcycles`/`mcyclesh`, `mip` etc.)

Spike modifications are maintained in a forked Spike tree integrated into the OpenHW `core-v-verif` repository (https://github.com/openhwgroup/core-v-verif/).

=== Fundamental limitations of the tandem verification approach

* By definition, a reference model based on an instruction set simulator has only the notion of instruction commits and is not capable of representing the behaviors between commits (transient signals, micro-architectural artefacts such as stalls etc.)
+
Such behaviors must be verified by other means.

* Simultaneous occurrence of the same error in both reference model and RTL implementation will not be detected since the two behaviors will match.
+
This stresses the importance of validation of the reference model against the specification.

* Since the reference model for tandem verification does not include an interrupt controller, the presence of pending interrupts must be notified to the `mip` register by other means.
+
In the OpenHW Group solution, the presence of the interrupt is detected by the RTL model and the information about the actual interrupt raised as the result of interrupt arbitration is passed to the reference model in the `mcause` register.  The value in `mcause` is then used to trigger the corresponding event in the reference model.

* Transient discrepancies can be caused by differences between purely sequential and concurrent model execution.
+
Spike executes instructions in sequence whereas a superscalar architecture may commit multiple instructions in a single cycle; this means that some CSR updates may differ between Spike and RTL, typically MINSTRET may advance by 2 or more in the RTL model, yet Spike will increment it by 1 at every instruction committed.

=== QuickStart Guide

==== Build instructions

The standard build procedure of Spike supplied with the CVA6 project (https://github.com/openhwgroup/cva6/) builds a Spike binary with all necessary modifications required for DPI interfacing to the RTL testbench.

Building Spike with tandem extensions requires the availability of `CMake` in version 3.16 or higher.  The `yaml-cpp` source package (https://github.com/jbeder/yaml-cpp) that is used to parse and manipulate Yaml configurations is a submodule of Spike and will be downloaded when performing a recursive clone or update of the CVA6 repository.

If a rebuild is required in an existing setup, it is necessary to either select a different installation directory for Spike or to remove the current installation directory of Spike.  It is also necessary to remove Spike build directories `verif/core-v-verif/vendor/riscv/riscv-isa-sim/build` (Spike proper) and `verif/core-v-verif/vendor/riscv/riscv-isa-sim/yaml-cpp/build` (`yaml-cpp` library).

==== Understanding the Spike configuration file

Spike configuration files represent a Spike parameter tree in the form of nested Yaml dictionaries under a common root node `spike_param_tree`:

[,yaml]
----
  spike_param_tree:
    bootrom_enable: true
    bootrom_base: 0x10000
    bootrom_size: 0x1000
    dram_enable: true
    dram_base: 0x80000000
    dram_size: 0x40000000
    core_configs:
      -
        isa: rv32imczicsr_zcb_zba_zbb_zbc_zbs
        priv: M
        extensions: cv32a60x,cvxif
        boot_addr: 0x80000000
        marchid_override_mask: 0xFFFFFFFF
        marchid_override_value: 0x3
        misa_write_mask: 0x0
----

The first-level entries define platform properties (`bootrom_*` and `dram_*`) and the default settings for all cores of the system (`core_configs`).

The configuration fragment above instructs the reference model that:

* the platform contains a boot ROM of size 4 KiB starting at address 0x10000 (0+64 KiB);
* the platform contains a DRAM memory of 1 GiB starting at the address 0x80000000 (0+2 GiB)
* all cores:
** implement an RV32IMC ISA with extensions `Zicsr`, `Zcb`, `Zba`, `Zbb`, `Zbc` and `Zbs`;
** support only the Machine privilege level;
** implement additional features modeled in Spike custom extensions `cv32a60x` (additional CSRs specific to `CV32A6*X` family of cores) and `cvxif` (the CV-X-IF interface);
** boot from address `0x80000000` (the start address of the DRAM memory);
** force the reset value of register `marchid` to value `0x3` (corresponding to the CVA6 architecture);
** discard writes into `misa` CSR register by marking all its bits as non-mutable.

==== Running simulations in tandem mode

Currently, tandem simulations are supported out-of-the-box only for CVA6 target `cv32a65x` using RTL simulator configurations `vcs-uvm` (recommended), `vcs-testharness`, `questa-uvm` and `questa-testharness`.  Tandem simulations of target `cv32a65x` with Verilator using the `veri-testharness` configuration require Verilator v5.016 or higher.

Tandem verification is enabled by setting environment variable `SPIKE_TANDEM` to a non-empty value prior to invoking any of the test scripts located in the `verif/regress` directory of the CVA6 source tree.

**Example:**
[,sh]
----
export SPIKE_TANDEM=1 ; bash verif/regress/dv-csr-embedded-tests.sh
----

The output of tandem simulation is sent to the log file of the RTL simulator, by default `verif/sim/out_<date>/<simconfig>_sim/<testname>.<target>.log.iss`.

A machine-processable Yaml report file summarizing the mismatches detected during the simulation is stored in a separate report file, by default `verif/sim/out_<date>/<simconfig>_sim/<testname>.<target>.log.yaml`.  It contains the configuration of the simulated design, the number of mismatches that were found (if any) and a list of entries describing the successive mismatches.

=== How to analyze tandem verification results

To simplify failure analysis, the RVFI state comparator reports mismatches between commit states of the reference model and the RTL model at all verbosity levels.  However, it produces no output if the commit states are identical and the verbosioty level is `UVM_LOW`.  By controlling output verbosity, the simulation log can be reduced to the sole list of state discrepancies, or it can contain a complete log of commits and exceptions to provide additional context at failure locations.

A scoreboard discrepancy is represented by a line describing the nature of the error, followed by two lines that provide a summary view of the RVFI state of the reference model and the RTL core.

**Example:**
[,text]
----
CSR 304  Mismatch   [REF]: 0x0                [CORE]: 0x8
UVM_INFO @ 45992.000 ns : uvmc_rvfi_scoreboard_utils.sv(206) reporter [spike_tandem] 45992.000 ns  | RVFI |        0 |      0 | 800036ac | 30419073 | M | x3  | 00000008 | x0  | 0000000000000000 | csrrw   zero, mie, gp
UVM_ERROR @ 45992.000 ns : uvmc_rvfi_scoreboard_utils.sv(211) reporter [spike_tandem] 45992.000 ns | RVFI |        0 |      0 | 800036ac | 30419073 | M | x3  | 00000008 | x0  | 0000000000000000 | csrrw   zero, mie, gp <- CORE
----

The first line indicates that for current instruction commit the content of the CSR with hexadecimal address `304`, i.e., `mie` ("Machine Interrupt Enable") differs between the reference model (value `0x0`) and the RTL core model (value `0x8`).  The next two lines provide details about the current instruction in both models:

* the current time stamp: `45992.000 ns`,
* the value of PC: `0x800036ac`,
* the encoding of the instruction: hex `30419073`,
* the current privilege mode: `M` (Machine),
* the values of general-purpose register operands of the instruction: `x3` (`gp` in RISC-V ABI) equals `0x00000008`, `x0` equals `0x00000000`.

Since the values stored into the `mie` CSR differ whereas the instruction executed is the same including the values of its input operands, the discrepancy can only come from the write behavior of the `mie` CSR.  In this specific case, bit 3 of the `mie` register is specified as "read-only zero" in the reference model, yet the RTL implementation accepts writes into this specific bit of `mie`.

The following is the corresponding fragment from the Yaml report, including the configuration and status summary of the simulation run:

[,yaml]
----
csrs_match_count: 461
exit_cause: MISMATCH
exit_code: 0
instr_count: 3280
isa: rv32imc_zba_zbb_zbs_zbc_zicsr_zifencei
mismatch_description: 'CSR 304  Mismatch   [REF]: 0x880              [CORE]: 0x888             '
mismatches:
- 0: null
  core:
    insn: 0000000030419073
    insn_disasm: csrrw   zero, mie, gp
    mode: 3
    pc_rdata: 00000000800036ac
    pc_wdata: 0
    rd1_addr: 0
    rd1_rdata: 0
    rs1_addr: 3
    rs1_rdata: 0000000000000008
    trap: 0
  reference_model:
    insn: 0000000030419073
    insn_disasm: csrrw   zero, mie, gp
    mode: 3
    pc_rdata: 00000000800036ac
    pc_wdata: 0
    rd1_addr: 0
    rd1_rdata: 0
    rs1_addr: 3
    rs1_rdata: 0000000000000008
    trap: 0
[...]
mismatches_count: 5
simulator: vcs-uvm
target: cv32a65x
test: csr_test
testlist: csr_embedded
----

=== Example uses of Yaml reference model parameters

==== Enable (make "accessible") or disable (make "inaccessible") a CSR

A Boolean parameter consisting of CSR name and suffix `_accessible`, placed in generic or per-core configuration, indicates whether the given register should be accessible or not.

**Example**:

[,yaml]
----
spike_param_tree:
  ...
  core_configs:
    -
      ...
      tinfo_accessible: False
----

makes the `tinfo` CSR inaccessible in the reference model for all cores, meaning that any attempt to access (read or write) the `tinfo` register when executing an instruction in the reference model will trigger an illegal instruction exception.

==== Force specific bits in a CSR to a constant value

The reference model parameters provide two levels of control over the content of CSRs:

* override the reset value of selected bits of any CSR;
* define which bits of a CSR cannot be modified.

Assuming that bit 5 of a certain CSR `<mycsr>` should always be set irrespective of the default setting in the reference model, it is necessary to indicate that:

* the reset value of that bit should be 1;
* writes to that bit should be ignored.

The parameters needed to express this property are:

* `<mycsr>_override_mask` which defines which bits of `<mycsr>` should be forced to a specific value at reset,
* `<mycsr>_override_value` which indicates the reset value of the forced bits of `<mycsr>`,
* `<mycsr>_write_mask` which indicates which bits of `<mycsr>` are writable (mask bit is set) and which are not (mask bit is cleared).

Assuming that the XLEN is 32 and only bit 5 of `<mycsr>` should be fixed, the Yaml settings to use are:

[,yaml]
----
spike_param_tree:
  ...
  core_configs:
    -
      ...
      <mycsr>_override_mask: 0x00000020
      <mycsr>_override_value: 0x00000020
      <mycsr>_write_mask: 0xffffffdf
----

=== Integration with the RISC-V Open Source ecosystem

* The entire tandem verification infrastructure is available as Open Source under SHL 2.0+ (an Apache-type license) in OpenHW Group repositories `cva6` (https://github.com/openhwgroup/cva6/) and `core-v-verif` (https://github.com/openhwgroup/cva6/).

* The OpenHW Group tandem verification infrastructure relies on Spike (https://github.com/riscv-software-src/riscv-isa-sim/) as reference model.
+
The tandem verification approach is a general template and is not bound to a single ISS or RTL simulator.  However, it requires that the ISS provides a basic platform capable of running software matching the capabilities of the RTL testbench, including a model of memory and buses.
Because of this constraint the current tandem framework relies on Spike which is fully Open Source and provides all necessary components.

* Verilator-based tandem verification
+
Starting with version 5.016, Verilator supports tandem simulations.  Earlier versions did not provide sufficient support for structured types in the DPI interface.

include::reference.adoc[]
