[doc] Add new Ibex testplan

doc/03_reference/index.rst

@@ -23,6 +23,7 @@ It describes the design in detail, discusses the verification approach and the r
    tracer
    verification
    cosim
+   testplan
    coverage_plan
    rvfi
    history

doc/03_reference/testplan.rst

@@ -0,0 +1,109 @@
+.. _testplan:
+
+.. todo::
+
+  Add detail about security hardening verification.
+
+.. note::
+
+  This testplan is a work in progress still being implemented so this document may not match the implemented verification in the repository.
+
+Test Plan
+=========
+
+Goals
+-----
+
+* Verify compliance with all the RISC-V specifications Ibex supports.
+* Verify Ibex's security hardening features.
+* Ensure correct functionality is maintained across all possible behaviours of external interfaces (interrupts, memory responses, debug requests etc).
+* Hit all functional coverage points, described in :ref:`coverage-plan`.
+
+Testbench Architecture
+----------------------
+
+.. figure:: images/tb2.svg
+    :alt: Testbench Architecture
+
+    Architecture of the UVM testbench for Ibex core
+
+Ibex utilises a co-simulation checking approach described in detail in :ref:`cosim`.
+With the co-simulation system all instructions Ibex executes and all external events such as an interrupts or memory errors are fed to a golden model.
+The results of every instruction execution and every memory access are crossed checked against the golden model with any mismatches resulting in a test failure.
+The aim is to check all possible externally observable behaviours of ``ibex_top`` against the golden model.
+The golden model used is the `Spike RISC-V ISS <https://github.com/riscv-software-src/riscv-isa-sim>`_.
+
+The testbench uses UVM.
+It consists of 3 agents:
+
+Co-simulation Agent:
+  This has multiple monitors.
+  One monitors the RVFI interface which provides details of retired instructions.
+  The other monitors relate to fetched instructions and instruction memory errors; more details are provided in :ref:`coverage-plan`.
+  Additionally it connects to the monitor of the Memory Interface Agent for the instruction and data side via analysis ports.
+  The monitored transactions are used by a scoreboard to provide information to the co-simulation system allowing it to step the golden model and check its execution and memory activity against Ibex's behaviour.
+
+Memory Interface Agent
+  This provides a driver and a monitor for the :ref:`Ibex Memory Interface Protocol<lsu-protocol>`.
+  The driver provides fully randomised and configurable timings for responses and randomisation of error responses.
+  Two agents are instantiated; one for the data memory interface the other for the instruction memory interface.
+  Read data for memory responses is provided from a backing memory; write requests update the contents of the backing memory.
+  This is separate from the memory used by the golden model in the co-simulation agent.
+  The contents of these two memories will be identical unless there is a mismatch resulting in a failure.
+  The backing memory is held in a memory model as a separate UVM component.
+  The two agents use the same backing memory so they have a coherent view of memory.
+
+IRQ Agent
+  This provides a driver and a monitor for the IRQ interface.
+  It provides randomised interrupt stimulus to Ibex when a test requests it.
+  Constraints can be used to control types of interrupts generated (e.g. NMI or not) and whether multiple interrupts should be raised together.
+
+Debug and reset signals are a single wire each so do not have a dedicated agent.
+Instead any sequence that wishes to use them will directly manipulate them via a virtual interface
+
+The testbench instantiates the agents described above along with the memory model used by both the data and instruction side memory agents.
+A test consists of executing a pre-built binary (which is loaded into the memory model at the start of the test via backdoor accesses) along with configuring agents to provide appropriate stimulus for the test.
+Some tests may use the agents to generate stimulus at particular times (e.g.  interrupts).
+A test may perform additional checking on top of the co-simulation golden model comparison where appropriate (e.g. ensuring a raised interrupt has caused an exception).
+
+Stimulus Strategy
+-----------------
+
+Stimulus falls into two categories:
+
+* Instructions to execute: These are generated by the `RISC-V DV random instruction <https://github.com/google/riscv-dv>`_ generator and provided to the testbench via a raw binary file.
+* Activity on external interfaces.
+
+Instructions are generated ahead of time so the test has no control over them at run time.
+All external interfaces have their stimulus generated at run time so can be controlled by the test.
+It is the responsibility of the regression run environment to ensure generated instructions are matched with appropriate tests (e.g. ensuring an exception handler is present where interrupts are expected).
+
+Stimulus generation will use a coverage based approach.
+Stimulus is developed based upon the :ref:`coverage-plan`.
+Where coverage is not being hit stimulus will be added to hit it.
+
+Tests
+-----
+
+As with stimulus, test sequence development uses a coverage based approach.
+Tests will be added such that all coverage in the :ref:`coverage-plan` can be hit.
+Not all the details of specific tests will be documented here.
+The test list (`dv/uvm/core_ibex/riscv_dv_extension/testlist.yaml <https://github.com/lowRISC/ibex/blob/master/dv/uvm/core_ibex/riscv_dv_extension/testlist.yaml>`_), provides an exhaustive list of all tests along with a brief description of what the test does.
+
+A test will execute a binary whilst running zero or more sequences that provide stimulus to external interfaces of ``ibex_top``.
+As the memory interfaces are all driven by Ibex, with any testbench generated activity in response to a request from Ibex, they do not require explicit sequences run by the test.
+Instead the test can configure the randomisation of memory delays as it wishes.
+Memory errors can be configured to always occur in statically defined areas of the memory map or a sequence can be used to inject them via the memory interface agent.
+
+The following sequences are available for tests to use.
+Each sequence is derived from a base sequence which provides controls to repeat the sequence at fixed or random internals, forever or after a random number of repeats.
+Full details can be found in `dv/uvm/core_ibex/tests/core_ibex_seq_lib.sv <https://github.com/lowRISC/ibex/blob/master/dv/uvm/core_ibex/tests/core_ibex_seq_lib.sv>`_.
+
+* ``irq_raise_seq`` - Raises one or more interrupts.
+  The testbench binary can write to a special memory location to acknowledge the interrupt and cause it to drop.
+  Alternatively the testbench can drop it after a given amount of time.
+* ``debug_seq`` - Raises the external debug request.
+  The testbench binary can write to a special memory location to acknowledge the request and cause it to drop.
+  Alternatively the testbench can drop it after a given amount of time.
+* ``mem_error_seq`` - Injects a memory error in either the instruction side or data side, so the next access results in an error response.
+* ``reset_seq`` - Resets the core.