Vortex GPGPU

Vortex is a full-stack open-source RISC-V GPGPU. Vortex supports multiple backend drivers, including our C++ simulator (simx), an RTL simulator, and physical Xilinx and Altera FPGAs-- all controlled by a single driver script. The chosen driver determines the corresponding code invoked to run Vortex. Generally, developers will prototype their intended design in simx, before completing going forward with an RTL implementation. Alternatively, you can get up and running by selecting a driver of your choice and running a demo program.

Website

Vortex news can be found on its website

Citation

@inproceedings{10.1145/3466752.3480128,
	author = {Tine, Blaise and Yalamarthy, Krishna Praveen and Elsabbagh, Fares and Hyesoon, Kim},
	title = {Vortex: Extending the RISC-V ISA for GPGPU and 3D-Graphics},
	year = {2021},
	isbn = {9781450385572},
	publisher = {Association for Computing Machinery},
	address = {New York, NY, USA},
	url = {https://doi.org/10.1145/3466752.3480128},
	doi = {10.1145/3466752.3480128},
	abstract = {The importance of open-source hardware and software has been increasing. However, despite GPUs being one of the more popular accelerators across various applications, there is very little open-source GPU infrastructure in the public domain. We argue that one of the reasons for the lack of open-source infrastructure for GPUs is rooted in the complexity of their ISA and software stacks. In this work, we first propose an ISA extension to RISC-V that supports GPGPUs and graphics. The main goal of the ISA extension proposal is to minimize the ISA changes so that the corresponding changes to the open-source ecosystem are also minimal, which makes for a sustainable development ecosystem. To demonstrate the feasibility of the minimally extended RISC-V ISA, we implemented the complete software and hardware stacks of Vortex on FPGA. Vortex is a PCIe-based soft GPU that supports OpenCL and OpenGL. Vortex can be used in a variety of applications, including machine learning, graph analytics, and graphics rendering. Vortex can scale up to 32 cores on an Altera Stratix 10 FPGA, delivering a peak performance of 25.6 GFlops at 200 Mhz.},
	booktitle = {MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture},
	pages = {754–766},
	numpages = {13},
	keywords = {reconfigurable computing, memory systems., computer graphics},
	location = {Virtual Event, Greece},
	series = {MICRO '21}
}

Specifications

• Support RISC-V RV32IMAFC and RV64IMAFDC

• Microarchitecture:

  • configurable number of cores, warps, and threads.
  • configurable number of ALU, FPU, LSU, and SFU units per core.
  • graphics fixed-function pipeline (rasterizer, texture units, output mergers).
  • tensor cores with WGMMA and 2:4 structured sparsity support.
  • hardware-accelerated command processor and kernel management unit.
  • configurable pipeline issue width.
  • optional local memory, L1, L2, and L3 caches.

• Software:

  • OpenCL 1.2
  • Vulkan
  • HIP

• Supported FPGAs:

  • Altera Arria 10
  • Altera Stratix 10
  • Xilinx Alveo U50, U55C, U250, U280
  • Xilinx Versal VCK5000

Directory structure

• doc: Documentation.
• hw: Hardware sources.
• sw: Software sources (kernel, runtime, and drivers).
• sim: Simulators repository.
• tests: Tests repository.
• ci: Continuous integration scripts.
• miscs: Miscellaneous resources.
• VX_config.toml / VX_types.toml: Hardware configuration system.

Quick Start

If you are interested in a stable release of Vortex, you can download the latest release here. Otherwise, you can pull the most recent, but (potentially) unstable version as shown below. The following steps demonstrate how to build and run Vortex with the default driver: SimX. If you are interested in a different backend, look here.

Supported OS Platforms

• Ubuntu 22.04
• Centos 7

Toolchain Dependencies

The following dependencies will be fetched prebuilt by toolchain_install.sh.

• POCL
• LLVM
• RISCV-GNU-TOOLCHAIN
• Verilator
• cvfpu
• SoftFloat
• Ramulator
• Yosys
• Sv2v
• Mesa
• chipStar

Install Vortex codebase

	git clone --depth=1 --recursive https://github.com/vortexgpgpu/vortex.git
	cd vortex

Install system dependencies

# ensure dependent libraries are present
sudo ./ci/install_dependencies.sh

Configure your build folder

    mkdir build
    cd build
    # for 32bit
    ../configure --xlen=32 --tooldir=$HOME/tools
    # for 64bit
    ../configure --xlen=64 --tooldir=$HOME/tools

Install prebuilt toolchain

   ./ci/toolchain_install.sh

Building and installing Vortex

make -s
make install
export VORTEX_PATH=$(pwd)/install
export PKG_CONFIG_PATH=$VORTEX_PATH/lib/pkgconfig:$PKG_CONFIG_PATH

make install lays out a sysroot under $VORTEX_PATH containing the public headers, libraries, and vortex-runtime.pc / vortex-kernel.pc pkg-config files. Downstream tools (mesa-vortex, pocl-vortex, chipstar) integrate with Vortex exclusively through $VORTEX_PATH and pkg-config — the same shape as the CUDA, ROCm and oneAPI SDKs. The source tree ($VORTEX_HOME) and build tree ($VORTEX_BUILD_DIR) are internal to Vortex and not exposed to consumers. Override the install root with ../configure --prefix=<path> or --installdir=<path> (default <build>/install).

Quick demo running vecadd OpenCL kernel on 2 cores

./ci/blackbox.sh --cores=2 --app=vecadd

Common Developer Tips

• Building Vortex 64-bit requires setting --xlen=64 configure option.

../configure --xlen=64 --tooldir=$HOME/tools

• No shell environment setup is required. ../configure bakes the full toolchain layout (paths, XCONFIGS, every tool binary) into the build dir's config.mk and domain common.mk files. Make recipes invoke tools by absolute path ($(VERILATOR_PATH)/bin/verilator etc.), so multiple Vortex trees on the same machine can coexist without any ~/.bashrc sourcing.
• Making changes to Makefiles in your source tree or adding new folders will require executing the "configure" script again without any options to get changes propagated to your build folder.

../configure

• To debug the GPU, the simulation can generate a runtime trace for analysis. See /docs/debugging.md for more information.

./ci/blackbox.sh --app=demo --debug=3

• For additional information, check out the documentation