Merge pull request #200 from Udit8348/develop-docker-micro

Develop Docker Micro

README.md

@@ -1,6 +1,30 @@
 # Vortex GPGPU
 
-Vortex is a full-stack open-source RISC-V GPGPU.
+Vortex is a full-stack open-source RISC-V GPGPU. Vortex supports multiple **backend drivers**, including our C++ simulator (simx), an RTL simulator, and physical Xilinx and Altera FPGAs-- all controlled by a single driver script. The chosen driver determines the corresponding code invoked to run Vortex. Generally, developers will prototype their intended design in simx, before completing going forward with an RTL implementation. Alternatively, you can get up and running by selecting a driver of your choice and running a demo program.
+
+## Website
+Vortex news can be found on its [website](https://vortex.cc.gatech.edu/)
+
+## Citation
+```
+@inproceedings{10.1145/3466752.3480128,
+	author = {Tine, Blaise and Yalamarthy, Krishna Praveen and Elsabbagh, Fares and Hyesoon, Kim},
+	title = {Vortex: Extending the RISC-V ISA for GPGPU and 3D-Graphics},
+	year = {2021},
+	isbn = {9781450385572},
+	publisher = {Association for Computing Machinery},
+	address = {New York, NY, USA},
+	url = {https://doi.org/10.1145/3466752.3480128},
+	doi = {10.1145/3466752.3480128},
+	abstract = {The importance of open-source hardware and software has been increasing. However, despite GPUs being one of the more popular accelerators across various applications, there is very little open-source GPU infrastructure in the public domain. We argue that one of the reasons for the lack of open-source infrastructure for GPUs is rooted in the complexity of their ISA and software stacks. In this work, we first propose an ISA extension to RISC-V that supports GPGPUs and graphics. The main goal of the ISA extension proposal is to minimize the ISA changes so that the corresponding changes to the open-source ecosystem are also minimal, which makes for a sustainable development ecosystem. To demonstrate the feasibility of the minimally extended RISC-V ISA, we implemented the complete software and hardware stacks of Vortex on FPGA. Vortex is a PCIe-based soft GPU that supports OpenCL and OpenGL. Vortex can be used in a variety of applications, including machine learning, graph analytics, and graphics rendering. Vortex can scale up to 32 cores on an Altera Stratix 10 FPGA, delivering a peak performance of 25.6 GFlops at 200 Mhz.},
+	booktitle = {MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture},
+	pages = {754–766},
+	numpages = {13},
+	keywords = {reconfigurable computing, memory systems., computer graphics},
+	location = {Virtual Event, Greece},
+	series = {MICRO '21}
+}
+```
 
 ## Specifications
 
@@ -30,12 +54,14 @@ Vortex is a full-stack open-source RISC-V GPGPU.
 - `ci`: Continuous integration scripts.
 - `miscs`: Miscellaneous resources.
 
-## Build Instructions
-More detailed build instructions can be found [here](docs/install_vortex.md).
+## Quick Start
+If you are interested in a stable release of Vortex, you can download the latest release [here](https://github.com/vortexgpgpu/vortex/releases/latest). Otherwise, you can pull the most recent, but (potentially) unstable version as shown below. The following steps demonstrate how to build and run Vortex with the default driver: SimX. If you are interested in a different backend, look [here](docs/simulation.md).
+
 ### Supported OS Platforms
 - Ubuntu 18.04, 20.04, 22.04, 24.04
 - Centos 7
 ### Toolchain Dependencies
+The following dependencies will be fetched prebuilt by `toolchain_install.sh`.
 - [POCL](http://portablecl.org/)
 - [LLVM](https://llvm.org/)
 - [RISCV-GNU-TOOLCHAIN](https://github.com/riscv-collab/riscv-gnu-toolchain)
@@ -105,4 +131,4 @@ echo "source <build-path>/ci/toolchain_env.sh" >> ~/.bashrc
 ```sh
 ./ci/blackbox.sh --app=demo --debug=3
 ```
-- For additional information, check out the /docs.
+- For additional information, check out the [documentation](docs/index.md)

ci/install_dependencies.sh

@@ -31,7 +31,7 @@ check_gcc_version() {
 apt-get update -y
 
 # install system dependencies
-apt-get install -y build-essential valgrind libstdc++6 binutils python3 uuid-dev ccache
+apt-get install -y build-essential valgrind libstdc++6 binutils python3 uuid-dev ccache cmake
 
 # Check and install GCC 11 if necessary
 if check_gcc_version; then

docs/altera_fpga_guide.md

@@ -1,92 +0,0 @@
-# FPGA Startup and Configuration Guide
-
-OPAE Environment Setup
-----------------------
-
-    $ source /opt/inteldevstack/init_env_user.sh
-    $ export OPAE_HOME=/opt/opae/1.1.2
-    $ export PATH=$OPAE_HOME/bin:$PATH
-    $ export C_INCLUDE_PATH=$OPAE_HOME/include:$C_INCLUDE_PATH
-    $ export LIBRARY_PATH=$OPAE_HOME/lib:$LIBRARY_PATH
-    $ export LD_LIBRARY_PATH=$OPAE_HOME/lib:$LD_LIBRARY_PATH
-
-OPAE Build
-------------------
-
-The FPGA has to following configuration options:
-- DEVICE_FAMILY=arria10 | stratix10
-- NUM_CORES=#n
-
-Command line:
-
-    $ cd hw/syn/altera/opae
-    $ PREFIX=test1 TARGET=fpga NUM_CORES=4 make
-
-A new folder (ex: `test1_xxx_4c`) will be created and the build will start and take ~30-480 min to complete.
-Setting TARGET=ase will build the project for simulation using Intel ASE.
-
-
-OPAE Build Configuration
-------------------------
-
-The hardware configuration file `/hw/rtl/VX_config.vh` defines all the hardware parameters that can be modified when build the processor.For example, have the following parameters that can be configured:
-- `NUM_WARPS`:   Number of warps per cores
-- `NUM_THREADS`: Number of threads per warps
-- `PERF_ENABLE`: enable the use of all profile counters
-
-You can configure the synthesis build from the command line:
-
-    $ CONFIGS="-DPERF_ENABLE -DNUM_THREADS=8" make
-
-OPAE Build Progress
--------------------
-
-You could check the last 10 lines in the build log for possible errors until build completion.
-
-    $ tail -n 10 <build_dir>/synth/build.log
-
-Check if the build is still running by looking for quartus_sh, quartus_syn, or quartus_fit programs.
-
-    $ ps -u <username>
-
-If the build fails and you need to restart it, clean up the build folder using the following command:
-
-    $ make clean
-
-The bitstream file `vortex_afu.gbs` should exist when the build is done:
-
-    $ ls -lsa <build_dir>/synth/vortex_afu.gbs
-
-
-Signing the bitstream and Programming the FPGA
-----------------------------------------------
-
-    $ cd <build_dir>
-    $ PACSign PR -t UPDATE -H openssl_manager -i vortex_afu.gbs -o vortex_afu_unsigned_ssl.gbs
-    $ fpgasupdate vortex_afu_unsigned_ssl.gbs
-
-Sample FPGA Run Test
---------------------
-
-Ensure you have the correct opae runtime for the FPGA target
-
-    $ TARGET=FPGA make -C runtime/opae
-
-Run the following from your Vortex build directory
-
-    $ TARGET=fpga ./ci/blackbox.sh --driver=opae --app=sgemm --args="-n128"
-
-Testing Vortex using OPAE with Intel ASE Simulation
----------------------------------------------------
-
-Building ASE synthesis
-
-    $ TARGET=asesim make -C runtime/opae
-
-Building ASE runtime
-
-    $ TARGET=asesim make -C runtime/opae
-
-Running ASE simulation
-
-    $ ASE_LOG=0 ASE_WORKDIR=<build_dir>/synth/work TARGET=asesim ./ci/blackbox.sh --driver=opae --app=sgemm --args="-n16"

docs/contributing.md

@@ -1,18 +1,37 @@
-# Contributing to Vortex on Github
+# Contributing to Vortex
 
-## Github Details
-- There are two main repos, `vortex` (public, this one) and `vortex-dev` (private)
-- todo: Most current development is on `vortex`
-- If you have a legacy version of `vortex`, you can use the releases branch or tags to access the repo at that point in time
+## Github
+Vortex uses Github to host its git repositories.
+There are a lot of ways to use the features on Github for collaboration.
+Therefore, this documentation details the standard procedure for contributing to Vortex.
+Development of Vortex is consolidated to this repo, `vortex` and any associated forks.
+Previously, there was active work done on a private repo named `vortex-dev`.
+`vortex-dev` has officially been deprecated and fully merged into this public repo, `vortex`.
+If you are returning to this project and have legacy versions of Vortex, you can use the releases branches to access older versions.
 
 ## Contribution Process
-- You should create a new branch from develop that is clearly named with the feature that you want to add
-- Avoid pushing directly to the `master` branch instead you will need to make a Pull Request (PR)
-- There should be protections in place that prevent pushing directly to the main branch, but don't rely on it
-- When you make a PR it will be tested against the continuous integration (ci) pipeline (see `continuous_integration.md`)
-- It is not sufficient to just write some tests, they need to be incorporated into the ci pipeline to make sure they are run
-- During a PR, you might receive feedback regarding your changes and you might need to make further commits to your branch
+In an effort to keep `vortex` organized, permissions to directly create branches and push code has been limited to admins.
+However, contributions are strongly encouraged and keep the project moving forward! Here is the procedure for contributing:
 
+1. Create a fork of `vortex`
+2. In your fork, create a branch from `master` that briefly explains the work you are adding (ie: `develop-documentation`)
+3. Make your changes on the new branch in your fork. You may create as many commits as you need, which might be common if you are making multiple iterations
+4. Since you are the owner of your fork, you have full permissions to push commits to your fork
+4. When you are satisfied with the changes on your fork, you can open a PR from your fork using the online interface
+5. If you recently made a push, you will get automatically get a prompt on Github online to create a PR, which you can press
+6. Otherwise, you can go to your fork on Github online and manually create a PR (todo)
+(todo): how to name and format your PR, what information you should add to the PR, does not need to be too strict if you are attending the weekly meetings*
+7. Github uses the following semantics: `base repository` gets the changes from your `head repository`
+8. Therefore, you should set the `base repository` to `vortexgpgpu/vortex` and the `base` branch to `master` since the master branch is protected by reviewed PRs.
+9. And you should assign the `head repository` to `<your-github-username>/vortex` (which represents your fork of vortex) and the `base` branch to the one created in step 2
+10. Now that your intended PR has been specified, you should review the status. Check for merge conflicts, if all your commits are present, and all the modified files make sense
+11. You can still make a PR if there are issues in step 10, just make sure the structure is correct according to steps 7-9
+12. Once the PR is made, the CI pipeline will run automatically, testing your changes
+13. Remember, a PR is flexible if you need to make changes to the code you can go back to your branch of the fork to commit and push any updates
+14. As long as the `head repository`'s `base` branch is the one you edited, the PR will automatically get the most recent changes
+15. When all merge conflicts are resolved, changes are made, and tests pass you can have an admin merge your PR
 
-## Creating  and Adding Tests
-see `testing.md`
+## What Makes a Good Contribution?
+- If you are contributing code changes, then review [testing.md](./testing.md) to ensure your tests are integrated into the [CI pipeline](continuous_integration.md)
+- During a PR, you should consider the advice you are provided by your reviewers. Remember you keep adding commits to an open PR!
+- If your change aims to fix an issue opened on Github, please tag that issue in the PR itself

docs/environment_setup.md

@@ -1,16 +1,19 @@
 # Environment Setup
+
 These instructions apply to the development vortex repo using the updated toolchain. The updated toolchain is considered to be any commit of `master` pulled from July 2, 2023 onwards. The toolchain update in question can be viewed in this [commit](https://github.com/vortexgpgpu/vortex-dev/commit/0048496ba28d7b9a209a0e569d52d60f2b68fc04). Therefore, if you are unsure whether you are using the new toolchain or not, then you should check the `ci` folder for the existence of the `toolchain_prebuilt.sh` script. Furthermore, you should notice that the `toolchain_install.sh` script has the legacy `llvm()` split into `llvm-vortex()` and `llvm-pocl()`.
 
-
 ## Set Up on Your Own System
+
 The toolchain binaries provided with Vortex are built on Ubuntu-based systems. To install Vortex on your own system, [follow these instructions](install_vortex.md).
 
-
 ## Servers for Georgia Tech Students and Collaborators
+
 ### Volvo
+
 Volvo is a 64-core server provided by HPArch. You need valid credentials to access it. If you don't already have access, you can get in contact with your mentor to ask about setting your account up.
 
 Setup on Volvo:
+
 1. Connect to Georgia Tech's VPN or ssh into another machine on campus
 2. `ssh volvo.cc.gatech.edu`
 3. Clone Vortex to your home directory: `git clone --recursive https://github.com/vortexgpgpu/vortex.git`
@@ -19,9 +22,11 @@ Setup on Volvo:
 6. Run a test program: `./ci/blackbox.sh --cores=2 --app=dogfood`
 
 ### Nio
+
 Nio is a 20-core desktop server provided by HPArch. If you have access to Volvo, you also have access to Nio.
 
 Setup on Nio:
+
 1. Connect to Georgia Tech's VPN or ssh into another machine on campus
 2. `ssh nio.cc.gatech.edu`
 3. Clone Vortex to your home directory: `git clone --recursive https://github.com/vortexgpgpu/vortex.git`
@@ -29,11 +34,12 @@ Setup on Nio:
 5. `make -s` in the `vortex` root directory
 6. Run a test program: `./ci/blackbox.sh --cores=2 --app=dogfood`
 
-
 ## Docker (Experimental)
+
 Docker allows for isolated pre-built environments to be created, shared and used. The emulation mode required for ARM-based processors will incur a decrease in performance. Currently, the dockerfile is not included with the official vortex repository and is not actively maintained or supported.
 
 ### Setup with Docker
+
 1. Clone repo recursively onto your local machine: `git clone --recursive https://github.com/vortexgpgpu/vortex.git`
 2. Download the dockerfile from [here](https://github.gatech.edu/gist/usubramanya3/f1bf3e953faa38a6372e1292ffd0b65c) and place it in the root of the repo.
 3. Build the Dockerfile into an image: `docker build --platform=linux/amd64 -t vortex -f dockerfile .`

docs/fpga_setup.md

@@ -0,0 +1,217 @@
+# FPGA Startup and Configuration Guide
+
+## Gaining Access to FPGA's with CRNCH
+If you are associated with Georgia Tech (or related workshops) you can use CRNCH's server to gain remote access to FPGA's. Otherwise, you can skip to the Xilinx or Intel (Altera) synthesis steps below.
+
+## What is CRNCH?
+
+**C**enter for **R**esearch into **N**ovel **C**omputing **H**ierarchies
+
+## What does CRNCH Offer?
+
+**The Rogues Gallery (RG)**: new concept focused on developing our understanding of next-generation hardware with a focus on unorthodox and uncommon technologies. **RG** will acquire new and unique hardware (ie, the aforementioned “*rogues*”) from vendors, research labs, and startups and make this hardware available to students, faculty, and industry collaborators within a managed data center environment
+
+## Why are the Rouges Important?
+
+By exposing students and researchers to this set of unique hardware, we hope to foster cross-cutting discussions about hardware designs that will drive future *performance improvements in computing long after the Moore’s Law era of “cheap transistors” ends*. Specifically, the Rouges Gallery contains FPGA's which can be synthesized into Vortex hardware.
+
+## How is the Rouges Gallery Funded?
+
+Rogues Gallery testbed is primarily supported by the National Science Foundation (NSF) under NSF Award Number [#2016701](https://www.nsf.gov/awardsearch/showAward?AWD_ID=2016701&HistoricalAwards=false)
+
+## Rouges Gallery Documentation
+
+You can read about RG in more detail on its official documentation [page](https://gt-crnch-rg.readthedocs.io/en/main/index.html#).
+
+You can listen to a talk about RG [here](https://mediaspace.gatech.edu/media/Jeff%20Young%20-%20Rogues%20Gallery%20-%20CRNCH%20Summit%202021/1_lqlgr0jj)
+
+[CRNCH Summit 2023](https://github.com/gt-crnch/crnch-summit-2023/tree/main)
+
+## Request Access for Rouges Gallery
+
+You should use [this form](https://crnch-rg.cc.gatech.edu/request-rogues-gallery-access/) to request access to RG’s reconfigurable computing (vortex fpga) resources. You should receive an email with your ticket item being created. Once it gets processed, you should get an email confirmed your access has been granted. It might take some time to get processed.
+
+## How to Access Rouges Gallery?
+There are two methods of accessing CRNCH's Rouges Gallery
+1) Web-based GUI: [rg-ood.crnch.gatech.edu](http://rg-ood.crnch.gatech.edu/)
+2) SSH: `ssh <your-gt-username>@rg-login.crnch.gatech.edu`
+
+
+## Where should I keep my files?
+The CRNCH servers have a folder called `USERSCRATCH` which can be found in your home directory: `echo $HOME`. You should keep all your files in this folder since it is available across all the Rouges Gallery Nodes.
+
+## **What Machines are Available in the Rogues Gallery?**
+
+Complete list of machines can be found [here](https://gt-crnch-rg.readthedocs.io/en/main/general/rg-hardware.html). Furthermore, you can find detailed information about the FPGA hardware [here](https://gt-crnch-rg.readthedocs.io/en/main/reconfig/xilinx/xilinx-getting-started.html).
+
+## Allocate an FPGA Node
+Once you’ve connected to the CRNCH login node, you can use the Slurm scheduler to request an interactive job using `salloc`. This [page](https://gt-crnch-rg.readthedocs.io/en/main/general/using-slurm.html) explains why we use Slurm to request resources. Documentation for `salloc` can be found [here](https://gt-crnch-rg.readthedocs.io/en/main/general/using-slurm-examples.html). And here.
+
+
+To request 16 cores and 64GB of RAM for 6 hours on flubber9, a fpga dev node:
+```bash
+salloc -p rg-fpga --nodes=1 --ntasks-per-node=16 --mem=64G --nodelist flubber1 --time=06:00:00
+```
+Synthesis for Xilinx Boards
+----------------------
+Once you are logged in, you will need to complete some first time configurations. If you are interested in the Intel (Altera) synthesis steps, scroll down below.
+
+### Source Configuration Scripts
+```
+# From any directory
+$ source /opt/xilinx/xrt/setup.sh
+$ source /tools/reconfig/xilinx/Vitis/2023.1/settings64.sh
+```
+
+### Check Installed FPGA Platforms
+`platforminfo -l` which tells us the correct name of the platform installed on the current fpga node. It should be used for the `PLATFORM` variable below. Otherwise, if there is an error then there was an issue with the previous two commands.
+
+### Install Vortex Toolchain
+The Xilinx synthesis process requires verilator to generate the bitstream. Eventually, you will need the whole toolchain to run the bitstream on the FPGA. Therefore, the Vortex toolchain and can be installed as follows. If you complete these steps properly, you should only need to complete them once and you can skip to `Activate Vortex Toolchain`
+```
+# Make a build directory from root and configure scripts for your environment
+mkdir build && cd build && ../configure --tooldir=$HOME/tools
+
+# Install the whole prebuilt toolchain
+./ci/toolchain_install.sh --all
+
+# Add environment variables to bashrc
+echo "source <full-path-to-vortex-root>/vortex/build/ci/toolchain_env.sh" >> ~/.bashrc
+```
+
+### Activate Vortex Toolchain
+```
+# From any directory
+source ~/.bashrc
+
+# Check environment setup
+verilator --version
+```
+
+### Build the FPGA Bitstream
+The root directory contains the path `hw/syn/xilinx/xrt` which has the makefile used to generate the Vortex bitstream.
+
+```
+    $ cd hw/syn/xilinx/xrt
+    $ PREFIX=test1 PLATFORM=xilinx_u50_gen3x16_xdma_5_202210_1 TARGET=hw NUM_CORES=1 make > build_u250_hw_1c.log 2>&1 &
+```
+Will run the synthesis under new build directory: BUILD_DIR := "\<PREFIX>\_\<PLATFORM>\_\<TARGET>"
+The generated bitstream will be located under <BUILD_DIR>/bin/vortex_afu.xclbin
+
+For long-running jobs, invocation of this makefile can be made of the following form:
+
+`[CONFIGS=<vortex macros>] [PREFIX=<prefix directory name>] [NUM_CORES=<#>] TARGET=hw|hw_emu PLATFORM=<platform baseName> nohup make > <log filename> 2>&1 &`
+
+For example:
+
+```bash
+CONFIGS="-DL2_ENABLE -DDCACHE_SIZE=8192" PREFIX=build_4c_u280 NUM_CORES=4 TARGET=hw PLATFORM=xilinx_u280_gen3x16_xdma_1_202310_1 nohup make > build_u250_hw_4c.log 2>&1 &
+```
+
+The build is complete when the bitstream file `vortex_afu.xclbin` exists in `<prefix directory name><platform baseName>hw|hw_emu/bin`.
+
+### Running a Program on Xilinx FPGA
+
+The [blackbox.sh](./simulation.md) script within the build directory can be used to run a test with Vortex’s xrt driver using the following command:
+
+`FPGA_BIN_DIR=<path to bitstream directory> TARGET=hw|hw_emu PLATFORM=<platform baseName> ./ci/blackbox.sh --driver=xrt --app=<test name>`
+
+For example:
+
+```FPGA_BIN_DIR=<realpath> hw/syn/xilinx/xrt/build_4c_u280_xilinx_u280_gen3x16_xdma_1_202211_1_hw/bin TARGET=hw PLATFORM=xilinx_u280_gen3x16_xdma_1_202211_1 ./ci/blackbox.sh --driver=xrt --app=demo```
+
+Synthesis for Intel (Altera) Boards
+----------------------
+
+### OPAE Environment Setup
+
+
+    $ source /opt/inteldevstack/init_env_user.sh
+    $ export OPAE_HOME=/opt/opae/1.1.2
+    $ export PATH=$OPAE_HOME/bin:$PATH
+    $ export C_INCLUDE_PATH=$OPAE_HOME/include:$C_INCLUDE_PATH
+    $ export LIBRARY_PATH=$OPAE_HOME/lib:$LIBRARY_PATH
+    $ export LD_LIBRARY_PATH=$OPAE_HOME/lib:$LD_LIBRARY_PATH
+
+### OPAE Build
+
+The FPGA has to following configuration options:
+- DEVICE_FAMILY=arria10 | stratix10
+- NUM_CORES=#n
+
+Command line:
+
+    $ cd hw/syn/altera/opae
+    $ PREFIX=test1 TARGET=fpga NUM_CORES=4 make
+
+A new folder (ex: `test1_xxx_4c`) will be created and the build will start and take ~30-480 min to complete.
+Setting TARGET=ase will build the project for simulation using Intel ASE.
+
+
+### OPAE Build Configuration
+
+The hardware configuration file `/hw/rtl/VX_config.vh` defines all the hardware parameters that can be modified when build the processor.For example, have the following parameters that can be configured:
+- `NUM_WARPS`:   Number of warps per cores
+- `NUM_THREADS`: Number of threads per warps
+- `PERF_ENABLE`: enable the use of all profile counters
+
+You configure the syntesis build from the command line:
+
+    $ CONFIGS="-DPERF_ENABLE -DNUM_THREADS=8" make
+
+### OPAE Build Progress
+
+You could check the last 10 lines in the build log for possible errors until build completion.
+
+    $ tail -n 10 <build_dir>/build.log
+
+Check if the build is still running by looking for quartus_sh, quartus_syn, or quartus_fit programs.
+
+    $ ps -u <username>
+
+If the build fails and you need to restart it, clean up the build folder using the following command:
+
+    $ make clean
+
+The file `vortex_afu.gbs` should exist when the build is done:
+
+    $ ls -lsa <build_dir>/synth/vortex_afu.gbs
+
+
+### Signing the bitstream and Programming the FPGA
+
+    $ cd <build_dir>
+    $ PACSign PR -t UPDATE -H openssl_manager -i vortex_afu.gbs -o vortex_afu_unsigned_ssl.gbs
+    $ fpgasupdate vortex_afu_unsigned_ssl.gbs
+
+### Sample FPGA Run Test
+Ensure you have the correct opae runtime for the FPGA target
+
+```
+$ TARGET=FPGA make -C runtime/opae
+```
+
+Run the [blackbox.sh](./simulation.md) from your Vortex build directory
+
+```
+$ TARGET=fpga ./ci/blackbox.sh --driver=opae --app=sgemm --args="-n128"
+```
+
+### FPGA sample test running OpenCL sgemm kernel
+
+You can use the `blackbox.sh` script to run the following from your Vortex build directory
+
+    $ TARGET=fpga ./ci/blackbox.sh --driver=opae --app=sgemm --args="-n128"
+
+### Testing Vortex using OPAE with Intel ASE Simulation
+Building ASE synthesis
+
+```$ TARGET=asesim make -C runtime/opae```
+
+Building ASE runtime
+
+```$ TARGET=asesim make -C runtime/opae```
+
+Running ASE simulation
+
+```$ ASE_LOG=0 ASE_WORKDIR=<build_dir>/synth/work TARGET=asesim ./ci/blackbox.sh --driver=opae --app=sgemm --args="-n16"```

docs/index.md

@@ -2,32 +2,8 @@
 
 ## Table of Contents
 
-- [Codebase Layout](codebase.md)
-- [Microarchitecture](microarchitecture.md)
-- [Cache Subsystem](cache_subsystem.md)
-- [Software](software.md)
-- [Simulation](simulation.md)
-- [Altera FPGA Setup Guide](altera_fpga_guide.md)
-- [Xilinx FPGA Setup Guide](xilinx_fpga_guide.md)
-- [Debugging](debugging.md)
-- [Useful Links](references.md)
-
-## Installation
-
-- For the different environments Vortex supports, [read this document](environment_setup.md).
-- To install on your own system, [follow this document](install_vortex.md).
-
-## Quick Start Scenarios
-
-Running Vortex simulators with different configurations:
-- Run basic driver test with rtlsim driver and Vortex config of 2 clusters, 2 cores, 2 warps, 4 threads
-
-    $ ./ci/blackbox.sh --driver=rtlsim --clusters=2 --cores=2 --warps=2 --threads=4  --app=basic
-
-- Run demo driver test with opae driver and Vortex config of 1 clusters, 4 cores, 4 warps, 2 threads
-
-    $ ./ci/blackbox.sh --driver=opae --clusters=1 --cores=4 --warps=4 --threads=2 --app=demo
-
-- Run dogfood driver test with simx driver and Vortex config of 4 cluster, 4 cores, 8 warps, 6 threads
-
-    $ ./ci/blackbox.sh --driver=simx --clusters=4 --cores=4 --warps=8 --threads=6  --app=dogfood
+- [Codebase Layout](codebase.md): Summary of repo file tree
+- [Microarchitecture](microarchitecture.md): Vortex Pipeline and cache microarchitectural details and reconfigurability
+- [Simulation](simulation.md): Details for building and running each simulation driver
+- [Contributing](contributing.md): Process for contributing your own features including repo semantics and testing
+- [Debugging](debugging.md): Debugging configurations for each Vortex driver

docs/microarchitecture.md

@@ -77,4 +77,7 @@ Vortex has a 6-stage pipeline:
 - Sockets
   - Grouping multiple cores sharing L1 cache
 - Clusters
-  - Grouping of sockets sharing L2 cache
+  - Grouping of sockets sharing L2 cache
+
+### Vortex Cache Subsystem
+More details about the cache subsystem are provided [here](./cache_subsystem.md).

docs/simulation.md

@@ -6,13 +6,16 @@
 
 ### Cycle-Approximate Simulation
 
-SimX is a C++ cycle-level in-house simulator developed for Vortex. The relevant files are located in the `simX` folder.
+SimX is a C++ cycle-level in-house simulator developed for Vortex. The relevant files are located in the `simx` folder. The [readme](README.md) has the most detailed instructions for building and running simX.
+
+- To install on your own system, [follow this document](install_vortex.md).
+- For the different Georgia Tech environments Vortex supports, [read this document](environment_setup.md).
 
 ### FGPA Simulation
 
-The current target FPGA for simulation is the Arria10 Intel Accelerator Card v1.0. The guide to build the fpga with specific configurations is located [here.](fpga_setup.md)
+The guide to build the fpga with specific configurations is located [here.](fpga_setup.md) You can find instructions for both Xilinx and Altera based FPGAs.
 
-### How to Test
+### How to Test (using `blackbox.sh`)
 
 Running tests under specific drivers (rtlsim,simx,fpga) is done using the script named `blackbox.sh` located in the `ci` folder. Running command `./ci/blackbox.sh --help` from the Vortex root directory will display the following command line arguments for `blackbox.sh`:
 
@@ -47,4 +50,20 @@ PERF: core1: instrs=90693, cycles=53108, IPC=1.707709
 PERF: core2: instrs=90849, cycles=53107, IPC=1.710678
 PERF: core3: instrs=90836, cycles=50347, IPC=1.804199
 PERF: instrs=363180, cycles=53108, IPC=6.838518
-```
+```
+
+## Additional Quick Start Scenarios
+
+Running Vortex simulators with different configurations and drivers is supported. For example:
+
+- Run basic driver test with rtlsim driver and Vortex config of 2 clusters, 2 cores, 2 warps, 4 threads
+
+    $ ./ci/blackbox.sh --driver=rtlsim --clusters=2 --cores=2 --warps=2 --threads=4  --app=basic
+
+- Run demo driver test with opae driver and Vortex config of 1 clusters, 4 cores, 4 warps, 2 threads
+
+    $ ./ci/blackbox.sh --driver=opae --clusters=1 --cores=4 --warps=4 --threads=2 --app=demo
+
+- Run dogfood driver test with simx driver and Vortex config of 4 cluster, 4 cores, 8 warps, 6 threads
+
+    $ ./ci/blackbox.sh --driver=simx --clusters=4 --cores=4 --warps=8 --threads=6  --app=dogfood

docs/testing.md

@@ -2,7 +2,7 @@
 
 ## Running a Vortex application
 
-The framework provides a utility script: blackbox.sh under the /ci/ folder for executing applications in the tests tree.
+The framework provides a utility script: blackbox.sh under the /ci/ folder for executing applications in the tests tree. It gets copied into the `build` directory with all the environment variables resolved, so you should run it from the `build` directory as follows:
 You can query the commandline options of the tool using:
 
     $ ./ci/blackbox.sh --help
@@ -49,4 +49,4 @@ Compile your test: `$ make -C tests/regression/<test-name>`
 Run your test: `$ ./ci/blackbox.sh --driver=simx --app=<test-name> --debug`
 
 ## Adding Your Tests to the CI Pipeline
-See `continuous_integration.md`
+If you are a contributor, then you will need to add tests that integrate into the continuous integration pipeline. Remember, Pull Requests cannot be merged unless new code has tests and existing tests do not regress. Furthermore, if you are contributing a new feature, it is recommended that you add the ability to enable / disable the new feature that you are adding. See more at [contributing.md](contributing.md) and [continuous_integration.md](continuous_integration.md).

docs/xilinx_fpga_guide.md

@@ -1,52 +0,0 @@
-# FPGA Startup and Configuration Guide
-
-XRT Environment Setup
-----------------------
-
-    $ source /opt/xilinx/Vitis/2023.1/settings64.sh
-    $ source /opt/xilinx/xrt/setup.sh
-
-
-Check Installed FPGA Platforms
-------------------------------
-
-    $ platforminfo -l
-
-
-Build FPGA image
-----------------
-
-    $ cd hw/syn/xilinx/xrt
-    $ PREFIX=test1 PLATFORM=xilinx_u50_gen3x16_xdma_5_202210_1 TARGET=hw NUM_CORES=4 make
-
-Will run the synthesis under new build directory: BUILD_DIR := "\<PREFIX>\_\<PLATFORM>\_\<TARGET>"
-
-The generated bitstream will be located under <BUILD_DIR>/bin/vortex_afu.xclbin
-
-Sample FPGA Run Test
---------------------
-
-Ensure you have the correct opae runtime for the FPGA target
-
-    $ make -C runtime/xrt clean
-    $ TARGET=hw make -C runtime/xrt
-
-Run the following from your Vortex build directory
-
-    $ TARGET=hw FPGA_BIN_DIR=<BUILD_DIR>/bin ./ci/blackbox.sh --driver=xrt --app=sgemm --args="-n128"
-
-Testing Vortex using XRT Hardware Emulation
--------------------------------------------
-
-Building XRT's hw_emu target
-
-    $ cd hw/syn/xilinx/xrt
-    $ PREFIX=test2 PLATFORM=xilinx_u50_gen3x16_xdma_5_202210_1 TARGET=hw_emu make
-
-Building XRT hw_meu runtime
-
-    $ TARGET=hw_emu make -C runtime/xrt
-
-Running XRT hw_emu simulation
-
-    $ TARGET=hw_emu FPGA_BIN_DIR=<BUILD_DIR>/bin ./ci/blackbox.sh --driver=xrt --app=sgemm

miscs/docker/Dockerfile.prod

@@ -18,41 +18,32 @@ FROM ubuntu:20.04
 ARG DEBIAN_FRONTEND=noninteractive
 
 # Install necessary dependencies and upgrade installed components
-RUN apt-get update -y && \
-    apt-get install -y \
+# Update and install necessary dependencies
+RUN apt-get update && apt-get install -y \
     software-properties-common \
     build-essential \
     python3 \
     git \
     wget \
     curl \
-    ca-certificates \
-    valgrind \
-    libstdc++6 \
-    binutils \
-    uuid-dev \
-    ccache \
-    cmake && \
-    apt-get upgrade -y && \
-    gcc_version=$(gcc -dumpversion) && \
-    if dpkg --compare-versions "$gcc_version" lt 11; then \
-        echo "GCC version is less than 11. Installing GCC 11..." && \
-        add-apt-repository -y ppa:ubuntu-toolchain-r/test && \
-        apt-get update -y && \
-        apt-get install -y g++-11 gcc-11 && \
-        update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-11 100 && \
-        update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 100; \
-    else \
-        echo "GCC version is 11 or greater. No need to install GCC 11."; \
-    fi && \
+    ca-certificates && \
     rm -rf /var/lib/apt/lists/*
 
+# upgrade installed components
+RUN apt-get upgrade && apt-get update
+
+# temporary until remote dependency script gets updated
+RUN apt-get install -y cmake
+
 # Clone the Vortex repository
 RUN git clone --depth=1 --recursive https://github.com/vortexgpgpu/vortex.git /vortex
 
 # Set the initial working directory
 WORKDIR /vortex
 
+# install system dependencies
+RUN ./ci/install_dependencies.sh
+
 # Configure the build folder
 RUN mkdir build && cd build && ../configure
 

miscs/docker/README.md

@@ -4,17 +4,32 @@ You can install Docker desktop on MAC or PC or Ubuntu.
 - MAC: https://docs.docker.com/desktop/install/mac-install
 - Ubuntu: https://docs.docker.com/desktop/install/ubuntu
 
-### 1- Create a Docker image from the Dockerfile
-    $ docker build -f Dockerfile.ubuntu -t vortex
+### 1- Build a Docker Image from the Dockerfile
+    $ docker build --platform=linux/amd64 -t vortex-packaged -f Dockerfile.prod .
 
-### 2- Build the Docker image
-    $ docker docker run -it vortex /bin/bash
+### 2- Construct and run a Container from the Docker Image
+    $ docker run -it --name vortex --privileged=true --platform=linux/amd64 vortex-packaged
 
-### 3- Build the project
+### 3- Build the Project
 One you login the Docker terminal, you will be in the build directory.
 
     $ make -s
 
-### 4- Run a simple test
+### 4- Run a Simple Test
+See `docs/` to learn more!
 
-    $ ./ci/blackbox.sh --cores=2 --app=vecadd
+    $ ./ci/blackbox.sh --cores=2 --app=vecadd
+
+### 5- Exit the Container
+    
+    $ exit
+    $ docker stop vortex
+
+### 6- Restart and Re-Enter the Container
+If you ran step `2` and then step `5` then, you have to start and re-enter the container
+    
+    $ docker start vortex
+    $ docker exec -it vortex
+
+---
+Note: Apple Silicon macs will run the container in emulation mode, so compiling and running will take a considerable amount of time -- but it still works!