This is where I plan to store all of my notes and documentation regarding my RISC-V computer build. Each section of this will have a boiled-down script-like version available in this repository as well, although this file will be the main source of information until the scripts have been built. # Configuration - Mainboard: [Milk-V Jupiter](https://milkv.io/jupiter) with SpacemiT M1-x processor (Original retail price) [^1] - RAM: 16GB of LPDDR4X (Soldered) - GPU: [Gigabyte GTX 1050Ti](https://www.techpowerup.com/gpu-specs/gigabyte-gtx-1050-ti-oc.b3909) (Used, ~$50 off eBay) - Power Supply: FPS SFX Pro 450W 80Plus Bronze PSU (something cheap off amazon) - Case: [SGPC K49 Case](https://www.sgpcdesign.com/products/sgpc-itx-mini-case-k49steel) (Something cheap off amazon) - CPU Cooler: Cheap generic heatsink off amazon; not strictly necessary I would recommend having a >16GB SD card available, a >16GB NVMe drive, and a USB-to-UART serial adapter for early boot observation and interruption when/if necessary. # Basics: Booting the Board So, let's start with the basics. This board ships with 2 manufacturer-supported compatible linux distributions; Bianbu OS (based on Ubuntu), and Ubuntu Linux 23.10. Ubuntu 23.10 is EOL, so that's no good. Bianbu OS is a Chinese-first distribution, developed by SpacemiT. Firstly, I personally don't trust the longevity and security of board-manufacturer linux distributions, and secondly, I can't read hanzi (chinese characters). So, already at square one, we're stuck. Luckily, Milk-V is not the only manufacturer of devices containing the SpacemiT K1/M1 family of processors. Other implementations include the BananaPi BPI-F3 single-board computer, the SpacemiT MuseBook laptop, and the DC ROMA II laptop. The DC ROMA II, which I also have, natively supports Ubuntu 23.10, Ubuntu 24.04, and Debian sid/trixie. Debian [officially supports the RISC-V architecture](https://riscv.org/ecosystem-news/2023/07/debian-gnu-linux-is-now-officially-supported-on-the-risc-v-architecture/), so for the remainder of this discussion I will be using Debian. So, all that needs doing now is to flash the ISO to an NVMe drive, install it, and boot the device right? Well, not quite so fast. You can just flash the ISO to an *SD* card, but not an NVMe. For how I've done this, you'll actually need both. For reasons that [are documented by Bianbu](https://git.blizzard.systems/bianbu-repo-mirror/bianbu-linux-docs/src/branch/main/en/device/boot.md) but not well understood by me, the partition structure the board wants is... not as you'd expect. For more information, see the aforementioned link, but the short version is, we need to flash the board with either `fastboot`, or their distributed tool [Titanflasher](https://github.com/milkv-jupiter/jupiter-tools/releases/tag/titanflasher). The fastboot commands are documented, and will be added to this repository, but I couldn't get it to play nicely, so I would recommend using the Titanflasher tool. Titanflasher has an auto-updater, and I'd also recommend that since the released version in the Github doesn't make it nearly as easy to change the language, and again defaults to Chinese. So the first real step is downloading the Ubuntu 23.10 release from [Milk-V's github page](https://github.com/milkv-jupiter/jupiter-ubuntu-build/releases). We'll use this to flash the onboard NOR storage with the appropriate uboot version and configuration. You will also need the [DC ROMA II Debian image](https://drive.google.com/file/d/14QySBh36lR1JhwN1AoZgiBQrxX6HsKGA/view?usp=drive_link). Yes, it's a Google Drive link, it's what DC Roma distributes at time of writing. Flash the Debian image to an SD card using your tool of choice (BalenaEtcher is nice if `dd` makes you nervous), and use Titanflasher to burn to the board storage (this will be documented in a separate document). Because the Jupiter board puts SD card as higher priority than NVMe, it should just boot to SD card if both devices are installed. The board boots as soon as power is given, so no need to press the power button. The board should boot properly to Debian Trixie, although this will take some time. The GUI will also show a complaint about not having much space left for storage; you can fix this if you want be resizing the partition, but it'll make the next step faster if you don't. We don't really care what's going on here, we just want to copy everything out onto the NVMe. The default login for this image is `roma` for both the user and password. Open a terminal, either in the GUI or over the serial connection, and run the following to transfer the Debian install over to the NVMe. ```bash sudo dd if=/dev/mmcblk0p5 of=/dev/nvme0n1p5 status=progress sudo dd if=/dev/mmcblk0p6 of=/dev/nvme0n1p6 status=progress ``` This will copy Debian over to the NVMe drive, along with the boot partition and it's appropriate configuration. Once both of these commands finish, you can resize the partitions on the NVMe drive using something like `cfdisk` and `resize2fs`. For some reason, `fdisk` doesn't seem to play nicely with this, and resizing doesn't always work, so keep the SD card handy in case it refuses to boot later. turn off the device, remove the SD card, and boot the device again. If all goes according to plan, you should end up with the same setup as before, but now it's on an NVMe drive. # New Kernel: Catching up Now that we're booted and into linux, the very first thing we need to do is fix the distro's identity crisis. The image is shipped as `sid/trixie`, as `trixie` wasn't released when it was built. However, rather than using `trixie` as the distro name in `/etc/apt/sources.list`, the developers opted to use `sid` instead. Change this back to `trixie`, so things don't potentially break as we move forward. We now need to get an updated version of the kernel. Make sure you have network access (wifi won't work right now, we'll get to that in a bit), then download the [SpacemiT kernel](https://gitee.com/bianbu-linux/linux-6.6). According to [this forum post](https://forum.spacemit.com/t/topic/399), Spacemit seems to be working to upstream as much as possible, and are tracking their upstreaming efforts in [this Github wiki](https://github.com/spacemit-com/linux/wiki). As documented [here by Bianbu](https://bianbu-linux.spacemit.com/en/faqs/), you will also need to download the [Real-Time CPU firmware](https://gitee.com/bianbu-linux/buildroot-ext/tree/bl-v2.0.y/board/spacemit/k1/target_overlay/lib/firmware) (you're looking for `esos.elf`), and it will need to be added into either the linux kernel build, `/lib/firmware`, or both. While you are waiting for that to download, you should also install `linux-firmware` from the package manager. You can also use the `linux-firmware` git repository hosted [on the kernel git page](https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware) if you'd prefer. You will also need to install the dependencies required for kernel development, as documented by Milk-V: ```bash sudo apt-get install flex bison libncurses-dev debhelper libssl-dev u-boot-tools libpfm4-dev libpfm4-dev libtraceevent-dev asciidoc ``` Once you have everything downloaded and installed, go to the kernel folder. Start with `make k1_defconfig`, to build a base config file for your board. From here, build the kernel with `make -j$(nproc)` to build the image and DeviceTree files. The final files you want are in the following locations: - Kernel: `arch/riscv/boot/Image{.gz{.itb}}` - DeviceTree files: `arch/riscv/boot/dts/spacemit/k1-x_milkv-jupiter.dtb` Note that you can either use the `k1-x` or `m1-x` DeviceTree files. --- If you need/want to use wifi, be sure to also run the following, in the same kernel directory: ``` make modules && sudo make modules_install && sudo make headers_install ``` Then, run the following to add the wifi drivers to the kernel's list of modules to load: ``` echo "8852bs" | tee -a /etc/modules-load.d/modules.conf ``` You can also try building in the driver directly into the kernel, but at time of writing this fails due to multiple definitions of several functions. --- Copy your kernel and DeviceTree into the `/boot` partition, so they can be booted from. Then, copy `/boot/env_k1-x.txt` to a filename you can remember. This will be your backup for ~~when~~ if something goes wrong and you need to boot back to your old kernel. Then, edit the file to contain this information: ``` ramdisk_addr_r=0x21000000 fdt_addr_r=0x31000000 load_files=load nvme 0:5 ${kernel_addr_r} /Image.gz; load nvme 0:5 ${fdt_addr_r} /k1-x_milkv-jupiter.dtb; load nvme 0:5 ${ramdisk_addr_r} /initramfs-6.1.15+.img set_bootargs=setenv bootargs 'console=ttyS0,115200 root=UUID=ffafcae9-30cd-4b81-b4d2-23ad3ff5b340 rootfstype=ext4 rootwait rw earlycon clk_ignore_unused loglevel=7 swiotlb=131072 stmmaceth=chain_mode:1 selinux=0' bootcmd=run set_bootargs; run load_files; booti $kernel_addr_r $ramdisk_addr_r:$filesize $fdt_addr_r ``` `load_files` in order will: 1. Load the kernel from the file `/Image.gz`, located on the `nvme` drive `0`, partition `5`, into the address listed in `kernel_addr_r`. 2. Load the Flattened DeviceTree from the file `/k1-x_milkv-jupiter.dtb`, located on the `nvme` drive `0` partition `5`, into the address listed in `fdt_addr_r` 3. Load the initramfs from the file `/initramfs-6.1.15+.img`, located on `nvme` drive `0` partition `5`, into the address listed in `ramdisk_addr_r` `set_bootargs` adds boot arguments for: - `console=ttyS0,115200`: Open a console at ttyS0 with a baudrate of 115200 - `root=UUID=ffafcae9-30cd-4b81-b4d2-23ad3ff5b340`: Define where the root filesystem is located, by UUID. Ensure the UUID listed in your file matches the UUID of the root filesystem on the NVMe drive (this should be `/dev/nvme0n1p6`. - `rootfstype=ext4`: define what type filesystem is used for root - `rootwait`: Wait until root mounts before continuing the boot process - `rw`: Mount root as readable and writeable - `earlycon`: Open the console as early as possible - `clk_ignore_unused`: Clocks not explicity in use remain enabled, rather than disabling them - `loglevel=7`: Set's the kernel log level to be as verbose as possible, for boot issue diagnosis purposes - `swiotlb=131072`: This allows for using a temporary buffer of size of 131072 bits (16KiB) during DMA. This temporary buffer allows for accessing via DMA physical memory sizes in excess of 4GiB. More information can be found [here](https://docs.kernel.org/core-api/swiotlb.html). - `stmmaceth=chain_mode:1`: Informs the Ethernet driver that the ethernet ports should be working in chain mode (expected behaviour) rather than ring mode. - `selinux=0`: Explicitly disable SELinux U-Boot is configured to find this textfile, and execute the `bootcmd` portion, which runs the previous two commands before booting to the initramfs to begin the boot process. Once you've built and installed the kernel, and modified the `env` textfile, reboot the board. This will ensure that the basic build works properly. This isn't strictly necessary, but checking every step along the way is good practice. In this repo, there is a [config file](./kernel.config) that you can save and copy into your kernel repository, rename to `.config`, rebuild the kernel, and happily be on your way. There is also a `diff` available, showing the changes made from the default config for kernel version 6.6.63. In general, you need to disable the SpacemiT GPU, the PowerVR drivers that drive it, enable nouveau, and enable the NVidia framebuffer. Once you've rebuilt and installed all the bits and bobs just like before, reboot to check that everything still works. You should no longer have any display outputs, but you should be able to access the board over SSH and over serial. # NeoVim: Text Editor Nowadays, I use NeoVim for my text editing, and I really enjoy its behaviour ovarall. However, one thing I can't easily do is use it on RISC-V, because at time of writing, [LuaJIT does not support RISC-V](https://github.com/LuaJIT/LuaJIT/pull/1267). While I'm here, I'll document my process for installing NeoVim, for complete documentation of my system. Debian has a port for RISC-V, but it is lacking in many features, rather than building in LuaJIT as we will do here. Using the [fork of LuaJIT](https://github.com/plctlab/LuaJIT/tree/v2.1-riscv64-pr) which supports RISC-V, git clone, `make`, and `sudo make install`, just like any other from-source project. Once this is done, or while it's building in the background, clone [NeoVim](https://github.com/neovim/neovim). Checkout the stable branch, install the dependencies, and build the package without LuaJIT (it should be able to find the previously built and installed version mentioned above): ```bash git checkout stable && sudo apt-get install ninja-build gettext cmake unzip curl build-essential && make BUNDLED_CMAKE_FLAG="-DUSE_BUNDLED_LUAJIT=OFF" ``` Then, run `cd build && cpack -G DEB && sudo dpkg -i nvim-linux-riscv64.deb` to install using APT. NeoVim no longer supports this feature, and marks it as best effort, so expect this behaviour to change in the future, and require the same `make install` as for LuaJIT, but at time of writing for tag `v0.10.4` it works fine. # DeviceTree Modification Now, you'd think that the `swiotlb` command is enough for the GPU to at least register properly. However, unfortunately that's not true. If you watch the boot logs carefully, you'll see that `BAR`s don't get initialised properly. This may ring a bell for those familiar with recent GPU developments and "Resizeable BAR support". We have to do this BAR resizing by hand, which is tricky when not everything is necessarily documented succinctly. I ended up manually reassigning BAR space until I could find one that was big enough that didn't overlap anything important. This is done by editing the DeviceTree file, but not the one you'd expect. in `arch/riscv/boot/dts/spacemit/`, there are a myriad of DeviceTree Source (`.dts`) files, including some with the name of the board we're using. However, these all inherit from `k1-x.dtsi`, which is where the bulk of the declarations are made. We need to modify the `range` section for `pcie2` to increase the available register space to fit the requirements of the board. According to a discussion [on this Raspberry Pi CM4 forum](https://forums.raspberrypi.com/viewtopic.php?t=288902), we need to meet the following requirements: > * The size has to be a power of 2. > * The base address has to be aligned to the size. > * The range 0xfc000000-0xffffffff is taken for the other peripherals. Along with these, we have the additional requirements of - it can't overlap with existing devices - The amount of space available needs to be at least `0x18000000` (this is a requirement defined by the board if the BAR cannot be fully allocated as expected) Through trial and error, along with some help from `Opvolger`s patches on [his github](https://github.com/Opvolger/spacemit-k1-linux-6.6), I've determined that the modifications listed [in this patch file](./devicetree.patch) are all that's necessary to get all BARs initialised properly. A map of the implemented registers available to use and currently used registers are also available [in this repository](./devicetreeTesting.md), although this document should not be considered complete. Once you've modified the DeviceTree file, return to the root of your kernel build folder and rebuild the kernel, modules, and headers as mentioned previously, but ***do not install them***. Building should take significantly less time now, since the majority of the build is consistent. Everything from the Linux setup is ready, but the board isn't quite prepared yet. # x86 Emulation With this, the PCIe interface is now all set up for the GPU, and since the open-source `nouveau` driver is built into the kernel we should be all set to boot. However, if you try to boot right now, you will be met with a kernel panic. This is because of NVidia's design decisions following Kepler (GTX600 and GTX700 series GPUs). These devices come with an onboard UEFI Expansion ROM, also known as an Option ROM, for initialising the GPU. This ROM contains an executable binary which the CPU has to complete for the GPU to initialise properly, and as far as I am aware it has not been reverse-engineered or otherwise documented by open-source users. This is not a problem on traditional x86 systems, as it can simply be loaded and executed at boot time. However, as discussed in [this talk](https://www.youtube.com/watch?v=uxvAH1Q4Mx0) by Alexander Graf, other architectures such as 64-bit ARM have an issue trying to run this code, and will panic by themselves as the instructions won't make sense. Graf does have a solution for this though, and has a GitHub repo with documentation for how to go about building his implementation. That is, import the core-emulation features from QEMU into the board's UEFI implementation, and run the x86 code in the emulator during the boot process. This is a clever solution, and I highly recommend listening to the talk for the implementation details. As an additional bonus, Intel seems to have liked the idea enough to build and maintain [their own implementation](https://github.com/intel/MultiArchUefiPkg), which helpfully has been expanded to include RISC-V processors. So, we should be able to build the UEFI implementation with the proper extensions in place, and the board should be fine and dandy. But we have one more hurdle to jump through; the Milk-V Jupiter doesn't actually use a UEFI implementation for boot operations. Instead, it uses Das U-Boot, which has its own implementation for the boot protocol. This would be a roadblock that means that this entire project is dead in the water, if not for a lesser known feature of U-Boot; [the xPL framework](https://docs.u-boot.org/en/latest/develop/spl.html). This framework allows us to use U-Boot to load EDK2, which we can then use normally. Starfive Tech has built a version of EDK2, and has documented building U-Boot to support EDK2. Both of these are originally intended for the VisionFive2, which I'm using as a starting point. TODO: Document behaviour # Acknowledgements This was developed with the help of many various forum posts, along with the discussion and documentation from these specific people: - `opvolger`, for documenting the custom kernel process on his blog and [on GitHub](https://github.com/Opvolger/Opvolger/blob/master/milkVjupiter/OpenSUSEATIRadeonHD5850.md) - Ard Biesheuvel and Alexander Graf, for their work developing, documenting, and publicly discussing [u-Boot, UEFI, Grub](https://www.youtube.com/watch?v=qJAkJ3nmWgM), and [emulation in UEFI](https://www.youtube.com/watch?v=uxvAH1Q4Mx0) - Jeff Geerling, and the many contributors to the Raspberry Pi forums for their discussions of defining manually modifying DeviceTree for [expanding BAR space](https://forums.raspberrypi.com/viewtopic.php?t=288902) - Tianocore's publicly available draft documentation, last publicly edited and documented in 2020; links to come [^1]: The Milk-V Jupiter For context, the K1 and M1 are functionally identical silicon design; the K1 uses a plastic housing and is clocked slightly lower than the M1, which has a metal case for better thermal conductivity. They're also collectively referred to as the Stone series or Key Stone series, depending on which page of the SpacemiT website you read (this is likely a machine translation inconsistency, at a guess).