https://lupyuen.codeberg.page/articles/sg2000b.html Pick another theme! RISC-V Emulator for Sophgo SG2000 SoC (Pine64 Oz64 / Milk-V Duo S) * 1 Update the Memory Map * 2 auipc Overflow in Boot Code * 3 Change auipc to li in Boot Code * 4 Emulate the 16550 UART Controller * 5 Emulate the UART Output Registers * 6 Emulate the UART Input Registers * 7 UART Input triggers SegFault * 8 Increase TinyEMU Interrupts to 64 * 9 Emulate the SG2000 Peripherals * 10 Daily Automated Testing * 11 What's Next 7 Jul 2024 Pine64 Oz64 SBC running on Sophgo SG2000 SoC Earlier this year we made a RISC-V Emulator for Ox64 BL808 SBC. Every day we run it for testing the daily build of Apache NuttX RTOS, thanks to our customised TinyEMU RISC-V Emulator (not the small flightless bird)... * "Automated Testing with Ox64 BL808 Emulator" * Watch the Presentation (YouTube) Now that NuttX supports Sophgo SG2000 SoC: Let's create a similar emulator for Pine64 Oz64 SBC (pic above) and Milk-V Duo S... 1. We take TinyEMU Emulator for Ox64 BL808 SBC 2. Update the RISC-V Memory Map to match Sophgo SG2000 SoC 3. Fix the auipc Overflow in TinyEMU Boot Code 4. We emulate the 16550 UART Controller 5. By intercepting Reads and Writes to the UART I/O Registers 6. But TinyEMU supports only 32 Interrupts, we bump up to 64 7. Eventually we'll emulate SG2000 Peripherals like GPIO 8. Right now it's good enough for Daily Automated Testing of NuttX for SG2000 SG2000 Reference Manual (Page 17) SG2000 Reference Manual (Page 17) SS1 Update the Memory Map We begin with the TinyEMU RISC-V Emulator for Ox64 BL808 SBC, and we tweak it for Sophgo SG2000 SoC. This is how we update the RISC-V Memory Map for SG2000: riscv_machine.c // Base Addresss of System RAM, Core Local Interrupt Controller (unused) // And Platform-Level Interrupt Controller #define RAM_BASE_ADDR 0x80200000ul #define CLINT_BASE_ADDR 0x74000000ul #define PLIC_BASE_ADDR 0x70000000ul (NuttX boots at 0x8020_0000) (Interrupt Controller is at 0x7000_0000) Then we build and run TinyEMU Emulator for SG2000... ## Build SG2000 Emulator for macOS ## For Linux: See https://github.com/lupyuen/nuttx-sg2000/blob/main/.github/workflows/sg2000-test.yml#L29-L45 cd $HOME git clone https://github.com/lupyuen2/sg2000-emulator cd sg2000-emulator make \ CFLAGS="-I$(brew --prefix)/opt/openssl/include -I$(brew --prefix)/opt/sdl2/include" \ LDFLAGS="-L$(brew --prefix)/opt/openssl/lib -L$(brew --prefix)/opt/sdl2/lib" \ CONFIG_MACOS=y ## Build NuttX for SG2000 cd $HOME/nuttx tools/configure.sh milkv_duos:nsh make ## Omitted: Create the NuttX Image ## See https://lupyuen.codeberg.page/articles/sg2000#appendix-build-nuttx-for-sg2000 ## Boot TinyEMU with NuttX for SG2000 ## To Exit TinyEMU: Ctrl-A x wget https://raw.githubusercontent.com/lupyuen/nuttx-sg2000/main/nuttx.cfg $HOME/sg2000-emulator/temu nuttx.cfg (nuttx.cfg points to the NuttX Image) Something baffling appears... NuttX on TinyEMU crashes at 0xffffffff_80200000 SS2 auipc Overflow in Boot Code When we boot NuttX on TinyEMU, it crashes at a curious location (pic above)... $ sg2000-emulator/temu nuttx.cfg TinyEMU Emulator for Sophgo SG2000 SoC raise_exception2: cause=1 tval= 0xffffffff_80200000 pc= 0xffffffff_80200000 tinyemu: Illegal instruction, quitting What just happened? Our Emulator tried to execute the code at MTVAL 0xffffffff_80200000. And crashed because it's not a valid address! The Effy Address looks sus, it seems related to RAM Base Address: 0x80200000 Our Emulator is booting the wrong address? Apparently. We check the Original TinyEMU Boot Code: riscv_machine.c // Init the TinyEMU Boot Code void copy_bios(...) { ... // `q` points to the Boot Code q = (uint32_t *)(ram_ptr + 0x1000); // Load `RAM_BASE_ADDR` into Register T0: // `auipc t0, RAM_BASE_ADDR` // `RAM_BASE_ADDR` is 0x80200000 q[0] = 0x297 + RAM_BASE_ADDR - 0x1000; // Later: Jump to Register T0 To load the RAM Base Address into Register T0: TinyEMU tries to assemble this RISC-V Instruction (into our Boot Code)... auipc t0, 0x80200000 Maybe auipc has a problem? We verify with the RISC-V Online Assembler. When we assemble the auipc instruction above, the Online Assembler fails with an error (pic below)... Error: lui expression not in range 0..1048575 Error: value of 0000080200000000 too large for field of 4 bytes at 0000000000000000 Aha 0x8020_0000 is too big to assemble as an auipc Address! But 0x8020_0000 is a perfectly valid address? Remember that RISC-V is a RISC Platform after all. Some operations won't fit into 4 bytes of Machine Code. We upsize to 8 bytes of Machine Code... RISC-V Online Assembler SS3 Change auipc to li in Boot Code How else can we load 0x8020_0000 into Register T0? Instead of aupic (which has size limits), we load our RAM Base Address with the li Instruction... li t0, 0x80200000 When we feed the above into RISC-V Online Assembler, we see the resulting (8-byte) Machine Code... 4010029b addiw t0, zero, 1025 01529293 slli t0, t0, 0x15 That's because li is a Pseudo-Instruction that expands into two RISC-V Instructions... * addiw: Add zero to 1025 and store into T0 * slli: Shift-Left T0 by 0x15 bits * Producing: 1025 << 0x15 = 0x8020_0000 Thus we copy the above Machine Code into our TinyEMU Boot Code: riscv_machine.c // Init the TinyEMU Boot Code void copy_bios(...) { ... // Load `RAM_BASE_ADDR` into Register T0: // `li t0, 0x80200000` // Which is assembled as... q[pc++] = 0x4010029b; // addiw t0, zero, 1025 q[pc++] = 0x01529293; // slli t0, t0, 0x15 // TODO: Remove the hardcoding of 0x80200000 Our Emulator now boots NuttX correctly at 0x8020_0000! TinyEMU boots in RISC-V Machine Mode, but NuttX expects to boot in RISC-V Supervisor Mode So TinyEMU will jump directly to 0x8020_0000? Not quite. TinyEMU boots in RISC-V Machine Mode, but NuttX expects to boot in RISC-V Supervisor Mode! (Pic above) That's why we customised the TinyEMU Boot Code, so that it jumps from Machine Mode to Supervisor Mode via the MRET Instruction. (Pic below) (Which will start NuttX at 0x8020_0000) TinyEMU Boot Code jumps from Machine Mode to Supervisor Mode via the MRET Instruction SS4 Emulate the 16550 UART Controller Nothing appears when we boot NuttX? $ sg2000-emulator/temu nuttx.cfg TinyEMU Emulator for Sophgo SG2000 SoC [...crickets...] That's because we haven't emulated the 16550 UART Controller in TinyEMU! To figure out what's needed, we refer to the 16550 UART Driver in NuttX: uart_16550.c // To send one byte to UART Output... void u16550_send(struct uart_dev_s *dev, int ch) { ... // We write the byte to the 16550 UART Register... u16550_serialout( priv, // UART Device UART_THR_OFFSET, // UART Register: Transmit Holding Register (THR) ch // Byte to be sent ); } // To check if the UART Transmit FIFO is ready... bool u16550_txready(struct uart_dev_s *dev) { ... // We read the 16550 UART Register... return (( u16550_serialin( priv, // UART Device UART_LSR_OFFSET // UART Register: Line Status Register (LSR) ) & UART_LSR_THRE // And check the THRE Bit (Transmit Holding Register Empty) ) != 0); } Which says that... * UART_THR: Transmit Holding Register Will receive the byte that NuttX is transmitting * UART_LSR: Line Status Register Will be read by NuttX to check if the Transmit FIFO is Ready * UART_LSR_THRE: Transmit Holding Register Empty This is the bit in UART_LSR that will indicate whether Transmit FIFO is Ready Let's fix this in TinyEMU... Emulate the UART Output Registers SS5 Emulate the UART Output Registers How will we emulate the 16550 UART Registers in TinyEMU? When TinyEMU needs to Read or Write a Memory Address, it will call the functions below. This is how we Intercept the Memory Writes to emulate the UART Output Register (Transmit Holding): riscv_cpu.c // TinyEMU calls this function to execute Memory Writes int target_write_slow(...) { ... // If TinyEMU is writing to this address... switch(paddr) { // Address is UART Transmit Holding Register case UART0_BASE_ADDR + UART_THR_OFFSET: // Print the character that's written by NuttX char buf[1] = { val }; print_console(NULL, buf, 1); (UART Addresses are here) And this is how we Intercept the Memory Reads to emulate the UART Status Register (Line Status): riscv_cpu.c // TinyEMU calls this function to execute Memory Reads int target_read_slow(...) { ... // If TinyEMU is reading from this address... switch(paddr) { // Address is UART Line Status Register case UART0_BASE_ADDR + UART_LSR_OFFSET: // Always tell NuttX that // Transmit Holding Register is Empty ret = UART_LSR_THRE; // If UART Input is available: // Tell NuttX that Receive Data is Available if (read_input() != 0) { ret |= UART_LSR_DR; } (UART Addresses are here) (More about UART Input in a while) What happens when we run this? Now we see the NuttX Shell yay! $ sg2000-emulator/temu nuttx.cfg TinyEMU Emulator for Sophgo SG2000 SoC NuttShell (NSH) NuttX-12.5.1 nsh> NuttX Shell won't accept input, let's fix it... SG2000 Reference Manual (Page 636) SG2000 Reference Manual (Page 636) SS6 Emulate the UART Input Registers What about UART Input? How to emulate the UART Registers? Again we refer to the NuttX Driver for 16550 UART, to discover the inner workings of UART Input: uart_16550.c // NuttX Interrupt Handler for 16550 UART int u16550_interrupt(int irq, FAR void *context, FAR void *arg) { ... // Loop until no characters to be transferred for (passes = 0; passes < 256; passes++) { // Get the current UART Status status = u16550_serialin(priv, UART_IIR_OFFSET); // If no Pending Interrupts, exit if ((status & UART_IIR_INTSTATUS) != 0) { break; } // Handle the UART Interrupt switch (status & UART_IIR_INTID_MASK) { // If UART Input is available, // receive the Input Data case UART_IIR_INTID_RDA: uart_recvchars(dev); break; ... } // Receive one character from UART. // Called by the above Interrupt Handler. int u16550_receive(struct uart_dev_s *dev, unsigned int *status) { ... // Return the Line Status and Receive Buffer *status = u16550_serialin(priv, UART_LSR_OFFSET); rbr = u16550_serialin(priv, UART_RBR_OFFSET); return rbr; } Which says that (pic above)... * UART_IIR: Interrupt ID Register Should return INTID_RDA (data available) Followed by IIR_INTSTATUS (no more data) * UART_LSR: Line Status Register Should return LSR_DR (data available) Followed by 0 (no more data) * UART_RBR: Receiver Buffer Register Should return the UART Input Data Thus we emulate the above UART Registers in TinyEMU: riscv_cpu.c // TinyEMU calls this function to execute Memory Reads int target_read_slow(...) { ... // If TinyEMU is reading from this address... switch(paddr) { // Address is UART Interrupt ID Register case UART0_BASE_ADDR + UART_IIR_OFFSET: // If the Input Buffer is Empty: // Then Receive Data is NOT Available if (read_input() == 0) { ret = UART_IIR_INTSTATUS; } else { // Otherwise Receive Data is Available ret = UART_IIR_INTID_RDA; } break; // Address is UART Receive Buffer Register case UART0_BASE_ADDR + UART_RBR_OFFSET: // Return the Input Buffer ret = read_input(); // Clear the Input Buffer and UART Interrupt set_input(0); virtio_ack_irq(NULL); break; (virtio_ack_irq is here) What about UART_LSR? (Line Status Register) Check out our earlier implementation of target_read_slow. When we press a key: TinyEMU crashes with a Segmentation Fault SS7 UART Input triggers SegFault So UART Input is all hunky dory? Not quite. When we press a key: TinyEMU crashes with a Segmentation Fault (pic above)... $ sg2000-emulator/temu nuttx.cfg NuttShell (NSH) NuttX-12.5.1 nsh> [1] segmentation fault Maybe we got the Wrong Interrupt Number? We check our NuttX Config for UART Interrupt: nsh/defconfig ## NuttX IRQ for UART0 is 69 CONFIG_16550_UART0_IRQ=69 Which says... * RISC-V IRQ is 69 - 25 = 44 (25 is the NuttX IRQ Offset) * Which matches the SG2000 Reference Manual (Page 13)... "3.1 Interrupt Subsystem" "Int #44: UART0" * And matches the VirtIO IRQ in TinyEMU: riscv_machine.c // UART0 IRQ becomes VirtIO IRQ for TinyEMU #define VIRTIO_IRQ 44 Something seriously sinister is wrecking our rojak... TinyEMU supports only 32 RISC-V External Interrupts SS8 Increase TinyEMU Interrupts to 64 After lots of headscratching: We discover that TinyEMU supports only 32 RISC-V External Interrupts. Which is too few for our UART Controller! (IRQ 44) Here's the original definition in TinyEMU: riscv_machine.c // TinyEMU defines the RISC-V Virtual Machine typedef struct RISCVMachine { ... // Platform-Level Interrupt Controller: // Only 32 External RISC-V Interrupts! uint32_t plic_pending_irq; // 32 Pending Interrupts uint32_t plic_served_irq; // 32 Served Interrupts IRQSignal plic_irq[32]; // 32 Interrupt Signals ... } RISCVMachine; Therefore we increase the number of TinyEMU Interrupts from 32 to 64... * Increase the IRQs from 32 to MAX_IRQ * Set MAX_IRQ to 64 * Increase the IRQ Size from 32-bit to 64-bit Finally NuttX Shell runs OK yay! $ sg2000-emulator/temu nuttx.cfg TinyEMU Emulator for Sophgo SG2000 SoC virtio_console_init Patched DCACHE.IALL (Invalidate all Page Table Entries in the D-Cache) at 0x80200a28 Patched SYNC.S (Ensure that all Cache Operations are completed) at 0x80200a2c Found ECALL (Start System Timer) at 0x8020b2c6 Patched RDTIME (Read System Time) at 0x8020b2cc elf_len=0 virtio_console_resize_event ABC NuttShell (NSH) NuttX-12.5.1 nsh> uname -a NuttX 12.5.1 50fadb93f2 Jun 18 2024 09:20:31 risc-v milkv_duos nsh> (Complete Log is here) (OSTest works OK too) SG2000 Emulator seems slower than Ox64 BL808 Emulator? Yeah probably because SG2000 runs on MTIMER Frequency of 25 MHz. When we execute sleep 10, it completes in 25 seconds. We might need to adjust the TinyEMU System Timer. (CPU-bound Operations like getprime won't have this timing delay) Emulating the SG2000 GPIO Controller in TinyEMU SS9 Emulate the SG2000 Peripherals Where's the rest of our SG2000 Emulator? Yeah we need to emulate the SG2000 Peripherals: GPIO, I2C, SPI, ... Based on the SG2000 Reference Manual (Page 721): We'll probably emulate SG2000 GPIO Controller like this: riscv_cpu.c // TinyEMU calls this function to execute Memory Writes int target_write_slow(...) { ... // If TinyEMU is writing to this address... switch(paddr) { // Address is GPIOA Base Address // with Offset 0 (GPIO_SWPORTA_DR) case 0x03020000: // Check if GPIOA1 is Off or On (Bit 1) // `val` is the value written by NuttX to 0x03020000 const uint8_t gpio = 1; const uint32_t mask = (1 << gpio); const char b = ((val & mask) == 0) ? '0' : '1'; // Send an Emulator Notification to the Console: // {"nuttxemu":{"gpioa1":1}} char notify[] = "{\"nuttxemu\":{\"gpioa1\":0}}\r\n"; notify[strlen(notify) - 5] = b; print_console(NULL, notify, strlen(notify)); What's this nuttxemu? nuttxemu will be printed on the TinyEMU Console to notify the caller that GPIOA1 is set to On or Off. When we run TinyEMU in a Web Browser (via WebAssembly), our Web Browser can intercept this notification and visualise a Simulated LED on GPIOA1. (Pic above) We're running SG2000 Emulator for Daily Automated Testing at GitHub Actions SS10 Daily Automated Testing Why are we doing all this? 1. SG2000 Emulator will be helpful for testing NuttX Drivers and App , without a Real SBC 2. We're running SG2000 Emulator for Daily Automated Testing at GitHub Actions (pic above): sg2000-test.yml ## Build the SG2000 Emulator git clone https://github.com/lupyuen2/sg2000-emulator cd sg2000-emulator make ## Download the NuttX Daily Build for SG2000 date=2024-07-04 repo=https://github.com/lupyuen/nuttx-sg2000 release=$repo/releases/download/nuttx-sg2000-$date wget $release/Image wget $release/nuttx.hash ## Download the NuttX Test Script wget $repo/raw/main/nuttx.cfg wget $repo/raw/main/nuttx.exp ## Run the NuttX Test Script chmod +x nuttx.exp ./nuttx.exp Which calls this Expect Script to execute OSTest (and verify that everything is hunky dory): nuttx.exp #!/usr/bin/expect ## Expect Script for Testing NuttX with SG2000 Emulator ## Wait at most 300 seconds set timeout 300 ## For every 1 character sent, wait 0.01 milliseconds set send_slow {1 0.01} ## Start the SG2000 Emulator spawn ./temu nuttx.cfg ## Wait for the prompt and enter `uname -a` expect "nsh> " send -s "uname -a\r" ## Wait for the prompt and enter `ostest` expect "nsh> " send -s "ostest\r" ## Check the response... expect { ## If we see this message, exit normally "ostest_main: Exiting with status 0" { exit 0 } ## If timeout, exit with an error timeout { exit 1 } } (See the Automated Test Log) What about QEMU Emulator? QEMU Emulator is way too complex to customise for SG2000. Instead we're running QEMU Emulator for Daily Testing of NuttX QEMU. (Also based on GitHub Actions with Expect Scripting) Isn't it safer to run Daily Tests on a Real SG2000 SBC? Oh yes we're doing it too! (Pic below) Check out the article... "Daily Automated Testing for Milk-V Duo S RISC-V SBC (IKEA TRETAKT / Apache NuttX RTOS)" Daily Automated Testing for Milk-V Duo S RISC-V SBC (IKEA TRETAKT / Apache NuttX RTOS) SS11 What's Next Creating the SG2000 Emulator... Doesn't look so hard? Yeah I'm begging all RISC-V SoC Makers: Please provide a Software Emulator for your RISC-V SoC! Just follow the steps in this article to create your RISC-V Emulator. Some SoC Peripherals might be missing, but a Barebones Emulator is still super helpful for porting, booting and testing any Operating System. Many Thanks to my GitHub Sponsors (and the awesome NuttX Community) for supporting my work! This article wouldn't have been possible without your support. * Sponsor me a coffee * Discuss this article on Hacker News * My Current Project: "Apache NuttX RTOS for Sophgo SG2000" * My Other Project: "NuttX for Ox64 BL808" * Older Project: "NuttX for Star64 JH7110" * Olderer Project: "NuttX for PinePhone" * Check out my articles * RSS Feed Got a question, comment or suggestion? Create an Issue or submit a Pull Request here... lupyuen.github.io/src/sg2000b.md