https://blog.stephenmarz.com/2022/06/30/msi/ Skip to content Toggle navigation * Support me on Patreon * My Github RISC-V Is Getting MSIs! Home / RISC-V Is Getting MSIs! Operating Systems, Rust RISC-V Is Getting MSIs! By Stephen Marz June 30, 2022 Contents 1. Overview 2. Repository 3. Message Signaled Interrupts (MSI) 4. Incoming MSI Controller (IMSIC) 5. Conclusion 6. What's Next --------------------------------------------------------------------- Overview Message signaled interrupts or MSIs describe a way to signal an interrupt without a dedicated interrupt request pin (IRQ). One of the most prevalent uses for MSIs is the PCI bus, and the PCI specification defines the MSI and MSI-X standards. The potential benefits may include: (1) reduced number of direct wires from the device to the CPU or interrupt controller, (2) improve signaling performance, and (3) improving guest/host signaling for virtualized environments. --------------------------------------------------------------------- Repository All of the code for this post can be found here: https://github.com/ sgmarz/riscv_msi. The code is written for RV32I in Rust. I originally wrote it for RV64GC, but everything else I wrote is also for RV64GC, so I figured I should branch out and broaden my horizons. The AIA manual can be found here: https://github.com/riscv/riscv-aia. --------------------------------------------------------------------- Message Signaled Interrupts (MSI) An MSI is triggered by a "message", which is a fancy term for a "memory write". In fact, we can trigger a message by simply dereferencing an MMIO pointer. let message = 0xdeadbeef; // QEMU's 'virt' machine attaches the M-mode IMSIC for HART 0 to 0x2400_0000 write_volatile(0x2400_0000 as *mut u32, message); The code above writes to the MMIO address 0x2400_0000, which is where QEMU's virt machine connects the M-mode IMSIC for HART 0. The S-mode IMSIC for HART 0 is connected to 0x2800_0000. Each HART is a page away from each other, meaning the M-mode IMSIC for HART 1 is at 0x2400_1000, and the S-mode IMSIC for HART 1 is 0x2800_1000. IMSICs were added to QEMU's virt machine fairly recently, so you may mean to clone and build your own QEMU. QEMU's repository can be found here: https://github.com/qemu. After a device writes a word to a specific MMIO address, the interrupt is triggered. This means that devices do not need a wire connecting it to an IRQ controller, such as RISC-V's platform-level interrupt controller, or PLIC. Instead, as long as the device can make a memory write, it can trigger an interrupt. Even though triggering a message is that simple, we need a mechanism to enable and prioritize these messages. There might be some circumstances where we don't want to hear certain messages. This is where the incoming MSI controller or IMSIC comes into play. --------------------------------------------------------------------- Incoming MSI Controller (IMSIC) To be able to support MSIs, some device needs to be able to take memory writes and turn them into interrupts. Furthermore, the device needs to provide a mechanism to enable/disable and to prioritize interrupts just like a regular interrupt controller. This is done by the incoming MSI controller (IMSIC) device. WARNING: The Advanced Interrupt Architecture (AIA) manual is still a work-in-progress, and already, there have been major changes that removed or added CSRs or other pertinent information. So, some of the code and tables might be outdated. The register mechanism for the IMSIC consists of several control and status registers (CSRs) as well as internal registers accessibly through a selection mechanism. Newly Added Registers The AIA defines several new CSRs separated between the machine and supervisor modes. Register Register Description Name Number MISELECT 0x350 Machine register select SISELECT 0x150 Supervisor register select MIREG 0x351 A R/W view of the selected register in MISELECT SIREG 0x151 A R/W view of the selected register in SISELECT MTOPI 0xFB0 Machine top-level interrupt STOPI 0xDB0 Supervisor top-level interrupt MTOPEI 0x35C Machine top-level external interrupt (requires IMSIC) STOPEI 0x15C Supervisor top-level external interrupt (requires IMSIC) New CSRs defined for the AIA. The registers MISELECT and MIREG allow us to select a register by writing its number into the MISELECT register. Then the MIREG will represent the selected register. For example, if we read from MIREG, we read from the selected register, and if we write to MIREG, we write to the selected register. There are four selectable registers. There are machine and supervisor versions of these registers. For example, if we write to SISELECT, we will view the supervisor version of the register. Register Name MISELECT/ Description SISELECT EIDELIVERY 0x70 External Interrupt Delivery Register EITHRESHOLD 0x72 External Interrupt Threshold Register EIP0 through 0x80 through External Interrupt Pending EIP63 0xBF Registers EIE0 through 0xC0 through External Interrupt Enable EIE63 0xFF Registers Registers selectable by MISELECT/SISELECT and readable/writeable via MIREG/SIREG. The first thing we need to do is enable the IMSIC itself. This is done through a register called EIDELIVERY for "enable interrupt delivery" register. This register may contain one of three values: Value Description 0 Interrupt delivery is disabled 1 Interrupt delivery is enabled 0x4000_0000 Optional interrupt delivery via a PLIC or APLIC Value written to EIDELIVERY register Enabling the IMSIC So, we need to write 1 (Interrupt delivery is enabled) into the EIDELIVERY register: // First, enable the interrupt file // 0 = disabled // 1 = enabled // 0x4000_0000 = use PLIC instead imsic_write(MISELECT, EIDELIVERY); imsic_write(MIREG, 1); The EITHRESHOLD register creates a threshold that interrupt priorities must be before it can be heard. If an interrupt has a priority less than the value in EITHRESHOLD, it will be "heard" or unmasked. Otherwise, it will be masked and will not be heard. For example, an EITHRESHOLD of 5 would only permit messages 1, 2, 3, and 4 to be heard. The message 0 is reserved to mean "no message". Since a higher threshold opens more messages, messages with a lower number have a higher priority. // Set the interrupt threshold. // 0 = enable all interrupts // P = enable < P only imsic_write(MISELECT, EITHRESHOLD); // Only hear 1, 2, 3, and 4 imsic_write(MIREG, 5); Interrupt Priorities The AIA documentation uses the message itself as the priority. So, a message of 1 has a priority of 1, whereas a message of 1234 has a priority of 1234. This is more convenient since we can control messages directly. However, since each message number has associated enable and pending bits, there is a limit to the highest numbered interrupt. The specification has a maximum of \(32\times64 - 1 = 2,047\) total messages (we subtract one to remove 0 as a valid message). Enabling Messages The EIE register controls whether a message is enabled or disabled. For RV64, these registers are 64-bits wide, but still take up two adjacent register numbers. So, for RV64, only even numbered registers are selectable (e.g., EIE0, EIE2, EIE4, ..., EIE62). If you try to select an odd numbered EIE, you will get an invalid instruction trap. This took me many hours to figure out even though the documentation states this is the desired behavior. For RV32, the EIE registers are only 32-bits, and EIE0 through EIE63 are all selectable. The EIE register is a bitset. If the bit for a corresponding message is 1, then it is unmasked and enabled, otherwise it is masked and is disabled. For RV64, messages 0 through 63 are all in EIE0[63:0]. The bit is the message. We can use the following formulas to determine which register to select for RV64: // Enable a message number for machine mode (RV64) fn imsic_m_enable(which: usize) { let eiebyte = EIE0 + 2 * which / 64; let bit = which % 64; imsic_write(MISELECT, eiebyte); let reg = imsic_read(MIREG); imsic_write(MIREG, reg | 1 << bit); } RV32 behaves much the same, except we don't have to scale it by 2. // Enable a message number for machine mode (RV32) fn imsic_m_enable(which: usize) { let eiebyte = EIE0 + which / 32; let bit = which % 32; imsic_write(MISELECT, eiebyte); let reg = imsic_read(MIREG); imsic_write(MIREG, reg | 1 << bit); } With the code above, we can now enable the messages we want to hear. The following example enables messages 2, 4, and 10. imsic_m_enable(2); imsic_m_enable(4); imsic_m_enable(10); Pending Messages The EIP registers behave in the exact same way as the EIE registers except that a bit of 1 means that that particular message is pending, meaning a write to the IMSIC with that message number was sent. The EIP register is read/write. If we read from it, we can determine which messages are pending. If we write to it, we can manually trigger an interrupt message by writing a 1 to the corresponding message bit. // Trigger a message by writing to EIP for Machine mode in RV64 fn imsic_m_trigger(which: usize) { let eipbyte = EIP0 + 2 * which / 64; let bit = which % 64; imsic_write(MISELECT, eipbyte); let reg = imsic_read(MIREG); imsic_write(MIREG, reg | 1 << bit); } Testing Now that we can enable the delivery as well as indiviual messages, we can trigger them one of two ways: (1) write the message directly to the MMIO address or (2) set the interrupt pending bit of the corresponding message to 1. unsafe { // We are required to write only 32 bits. write_volatile(imsic_m(hartid) as *mut u32, 2) } imsic_m_trigger(4); Message Traps Whenever an unmasked message is sent to an enabled IMSIC, it will come to the specified HART as an external interrupt. For the machine-mode IMSIC, this will come as asynchronous cause 11, and for the supervisor-mode IMSIC, this will come as asynchronous cause 9. When we receive an interrupt due to a message being delivered, we will need to "pop" off the top level pending interrupt by reading from the MTOPEI or STOPEI registers depending on the privilege mode. This will give us a value where bits 26:16 contain the message number and bits 10:0 contain the interrupt priority. Yes, the message number and message priority are the same number, so we can choose either. // Pop the top pending message fn imsic_m_pop() -> u32 { let ret: u32; unsafe { // 0x35C is the MTOPEI CSR. asm!("csrrw {retval}, 0x35C, zero", retval = out(reg) ret), } // Message number starts at bit 16 ret >> 16 } My compiler does not support the names of the CSRs in this specification, so I used the CSR number instead. That is why you see 0x35C instead of mtopei, but they mean the same thing. When we read from the MTOPEI register (0x35C), it will give us the message number of the highest priority message. The instruction csrrw in the code snippet above will atomically read the value of the CSR into the return register and then store the value zero into the CSR. When we write zero into the MTOPEI register (0x35C), we are telling the IMSIC that we are "claiming" that we are handling the topmost message, which clears the EIP bit for the corresponding message number. /// Handle an IMSIC trap. Called from `trap::rust_trap` pub fn imsic_m_handle() { let msgnum = imsic_m_pop(); match msgnum { 0 => println!("Spurious message (MTOPEI returned 0)"), 2 => println!("First test triggered by MMIO write successful!"), 4 => println!("Second test triggered by EIP successful!"), _ => println!("Unknown msi #{}", v), } } Message 0 is not a valid message since when we pop from the MTOPEI, a 0 signals "no interrupt". Example Output If you run the repo with a new QEMU, you should see the following after a successful test. [image-1024x160] --------------------------------------------------------------------- Conclusion In this post, we saw the new registers added to support the incoming MSI controller (IMSIC) for RISC-V. We also enabled the IMSIC delivery, the individual messages, and handled two ways of sending a message: via MMIO directly or by setting the corresponding EIP bit. Finally, we handled the interrupt from the trap. --------------------------------------------------------------------- What's Next? The second part of the AIA manual includes the new Advanced Platform-Level Interrupt Controller or APLIC. We will examine this system as well as write drivers to start looking at how this new APLIC can signal using wires or messages. After the APLIC, we will write a driver for a PCI device and use it to send MSI-X messages. Visit me on Patreon Become a Patron! Visit me on Github GitHub --------------------------------------------------------------------- (c) 2022 - Stephen Marz | WordPress Theme By A WP Life | Powered by WordPress.org