https://popovicu.com/posts/789-kb-linux-without-mmu-riscv/ Skip to content Uros Popovic * Posts * Tags * About * * --------------------------------------------------------------------- Go back 789 KB Linux Without MMU on RISC-V Posted on:October 4, 2023 | at 10:00 AM Follow @popovicu94 In this guide, we'll build a very tiny Linux kernel, weighing in at 789 K, and requiring no MMU support. We'll write some userspace code and this will be deployed on a virtual RISC-V 64-bit machine, without MMU, and we'll run some tiny programs of our own. As a reminder, please go through the guide for a micro Linux distro to understand the concepts behind what we're doing today: building the kernel, initramfs, etc. This guide is basically a continuation of that one and an exercise in making an absolutely minimal Linux deployment for (in theory) extremely cheap hardware. Like before, there's very little here that is specific to RISC-V, I just want to stay consistent with my previous guides. This exercise should be easily repeatable for other architectures too (though x86 may be somewhat sticky). Table of contents Open Table of contents * MMU and Linux + Brief history of uClinux * Challenges with MMU-less Linux + Executable file format + Pointers are dangerous (obviously!) + vfork instead of fork * Building binaries for an MMU-less kernel + Getting the toolchain + Building the toolchain + Building bFLT files * Building an extremely tiny Linux kernel * Building an initramfs image for the tiny MMU-less kernel * Slightly more complicated build: verifying system calls and multiprocessing * Conclusion * GitHub repo MMU and Linux An important piece of hardware when running an operating system is an MMU: memory management unit. This unit oversees memory accesses from a running CPU and translates them to different physical addresses. The reason why this is hugely important is because that is how the memory is virtualized, and the MMU is heavily used by the operating systems to implement a virtual address space. What this means is that the kernel typically enables the applications to not worry about how other running applications occupy the memory on a machine. Each process has an illusion of complete ownership of the memory, and the kernel facilitates that by virtualizing the memory. In other words, 2 processes on the same machine may use a memory location 0x12345678 and what they'd really be hitting inside the physical memory could be locations 0xAABBCCDD and 0xDDCCBBAA, respectively. The MMU, once set up by the kernel, will route the 0x12345678 accesses from each one of these applications to the relevant end physical address. Your typical Linux build virtualizes the address space and uses the MMU for this, and while we didn't explicitly set it in the previous guide, MMU was used and the memory contents of one process were insulated from the other processes. Now, as useful as MMU is, not all machines have it. The super simple budget-friendly microcontrollers typically do not have it, and the memory addresses coming out of the program running are actually what is used to access the memory. In this case, if a process accesses 0x12345678, it will go to the address 0x12345678 indeed. Brief history of uClinux As Linux has been rising in popularity since its inception, there have been attempts to run it on pretty much any digital device. This includes popular microcontrollers that do not have an MMU attached. Enter uClinux. The proper name for uClinux is actually mClinux, with a Greek 'mu', and so mC stands for 'microcontroller'. We spell it as uClinux for ASCII-friendly simplicity. The uClinux effort, per my understanding, was running independently for some time, before the decision has been made to mainline it, meaning it is now part of the Linux source code itself. Basically, what this means for us is that uClinux is now a set of configurations in the kernel build config, and we'll get our MMU-less build easily. Challenges with MMU-less Linux Before we get our hands dirty, I would like to highlight a few challenges with running Linux in this way. I will from this point refer to our Linux as MMU-less Linux, rather than uClinux to avoid confusion with the old project that has since been mainlined. Executable file format First, ELF binaries will not work anymore. I have not been able to build a MMU-less flavor of Linux for ELF, and I strongly believe it is impossible. I'm not sure if something about the ELF format explicitly assumes a virtual address space necessarily, I don't think it does, but it just seems impossible to load an ELF binary for running. What MMU-less Linux likes are the bFLT binaries. FLT stands for flat, and this initially made me believe that there really is no file format here, that we're supposed to just dump the machine instructions into a bare file, but this really isn't the case. bFLT has some structure, which is far simpler than ELF, but it's structured nonetheless. I personally found it very difficult to find any good documentation on what this file format really looks like, and really the only useful page I have found on the Internet around it is this page from someone's personal blog. It's a bit interesting to think that if this blog goes offline, there really isn't anything out there left except the actual source code in the Linux codebase, and I'm sure you agree that's not the most elegant way to learn. Pointers are dangerous (obviously!) It should be obvious, but I'll still call it out -- MMU is not routing our memory accesses now; whatever we access is really what we hit in the physical memory and this means that bad pointers are now very dangerous. You could easily corrupt the memory contents of another process or even the kernel itself. This would be a great place to introduce something like Rust programming, but given that it was tricky to build bare C programs for this kind of a platform (more about it below), I would now just say it's too early for that. vfork instead of fork The fork system call does not work on an MMU-less Linux since that heavily depends on virtualizing the memory, and this may lead some people to believe that you can run only one process on a MMU-less Linux, but that is not correct, and we will show it later -- multiprocessing is definitely possible. The way you achieve multiprocessing is by simply using vfork instead of fork. I'll keep things simple here and say that the difference between vfork and fork is that the parent process is sleeping until the child exits or calls execve. Please note that unless you exec into another binary from the child process, the child is running in the same memory space. Again, there is no MMU to help us out here. Building binaries for an MMU-less kernel As we mentioned before, we need to build bFLT binaries rather than ELFs. Figuring out how to exactly do this is what took most of my time in this exercise. I'll skip a lot of details and give you the end result, and I'm sure if you dig around why this is the end result, you'll be able to quickly figure out what's going on. Here it is: the simplest way to build bFLT binaries is with a uClibc toolchain. uClibc is basically a very slimmed down version of the C standard library, suitable for embedded systems. Of course, nothing is stopping you from using it outside of this context, for example, in a full blown desktop Linux distrbution. Quick note: which flavor of the standard C library is the best is another frequent topic of debate. Getting the toolchain I strongly suggest building the uClibc toolchain from source. It's very easy, since it is based on Buildroot. Therefore, let's head over to the Buildroot download page, fetch the latest release and unpack it. You can cd into the unpacked directory. tl;dr on what Buildroot is: it is a massive collection of Makefiles that is typically used to build a Linux distribution for an embedded system. You can use it to drive the kernel download + build as well, though I personally prefer to do it separately. What I typically use Buildroot for is to fetch the sources of common tools and build the initramfs cpio archive containing them. If you don't know what this means, please check the previous guide on making a micro Linux distro . You configure the Buildroot flow almost the same as you configure the Linux kernel build flow: you run make menuconfig, toggle a bunch of options and you're good to go. Buildroot is great because it has the scripts to pull and build a super wide range of common Linux tools and applications: busybox, ip, vi, python, Xorg server you name it. Sadly, this is as much as the uClibc toolchain page says. Personally, I was a bit puzzled after reading it: how do I use the toolchain and how do I build the binaries for uClinux? I'll cover that part right now. Building the toolchain Now that you have Buildroot downloaded, you can build the uClibc toolchain, which basically means you'll have a GCC set of tools to compile with against it. I was always under the impression that changing which standard library you want is a matter of just providing a different standard library to any GCC compiler, but after some digging online, my understanding now is that the GCC needs to be specifically compiled against it somehow. It's a bit beyond my understanding at the moment, to be perfectly honest. 1. As mentioned above, you can do make menuconfig from your unpacked Buildroot directory. 2. Go to Target options, and select RISCV as the architecture. 3. Target architecture size should be 64-bit. 4. Unselect the MMU support option. 5. Target Binary Format should be FLAT. 6. Target ABI is fine as lp64d. 7. Go back to the main menu and head over to the Toolchain section. 8. C library should be uClibc-ng. 9. Select Compile and install uClibc utilities. 10. Save and exit. You should be all set up to build the toolchain. From the Buildroot directory, run the following: make sdk This could take a while. Once it's done, you should have your new toolchain under output/host/bin/ within the Buildroot directory. Go ahead and ls that directory and you'll see a bunch of stuff built in there, including your well known GCC tools. All we'll be using is riscv64-buildroot-linux-uclibc-gcc, so make sure you hold on to that one. Building bFLT files bFLT files are, per my understanding, derived from the ELF files, actually. You use your GCC to get an ELF file, and then you use a tool called elf2flt to construct the bFLT file. I have tried building elf2flt from source and using it, but I had a hard time getting the build to work in the first place. After many attempts, I gave up, but I was happy to realize that Buildroot provides elf2flt. In fact, when you ls'ed the directory for your toolchain just a minute ago, you could see something like riscv64-buildroot-linux-uclibc-elf2flt in that directory. Once I realized this, I thought it was easy and I could just build a regular RISC-V ELF and convert it to bFLT with this tool -- turns out it's not that straightforward. I lost hope I'd ever get this done, and then I tried having Buildroot build the whole initramfs for me. I was skeptical it would run properly with the MMU-less Linux, but it did, so that means there really is a way to do this properly and quickly. My hack was to look at the Buildroot logs for how it built some of the packages an observe what are the GCC flags it uses. It turns out that I was missing the linker flag -Wl,-elf2flt=-r to make things work. This makes sure that the elf2flt is involved during the linking process and in the end, I got my bFLT file. I never had to run elf2flt myself. We'll see a concrete detailed example later as we build a sample application. First, let's build a MMU-less kernel that we have something to run against. Building an extremely tiny Linux kernel As this is an exercise in minimalism, we'll build an extremely minimal Linux kernel. It will offer little more than a basic filesystem and running binaries on top of the kernel. At this point, I assume you know how to build a Linux kernel in general. If you don't, please look at the micro Linux distro guide and familiarize yourself with the basic concepts before proceeding. For simplicity, I'll simply get the tar file from kernel.org for stable version 6.5.5, though I believe if you see something newer as the latest, you can just go for it and nothing should change. Let's download and unpack that. Here and below: my CROSS_COMPILE prefix may be different from what you need on your machine to invoke the build tools. Let's begin by setting up a super minimal Linux configuration: make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- tinyconfig This tinyconfig really strips down the build to the most basic things. It doesn't even have printk support, so you won't be getting a whole lot of output by booting this. More accurately, it doesn't even have the TTY support, so you really can't even output anything through the standard methods like printk, and you wouldn't be able to display something through UART or otherwise via standard output in the user space. Let's, however, build this (I run make ARCH=riscv CROSS_COMPILE= riscv64-linux-gnu- -j16) and see what we get out of it. I'm getting the following: * arch/riscv/boot/Image weighs in at 943 K. * arch/riscv/boot/Image.gz weighs in at 559 K. This is good to know and we'll use it as the baseline as we make the changes. Let's do the following and make some changes to our build: make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- menuconfig The most important change we want to make is we want to disable the MMU facilities in the kernel. I've seen guides that say you can just drop a config line into the .config file because they don't see the option to disable the MMU from menuconfig, but I would advise against it. I was able to get the option to appear, and I'm not sure if what they're expericing is architecture specific, but for RISC-V, I was definitely able to get the option to disable the MMU stuff. 1. CONFIG_NONPORTABLE should be set to y. We are building for a specific machine and we want to enable non-portable builds as we know exactly what are the memory addresses we'd be deploying to. 2. CONFIG_MMU: n. It should be obvious what this does. Note, we were unable to flip this to n before doing the step 1; we should be all good now. 3. PHYS_RAM_BASE_FIXED: y. This means we'll be deploying the kernel to a specific address in the memory. 4. Once you flip that, you can set CONFIG_PHYS_RAM_BASE to 0x80000000. This makes perfect sense for us as this is where our execution will begin on the QEMU virtual machine. If you want to know more about it, please read the bare metal guide and the SBI and boot process guide. Those guides are lengthy, but they will enable you to fully understand what's going on. That said, for the latter guide, please note that we are not using OpenSBI to boot here. We're running a very light Linux, and we are not relying on the SBI infrastructure. This is much closer to bare metal programming than running Linux as you're used to it. Let's build this really quick and see if there are any changes. Save and exit, and run: make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- -j16 Let's weigh our kernel again: * arch/riscv/boot/Image weighs in at 789 K. * arch/riscv/boot/Image.gz weighs in at 472 K. Wow, it's significantly lighter! However, even though we disabled the MMU smartness from it, we still haven't enabled any features and again, the tinyconfig is so tiny that it barely does anything. Let's flip some more configurations: 1. CONFIG_BLK_DEV_INITRD: y. We need an initramfs image to be able to run our init process and start off some magic in the user space. Flip that to y. 2. The above will by default enable various compression support for the initramfs, but you can go ahead and disable them all. I will not list out all of them one by one (a single example is CONFIG_RD_GZIP), they should expand in menuconfig as y right under the option for enabling initramfs. You can flip them all to n, we don't want to beef up our kernel image with compression algorithms now. 3. CONFIG_BINFMT_FLAT: y. We need to add support for bFLT binaries. If you look at the same menu, there's no mention of ELF: as soon as you disabled the MMU support, you lost the ELF binary support as well. We already knew this, so it's not an issue, and as promised, we'll talk about how to build bFLTs below on a concrete example. Let's save and exit and build again: make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- -j16 * arch/riscv/boot/Image weighs in at 789 K. * arch/riscv/boot/Image.gz weighs in at 476 K. The only difference from the previous step is that the compressed image is slightly thicker, but it's still tiny! At this point, we have what we need to run an init process in user space! Building an initramfs image for the tiny MMU-less kernel We'll have a useless init that prints a message to UART and just goes to sleep. Notice an inconsistency? I said we'll print to UART, but our tinyconfig has no UART drivers, no TTY, nothing. So how could this possibly work? Well, since we're without an MMU, we can really target any physical address on the system. If you go through the bare metal guide, you'll learn that the QEMU system we'll be using to run this image has an UART device mapped out at 0x10000000. We'll be printing to UART through this address for two reasons: 1. Most importantly, to illustrate that on MMU-less kernel you can access any physical address. This can be both good and bad, probably more bad and not only in terms of stability and being resilient to bugs, but also in terms of security if your system accepts any sort of user input. 2. We want to achieve extreme minimalism here in terms of the size of the build and see how far we can push the boundaries on how much we can slim the kernel down. Right now, it's under 800 K, which is pretty cool. Bringing in drivers for TTY, UART, etc. would certaily add some weight to the size of our image. That said, here's our C program: #include #include volatile char *UART = (char*) 0x10000000; void print_to_uart(char *message) { for (int i = 0; i < strlen(message); i++) { *UART = message[i]; } } int main() { print_to_uart("Hello world! Welcome to the Tiny Linux MMU-less kernel!\n"); while (1) { sleep(1000); } return 0; } Notice that we have an infinite loop: the init process is not supposed to ever really exit or the kernel falls into a panic (not that you would see it without printk, though). Now is the time to compile this init to a bFLT file. This is where we'll invoke the uClibc toolchain we had previously built. I run the following on my machine, and you only need to adjust the relevant paths to reproduce: uros@uros-debian-desktop:/tmp/tiny/init$ /tmp/buildroot/buildroot-2023.02.5/output/host/bin/riscv64-buildroot-linux-uclibc-gcc -fPIC -Wl,-elf2flt=-r -Wall -static -o init init.c * -fPIC is used to produce position independent code. This is crucial for understanding how bFLT binaries work on an MMU-less Linux. Since we're directly going for the physical addresses, but we don't know where exactly in memory our binary would be loaded, we can't depend on any absolute address in our binary. If we did, we risk (and most likely will) hurting the memory of another process or the kernel itself. We simply can't make any assumptions about the end memory addresses. Therefore, everything needs to be PC-relative in our code. If you don't know what this means, please look this up online as it's extremely important. tl;dr is that every memory access must be made with an offset relative to the CPU's pointer to the current machine instruction, instead of accessing a hardcoded address. In other words, instead of accessing 0x1001 directly, we access with an offset of 1 if our program counter is at 0x1000. This will still work if the binary loader places us at 0x2000 instead of 0x1000; the former wouldn't! Again, if you do not understand this concept, please review CPU addressing modes. * -Wl,-elf2flt=-r is the magic flag we really needed from the toolchain. This ensures that elf2flt is invoked and that the end binary is able to relocate, meaning the loader can place it somewhere differently in memory and the binary should still work (owing to the -fPIC flag as well). * The rest of the flags should be familiar. This binary weighs in at onl 3.3 K, which is pretty light, especially for a statically linked binary. uClibc delivered on its promise of being light. Quick note here is that elf2flt also dropped a file called init.gdb which is an ELF file that you can use to do your objdump and whatnot with it to debug. If you run file init, you should see something like this: init: BFLT executable - version 4 ram gotpic This is what we needed. Let's make a file called file_list.txt with a single line saying init and let's make the initramfs image for this setup. cpio -o -H newc < file_list.txt > initramfs.cpio Now that we have the kernel image and the initramfs.cpio file, we can fire this up in QEMU and see what we get. uros@uros-debian-desktop:/tmp/linux/linux-6.5.5$ qemu-system-riscv64 -machine virt -cpu rv64,mmu=false -kernel /tmp/tiny/linux-6.5.5/arch/riscv/boot/Image -bios none -initrd /tmp/tiny/init/initramfs.cpio -nographic Hello world! Welcome to the Tiny Linux MMU-less kernel! Awesome, we ran userspace code and even depended on a dangerous pointer to do something useful. Slightly more complicated build: verifying system calls and multiprocessing Instead of starting from a tinyconfig and working my way up, this time I start with the typical defconfig. From that point, I disabled a bunch of things like MMU (of course), networking, virtualization, etc. while retaining TTY, UART drivers and other useful goodies. Of course, nothing stops you from beginning with tinyconfig and adding incrementally, I just wanted to save myself some time. This time I ended up with images of these sizes: * arch/riscv/boot/Image weighs in at 4.1 M. * arch/riscv/boot/Image.gz weighs in at 2.1 M. This is all much heavier than previously, but it comes with a bunch of cool features like ability to dump core, UART drivers, filesystem support, etc. The message I'm trying to send here is that kernel is full of amazing drivers and features that you can start taking advantage of by only adjusting your kernel build configuration. I ran the same initramfs image as above, and got much richer output with a lot of debug messages from the kernel: uros@uros-debian-desktop:/tmp/linux/linux-6.5.5$ qemu-system-riscv64 -machine virt -cpu rv64,mmu=false -kernel /tmp/linux/linux-6.5.5/arch/riscv/boot/Image -bios none -initrd /tmp/tiny/init/initramfs.cpio -nographic [ 0.000000] Linux version 6.5.5 (uros@uros-debian-desktop) (riscv64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #9 Wed Oct 4 00:07:07 PDT 2023 [ 0.000000] Machine model: riscv-virtio,qemu [ 0.000000] Zone ranges: [ 0.000000] DMA32 [mem 0x0000000080000000-0x0000000087ffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000080000000-0x0000000087ffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x0000000087ffffff] [ 0.000000] riscv: base ISA extensions acdfim [ 0.000000] riscv: ELF capabilities acdfim [ 0.000000] Kernel command line: [ 0.000000] Dentry cache hash table entries: 16384 (order: 5, 131072 bytes, linear) [ 0.000000] Inode-cache hash table entries: 8192 (order: 4, 65536 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 32320 [ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off [ 0.000000] Memory: 124576K/131072K available (2526K kernel code, 684K rwdata, 809K rodata, 141K init, 276K bss, 6496K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 [ 0.000000] riscv-intc: 64 local interrupts mapped [ 0.000000] plic: plic@c000000: mapped 53 interrupts with 1 handlers for 2 contexts. [ 0.000000] clint: clint@2000000: timer running at 10000000 Hz [ 0.000000] clocksource: clint_clocksource: mask: 0xffffffffffffffff max_cycles: 0x24e6a1710, max_idle_ns: 440795202120 ns [ 0.000062] sched_clock: 64 bits at 10MHz, resolution 100ns, wraps every 4398046511100ns [ 0.003318] Console: colour dummy device 80x25 [ 0.003525] printk: console [tty0] enabled [ 0.006416] Calibrating delay loop (skipped), value calculated using timer frequency.. 20.00 BogoMIPS (lpj=40000) [ 0.006531] pid_max: default: 32768 minimum: 301 [ 0.007085] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear) [ 0.007131] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear) [ 0.019370] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1. [ 0.024447] devtmpfs: initialized [ 0.028181] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns [ 0.028380] futex hash table entries: 256 (order: 1, 12288 bytes, linear) [ 0.045655] clocksource: Switched to clocksource clint_clocksource [ 0.060880] Unpacking initramfs... [ 0.064002] workingset: timestamp_bits=62 max_order=15 bucket_order=0 [ 0.066160] io scheduler mq-deadline registered [ 0.066228] io scheduler kyber registered [ 0.113049] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled [ 0.119589] 10000000.uart: ttyS0 at MMIO 0x10000000 (irq = 4, base_baud = 230400) is a 16550A [ 0.120520] printk: console [ttyS0] enabled [ 0.134884] goldfish_rtc 101000.rtc: registered as rtc0 [ 0.135476] goldfish_rtc 101000.rtc: setting system clock to 2023-10-04T19:51:49 UTC (1696449109) [ 0.148278] sysfs: cannot create duplicate filename '/kernel/slab/:a-0000016' [ 0.148656] CPU: 0 PID: 1 Comm: swapper Not tainted 6.5.5 #9 [ 0.148979] Hardware name: riscv-virtio,qemu (DT) [ 0.149245] Call Trace: [ 0.149490] [<0000000080003230>] dump_backtrace+0x1c/0x24 [ 0.150053] [<000000008026c1ac>] show_stack+0x2c/0x38 [ 0.150297] [<0000000080271822>] dump_stack_lvl+0x20/0x32 [ 0.150546] [<0000000080271848>] dump_stack+0x14/0x1c [ 0.150784] [<000000008012b4d4>] sysfs_warn_dup+0x52/0x66 [ 0.151040] [<000000008012b590>] sysfs_create_dir_ns+0xa8/0xba [ 0.151331] [<0000000080252640>] kobject_add_internal+0x90/0x1ca [ 0.151638] [<0000000080252878>] kobject_init_and_add+0x50/0x84 [ 0.151937] [<00000000800cabe2>] sysfs_slab_add+0x102/0x1d4 [ 0.152216] [<0000000080283388>] slab_sysfs_init+0x8a/0xf6 [ 0.152451] [<000000008027959c>] do_one_initcall+0x64/0x11e [ 0.152736] [<0000000080279806>] kernel_init_freeable+0x158/0x1b0 [ 0.153051] [<00000000802726a2>] kernel_init+0x1c/0xea [ 0.153319] [<0000000080001cde>] ret_from_fork+0xa/0x1c [ 0.153728] kobject: kobject_add_internal failed for :a-0000016 with -EEXIST, don't try to register things with the same name in the same directory. [ 0.154345] SLUB: Unable to add boot slab kmalloc-rcl-8 to sysfs [ 0.156569] sysfs: cannot create duplicate filename '/kernel/slab/:0000016' [ 0.156885] CPU: 0 PID: 1 Comm: swapper Not tainted 6.5.5 #9 [ 0.157159] Hardware name: riscv-virtio,qemu (DT) [ 0.157378] Call Trace: [ 0.157500] [<0000000080003230>] dump_backtrace+0x1c/0x24 [ 0.157809] [<000000008026c1ac>] show_stack+0x2c/0x38 [ 0.158071] [<0000000080271822>] dump_stack_lvl+0x20/0x32 [ 0.158354] [<0000000080271848>] dump_stack+0x14/0x1c [ 0.158620] [<000000008012b4d4>] sysfs_warn_dup+0x52/0x66 [ 0.158899] [<000000008012b590>] sysfs_create_dir_ns+0xa8/0xba [ 0.159212] [<0000000080252640>] kobject_add_internal+0x90/0x1ca [ 0.159518] [<0000000080252878>] kobject_init_and_add+0x50/0x84 [ 0.159815] [<00000000800cabe2>] sysfs_slab_add+0x102/0x1d4 [ 0.160101] [<0000000080283388>] slab_sysfs_init+0x8a/0xf6 [ 0.160399] [<000000008027959c>] do_one_initcall+0x64/0x11e [ 0.160693] [<0000000080279806>] kernel_init_freeable+0x158/0x1b0 [ 0.161010] [<00000000802726a2>] kernel_init+0x1c/0xea [ 0.161278] [<0000000080001cde>] ret_from_fork+0xa/0x1c [ 0.161598] kobject: kobject_add_internal failed for :0000016 with -EEXIST, don't try to register things with the same name in the same directory. [ 0.162289] SLUB: Unable to add boot slab kmalloc-8 to sysfs [ 0.162761] SLUB: Unable to add boot slab alias ep_head to sysfs [ 0.163061] SLUB: Unable to add boot slab alias blkdev_ioc to sysfs [ 0.164939] Legacy PMU implementation is available [ 0.165494] clk: Disabling unused clocks [ 0.178717] Freeing unused kernel image (initmem) memory: 140K [ 0.178974] This architecture does not have kernel memory protection. [ 0.179296] Run /init as init process Hello world! Welcome to the Tiny Linux MMU-less kernel! Let's write an init that starts a bunch of other processes and verifies that multiprocessing keeps working after they're spawned. init.c below: #include #include #include int main(int argc, char *argv[]) { printf("Hello world\n"); for (int i = 0; i < 3; i++) { pid_t pid; if ((pid = vfork()) < 0) { fprintf(stderr, "Could not fork a worker at iteration %d\n", i); exit(1); } if (pid == 0) { // Child process char *args[2] = { "worker", NULL }; execv("/worker", args); } } while (1) { sleep(5); printf("Hello from init\n"); } return 0; } worker.c below: #include #include int main(int argc, char *argv[]) { while (1) { printf("Hello from worker!\n"); sleep(3); } return 0; } We build them the same way as the init we built before and package them into a cpio archive. Running this in QEMU for some time gives the following: uros@uros-debian-desktop:/tmp/linux/linux-6.5.5$ qemu-system-riscv64 -machine virt -cpu rv64,mmu=false -kernel /tmp/linux/linux-6.5.5/arch/riscv/boot/Image -bios none -initrd /tmp/linux/init/initramfs.cpio -nographic [ 0.000000] Linux version 6.5.5 (uros@uros-debian-desktop) (riscv64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #9 Wed Oct 4 00:07:07 PDT 2023 [ 0.000000] Machine model: riscv-virtio,qemu [ 0.000000] Zone ranges: [ 0.000000] DMA32 [mem 0x0000000080000000-0x0000000087ffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000080000000-0x0000000087ffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x0000000087ffffff] [ 0.000000] riscv: base ISA extensions acdfim [ 0.000000] riscv: ELF capabilities acdfim [ 0.000000] Kernel command line: [ 0.000000] Dentry cache hash table entries: 16384 (order: 5, 131072 bytes, linear) [ 0.000000] Inode-cache hash table entries: 8192 (order: 4, 65536 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 32320 [ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off [ 0.000000] Memory: 124544K/131072K available (2526K kernel code, 684K rwdata, 809K rodata, 141K init, 276K bss, 6528K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 [ 0.000000] riscv-intc: 64 local interrupts mapped [ 0.000000] plic: plic@c000000: mapped 53 interrupts with 1 handlers for 2 contexts. [ 0.000000] clint: clint@2000000: timer running at 10000000 Hz [ 0.000000] clocksource: clint_clocksource: mask: 0xffffffffffffffff max_cycles: 0x24e6a1710, max_idle_ns: 440795202120 ns [ 0.000059] sched_clock: 64 bits at 10MHz, resolution 100ns, wraps every 4398046511100ns [ 0.003249] Console: colour dummy device 80x25 [ 0.003442] printk: console [tty0] enabled [ 0.006336] Calibrating delay loop (skipped), value calculated using timer frequency.. 20.00 BogoMIPS (lpj=40000) [ 0.006447] pid_max: default: 32768 minimum: 301 [ 0.007026] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear) [ 0.007070] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear) [ 0.019398] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1. [ 0.024599] devtmpfs: initialized [ 0.028455] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns [ 0.028661] futex hash table entries: 256 (order: 1, 12288 bytes, linear) [ 0.046105] clocksource: Switched to clocksource clint_clocksource [ 0.061575] Unpacking initramfs... [ 0.064941] workingset: timestamp_bits=62 max_order=15 bucket_order=0 [ 0.066491] io scheduler mq-deadline registered [ 0.066556] io scheduler kyber registered [ 0.073683] Freeing initrd memory: 32K [ 0.114913] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled [ 0.121398] 10000000.uart: ttyS0 at MMIO 0x10000000 (irq = 4, base_baud = 230400) is a 16550A [ 0.122456] printk: console [ttyS0] enabled [ 0.134615] goldfish_rtc 101000.rtc: registered as rtc0 [ 0.135109] goldfish_rtc 101000.rtc: setting system clock to 2023-10-04T21:01:14 UTC (1696453274) [ 0.147392] sysfs: cannot create duplicate filename '/kernel/slab/:a-0000016' [ 0.147670] CPU: 0 PID: 1 Comm: swapper Not tainted 6.5.5 #9 [ 0.147955] Hardware name: riscv-virtio,qemu (DT) [ 0.148164] Call Trace: [ 0.148375] [<0000000080003230>] dump_backtrace+0x1c/0x24 [ 0.148861] [<000000008026c1ac>] show_stack+0x2c/0x38 [ 0.149056] [<0000000080271822>] dump_stack_lvl+0x20/0x32 [ 0.149257] [<0000000080271848>] dump_stack+0x14/0x1c [ 0.149435] [<000000008012b4d4>] sysfs_warn_dup+0x52/0x66 [ 0.149626] [<000000008012b590>] sysfs_create_dir_ns+0xa8/0xba [ 0.149837] [<0000000080252640>] kobject_add_internal+0x90/0x1ca [ 0.150048] [<0000000080252878>] kobject_init_and_add+0x50/0x84 [ 0.150284] [<00000000800cabe2>] sysfs_slab_add+0x102/0x1d4 [ 0.150502] [<0000000080283388>] slab_sysfs_init+0x8a/0xf6 [ 0.150695] [<000000008027959c>] do_one_initcall+0x64/0x11e [ 0.150887] [<0000000080279806>] kernel_init_freeable+0x158/0x1b0 [ 0.151102] [<00000000802726a2>] kernel_init+0x1c/0xea [ 0.151291] [<0000000080001cde>] ret_from_fork+0xa/0x1c [ 0.151561] kobject: kobject_add_internal failed for :a-0000016 with -EEXIST, don't try to register things with the same name in the same directory. [ 0.152043] SLUB: Unable to add boot slab kmalloc-rcl-8 to sysfs [ 0.153186] sysfs: cannot create duplicate filename '/kernel/slab/:0000016' [ 0.153443] CPU: 0 PID: 1 Comm: swapper Not tainted 6.5.5 #9 [ 0.153632] Hardware name: riscv-virtio,qemu (DT) [ 0.153778] Call Trace: [ 0.153873] [<0000000080003230>] dump_backtrace+0x1c/0x24 [ 0.154056] [<000000008026c1ac>] show_stack+0x2c/0x38 [ 0.154247] [<0000000080271822>] dump_stack_lvl+0x20/0x32 [ 0.154440] [<0000000080271848>] dump_stack+0x14/0x1c [ 0.154614] [<000000008012b4d4>] sysfs_warn_dup+0x52/0x66 [ 0.154816] [<000000008012b590>] sysfs_create_dir_ns+0xa8/0xba [ 0.155035] [<0000000080252640>] kobject_add_internal+0x90/0x1ca [ 0.155256] [<0000000080252878>] kobject_init_and_add+0x50/0x84 [ 0.155481] [<00000000800cabe2>] sysfs_slab_add+0x102/0x1d4 [ 0.155691] [<0000000080283388>] slab_sysfs_init+0x8a/0xf6 [ 0.155906] [<000000008027959c>] do_one_initcall+0x64/0x11e [ 0.156119] [<0000000080279806>] kernel_init_freeable+0x158/0x1b0 [ 0.156342] [<00000000802726a2>] kernel_init+0x1c/0xea [ 0.156525] [<0000000080001cde>] ret_from_fork+0xa/0x1c [ 0.156744] kobject: kobject_add_internal failed for :0000016 with -EEXIST, don't try to register things with the same name in the same directory. [ 0.157236] SLUB: Unable to add boot slab kmalloc-8 to sysfs [ 0.157663] SLUB: Unable to add boot slab alias ep_head to sysfs [ 0.157902] SLUB: Unable to add boot slab alias blkdev_ioc to sysfs [ 0.159238] Legacy PMU implementation is available [ 0.159650] clk: Disabling unused clocks [ 0.172328] Freeing unused kernel image (initmem) memory: 140K [ 0.172645] This architecture does not have kernel memory protection. [ 0.172948] Run /init as init process Hello world Hello from worker! Hello from worker! Hello from worker! Hello from worker! Hello from worker! Hello from worker! Hello from init Hello from worker! Hello from worker! Hello from worker! Hello from worker! Hello from worker! Hello from worker! Hello from init Hopefully this clears up the confusion about whether multiprocessing is possible on a no-MMU kernel build. Additionally, we verified that the system calls work properly. Conclusion We managed to build a very light Linux kernel for a system without an MMU. By building a custom uClibc toolchain we also managed to build bFLT binaries to run in the userspace. System calls and multiprocessing worked well and so we have a fully functional system. We pushed the boundaries of a lightweight deployment through tinyconfig, and ended up running a userspace process that writes to UART by directly writing to the UART device, bypassing the kernel. If all we want is to multi-task a bit on our device, Linux is likely an overkill, even in a tiny deployment like this. Additionally, if we're not leveraging any drivers from the kernel code base, it's probably another indicator we're going too heavy. I leave it to you to decide if this lightweight Linux deployment makes sense for your usecase or not. We could use Buildroot to add some of the well known tools to our system, potentially even make something interactive. I'll stop here, however, as my goal was to bring up the absolutely minimal set up. Please connect with me and let me know if there is a way to make the set up I described above even lighter from the tinyconfig-based one. I think this finally answers my question that I've had for a long time which is: what is the absolutely minimal Linux kernel? Something that runs on a single core, extremely simple, with as little cruft as possible, but that can still process some system calls and provide some sort of a filesystem. If anyone has been looking for this answer as well, I hope I am providing it correctly here. The way I initially got to this question was I wanted to implement a minimal machine (in simulation or otherwise in FPGA) that can run something useful like some sort of a minimal Linux, and that I can program with at least some standard tooling like GCC. After studying my options, I think RISC-V and Linux are the answer, and the last few guides I have written are the summary of my studies. GitHub repo As always, I'll be posting the code from above to GitHub as well. There won't be Makefiles due to the need for a custom toolchain, and I'd really like the readers to go through this writeup and understand what's going on exactly in order to build bFLT libraries. It stumped me for a few days and I hope this helps anyone with the same questions. The repo can be found here. * risc-v * linux * no-mmu * uclinux * uclibc * buildroot --------------------------------------------------------------------- Copyright (c) 2023 | All rights reserved.