https://lwn.net/SubscriberLink/1002371/0ff2be6a2c7624ca/ LWN.net Logo LWN .net News from the source LWN * Content + Weekly Edition + Archives + Search + Kernel + Security + Events calendar + Unread comments + ------------------------------------------------------------- + LWN FAQ + Write for us User: [ ] Password: [ ] [Log in] | [Subscribe] | [Register] Subscribe / Log in / New account Process creation in io_uring [LWN subscriber-only content] By Jonathan Corbet December 20, 2024 Back in 2022, Josh Triplett presented a plan to implement a "spawn new process" functionality in the io_uring subsystem. There was a fair amount of interest at the time, but developers got distracted, and the work did not progress. Now, Gabriel Krisman Bertazi has returned with a patch series updating and improving Triplett's work. While interest in this functionality remains, it may still take some time before it is ready for merging into the mainline. A new process in Linux is created with one of the variants of the clone() system call. As its name suggests, clone() creates a copy of the calling process, running the same code. Much of the time, though, the newly created process quickly calls execve() or execveat() to run a different program, perhaps after performing a bit of cleanup. There has long been interest in a system call that would combine these operations efficiently, but nothing like that has ever found its way into the Linux kernel. There is a posix_spawn() function, but that is implemented in the C library using clone() and execve(). Arguably, part of the problem is that, while the clone()-to-execve() pattern is widespread, the details of what happens between those two calls can vary quite a bit. Some files may need to be closed, signal handling changed, scheduling policies tweaked, environment adjusted, and so on; the specific pattern will be different for every case. posix_spawn() tries to provide a general mechanism to specify these actions but, as can be seen by looking at the function's argument list, it quickly becomes complex. Io_uring, meanwhile, is primarily thought of as a way of performing operations asynchronously. User space can queue operations in a ring buffer; the kernel consumes that buffer, executes the operations asynchronously, then puts the results into another ring buffer (the "completion ring") as each operation completes. Initially, only basic I/O operations were supported, but the list of operations has grown over the years. At this point, io_uring can be thought of as a sort of alternative system-call interface for Linux that is inherently asynchronous. Nobody covers the Linux kernel like LWN; be in the know with a one-month trial subscription, no credit card needed. An important io_uring feature, for the purposes of implementing something like posix_spawn(), is the ability to create chains of linked operations. When the kernel encounters a chain, it will only initiate the first operation; the next operation in the chain will only run after the first completes. The failure of an operation in a chain will normally cause all remaining operations to be canceled, but a "hard link" between two operations will cause execution to continue regardless of the success of the first of the two. Linking operations in this way essentially allows simple programs to be loaded into the kernel for asynchronous execution; these programs can run in parallel with any other io_uring operations that have been submitted. The new patch set creates two new io_uring operations, each with some special semantics. The first of those is IORING_OP_CLONE, which causes the creation of a new process to execute any operations that follow in the same chain. In a difference from a full clone() call, though, much of the calling task's context is unavailable to the process created by IORING_OP_CLONE. Without that context, io_uring operations in the newly created process can no longer be asynchronous; every operation in the chain must complete immediately, or the chain will fail. In practice, that means that operations like closing files can be executed, but complicated I/O operations are no longer possible. Krisman hopes to be able to at least partially lift that constraint in the future. Once the chain completes, the new process will be terminated, with one important exception: if it invokes the second new operation, IORING_OP_EXEC, which performs the equivalent of an execveat() call, replacing the running program with a new executable. At this point, the new process is completely detached from the original, is running its own program, and the processing of the io_uring chain is complete; the process will, rather than being terminated, go off to run the new program. Placing any other operations after IORING_OP_EXEC in the chain usually makes no sense; any operations after a successful IORING_OP_EXEC will be canceled. It also does not make sense to use IORING_OP_EXEC in any context other than a new process created with IORING_OP_CLONE, so that usage is not allowed. There is one case where it can be useful to link operations into the chain after IORING_OP_EXEC -- efficiently implementing a path search in the kernel. Often, the execution of a new program involves searching for it in a number of directories, usually specified by the PATH environment variable. One way of doing this in the io_uring context, as shown in this test program, is to enqueue a series of IORING_OP_EXEC operations, each trying a different location in the path. If hard links are used to chain these operations, execution will continue past failed operations until the one that actually finds the target program succeeds; after that, any subsequent operations will be discarded. The entire search runs in the kernel, without the need to repeatedly switch between kernel and user space. Most of the comments on the proposal so far have come from Pavel Begunkov, who has expressed some concerns about it. He did not like some aspects of the implementation, the special quirks associated with IORING_OP_CLONE and the process it creates, and the use of links, "`which already a bad sign for a bunch of reasons'" (he did not specify what the reasons are). He suggested that io_uring might not be the best place for this functionality; perhaps a list of operations could be passed to a future version of clone() instead, mirroring how the posix_spawn() interface works. Krisman answered that combining everything into a single system call would add complexity while making the solution less flexible. Io_uring makes it easy to put together a set of operations to be run in the kernel in an arbitrary order. The hope is to increase the set of possible operations over time, enabling the implementation of complex logic for the spawning of a new task. It is hard to see how combining all of this functionality into a single system call could work as well. In any case, this is early-stage work; getting it to a point where it can be considered for the mainline will require smoothing a number of the rough edges and reducing the number of limitations. It will also certainly require wider review; this work is proposing a significant addition to the kernel's user-space ABI that would have to be supported indefinitely. The developers involved will surely want to get the details right before committing to that support. Index entries for this article Kernel io_uring [Send a free link] ----------------------------------------- [Log in] to post comments BPF! Posted Dec 20, 2024 16:32 UTC (Fri) by willy (subscriber, #9762) [ Link] (6 responses) Clearly the right solution is to load a BPF program into the kernel to do the clone and setup. /s in case it wasn't clear. [Reply to this comment] BPF! Posted Dec 20, 2024 17:47 UTC (Fri) by gutschke (subscriber, #27910) [Link] (3 responses) I am not even sure the "/s" is warranted. clone()/exec() is a very powerful pattern that nicely fits in with how POSIX has designed its API. The ability to customize the newly launched process prior to loading the binary is crucial in a lot of scenarios. And I don't see that going away. But ever since the advent of threads (and possibly even in the presence of signals), this has gotten incredibly difficult to do correctly. There are just too many subtle race conditions that involve hidden state in the various run-time libraries or even in the dynamic link loader. If there was a way to do everything that you can currently do with systemcalls from userspace, but it instead moved entirely into the kernel, most of these problems would immediately go away. So, I see a lot of value with being able to call clone() and exec() from a BPF program, or maybe from io_uring. The elephant in the room with BPF is that this new API would then likely be limited to privileged processes. You can approximate a solution in userspace by very carefully picking what system calls you invoke, and by avoiding any calls into libc, including accidental calls into the dynamic link loader. This involves some amount of assembly code to get 100% reliable. It's very tedious and extremely fragile. It is often not worth the effort and instead you have to live with the occasional random crash. In some cases, a possible work-around is to launch a "zygote" helper process that executes before any threads are created. The latter is difficult to ensure though, as some libraries create threads when they are loaded into memory. [Reply to this comment] BPF! Posted Dec 20, 2024 19:05 UTC (Fri) by Cyberax ( supporter , # 52523) [Link] (1 responses) > clone()/exec() is a very powerful pattern that nicely fits in with how POSIX has designed its API. The ability to customize the newly launched process prior to loading the binary is crucial in a lot of scenarios. And I don't see that going away. POSIX's API is badly designed. clone() creates a copy of the entire VM and then just discards it. It's a lot of uselessly wasted work. A better API would create an "empty shell" suspended process, then the calling process can poke it (using FD-based APIs), and finally un-suspend it. There's a strange aversion in Linux/UNIX land to this model (it's too sane), so we get closer and closer to it with these kinds of workarounds. [Reply to this comment] BPF! Posted Dec 20, 2024 19:49 UTC (Fri) by epa (subscriber, #39769) [Link ] It's not only wasted work, but it makes it hard not to overcommit memory (at least in the case of a full fork()). If a process with a gigabyte of address space forks, requiring a gigabyte of free memory is far too cautious if it will exec() shortly afterwards, yet if you assume it always exec()s you will get caught out if the child process starts to use the memory you promised it. [Reply to this comment] BPF! Posted Dec 20, 2024 19:33 UTC (Fri) by magfr (subscriber, #16052) [ Link] I have been intrigued by the BeOS variant since I first saw it. They have some variant of posix_spawn which always can be called and they also have fork/exec but only allows those system calls in single threaded environments. To further mess with people this clone abstraction isn't strong enough to handle all cases - I have a little variation on tee which forks, sets up the child as a daemon process which does the writing, and then execs in the parent in order to keep the parent/child link with the grandparent. (The child terminates on end of input) [Reply to this comment] BPF! Posted Dec 20, 2024 17:50 UTC (Fri) by edeloget (subscriber, #88392) [Link] (1 responses) Right now, the commands really look like a set of instructions which are executed by a specific in-kernel VM, so my guess is more that with enough time, the complexity of the subsystem will grow enough to warrant the creation of an "io uring language" of some sort. Which will /then/ be interpreted by a BPF program :) [Reply to this comment] BPF! Posted Dec 20, 2024 18:08 UTC (Fri) by adobriyan (subscriber, #30858) [Link] > in-kernel VM It will be incomplete until it is possible to create new uring with uring interface __attribute__((sarcasm)). [Reply to this comment] Why not just have a one-step spawn? Posted Dec 20, 2024 18:44 UTC (Fri) by jbills (subscriber, #161176) [ Link] Dumb question: why can't we just have a single step function that starts a new process with a clean state without needing to do a whole load of operations in that process's context? Other operating systems get away with process creation without a magic dance. [Reply to this comment] Copyright (c) 2024, Eklektix, Inc. Comments and public postings are copyrighted by their creators. Linux is a registered trademark of Linus Torvalds