https://lwn.net/SubscriberLink/907685/0290fbfe1ba855ea/ LWN.net Logo LWN .net News from the source LWN * Content + Weekly Edition + Archives + Search + Kernel + Security + Distributions + Events calendar + Unread comments + ------------------------------------------------------------- + LWN FAQ + Write for us User: [ ] Password: [ ] [Log in] | [Subscribe] | [Register] Subscribe / Log in / New account A pair of Rust kernel modules [LWN subscriber-only content] Welcome to LWN.net The following subscription-only content has been made available to you by an LWN subscriber. Thousands of subscribers depend on LWN for the best news from the Linux and free software communities. If you enjoy this article, please consider subscribing to LWN. Thank you for visiting LWN.net! By Jonathan Corbet September 12, 2022 --------------------------------------------------------------------- Kangrejos The idea of being able to write kernel code in the Rust language has a certain appeal, but it is hard to judge how well that would actually work in the absence of examples to look at. Those examples, especially for modules beyond the "hello world" level of complexity, have been somewhat scarce, but that is beginning to change. At the 2022 Kangrejos gathering in Oviedo, Spain, two developers presented the modules they have developed and some lessons that have been learned from this exercise. An NVMe driver Andreas Hindborg was up first to talk about an NVM Express driver written in Rust. The primary reason for this project, he said, was to take advantage of the memory-safety guarantees that Rust offers and to gain some real-world experience with the language. His conclusions from this project include that Rust comes with a lot of nice tooling and that its type system is helpful for writing correct code. It is, he said, easier to write a kernel driver in Rust than in C. [AndreasHindborg] Why write an NVMe driver when the kernel already has one that works well? There are no problems with the existing driver, he said, but NVMe is a good target for experiments with driver abstractions. NVMe itself is relatively simple, but it has high performance requirements. It is widely deployed, and the existing driver provides a mature reference implementation to compare against. Hindborg talked for a while about the internals of the NVMe interface; in short, communications between the interface and the computer go through a set of queues. Often the driver will configure an I/O queue for each core in the system if the interface can handle it. Creating data structures in Rust to model these queues is a relatively straightforward task. In the end, the Rust driver, when tested with the FIO tool, performs almost as well as the existing C driver. The difference, Hindborg said, is that the C driver has already been highly tuned, while the Rust driver has not; it should be able to get to the same level of performance eventually. He concluded by saying that the Rust NVMe driver is still "a playground" and not production-ready at this point. To move things forward, he would like to create more abstractions that would allow the removal of the remaining unsafe blocks in the driver. It doesn't yet support device removal or the sysfs knobs for the nvme-cli tool. He would also like to look into using the Rust async model, which would "simplify a lot of things" in the driver, but possibly at the cost of performance. At the end of Hindborg's talk, Paul McKenney asked if there was any available information on the relative bug rates between the C and Rust drivers. Hindborg answered that there have certainly been some bugs; building Rust abstractions around existing C code can be hard to do correctly. That work needs a lot of care and review, but once it works, drivers built on it tend to show few problems. A 9P filesystem server Last year, Linus Walleij suggested that, rather than writing drivers in Rust, developers should target areas with a higher attack surface -- network protocols, for example. Wedson Almeida Filho has taken that advice and written an in-kernel server for the 9P filesystem protocol in the hopes that this project would demonstrate the productivity gains and security benefits that Rust can provide. Initially, he had started trying to replace the ksmbd server, but that turned out to not be an ideal project. The SMB protocol is too complex and the server needs some significant user-space components to work. He wanted something simpler; 9P fit the bill. [Wedson AlmeidaFilho] The 9P file protocol, he said, comes from the Plan 9 operating system. The kernel has a 9P client, but no 9P server. There is a 9P server in QEMU that can be used to export host filesystems into a guest. The protocol is simple, Almeida said, defining a set of only ten operations. His 9P server implementation works now, in a read-only mode, and required just over 1,000 lines of code. Almeida was also looking for a way to experiment with async Rust in the kernel. In the async model, the compiler takes thread-like code and turns it into a state machine that can be implemented with "executors" and "reactors", which are implemented in the kernel crate. He created an executor that can run async code in a kernel workqueue; anywhere such code would block, it will release the workqueue thread for another task. There is also a socket reactor that is called for socket-state changes; it will call Waker::wake() from the Rust kernel crate to get the appropriate executor going again. There is, of course, plenty of work yet to be done. He would like to implement reactors for other I/O submission paths, including KIOCBs (asynchronous I/O), URBs (USB devices), and BIOs (block devices). Memory allocation can still use some work; it would be good if a GFP_KERNEL could give up its thread while waiting for the memory-management subsystem to do complicated things. At the end, I asked whether the objective of demonstrating the security benefits of Rust had been achieved; has there been, for example, any fuzz testing of the server? Almeida answered that the Rust-based parsing interface makes a lot of mistakes impossible. No fuzz testing has been done -- the server has only been working for a couple of weeks -- but he will do it. He concluded that he will be interested to see how his server fares in such testing relative to the QEMU implementation. Index entries for this article Kernel Development tools/Rust Conference Kangrejos/2022 [Send a free link] ----------------------------------------- (Log in to post comments) A pair of Rust kernel modules Posted Sep 12, 2022 14:54 UTC (Mon) by iustin (subscriber, #102433) [ Link] Interesting, so it finally starts? Looking forward to more Rust usage... [Reply to this comment] A pair of Rust kernel modules Posted Sep 12, 2022 14:54 UTC (Mon) by koverstreet (subscriber, # 4296) [Link] The 9p server in particular is beyond slick. The async stuff in Rust works beautifully here; internally, the compiler is doing a CPS transformation. This is something (some of us) C programmers have known how to do for years, but in practice it's _tedious_, so when we do it in C it's always less than ergonomic and incomplete. And the RAII stuff means a lot of tricky cleanup code just doesn't exist. This the code I always wished I could write - and unlike in C++, we can write it and actually trust that it's correct. [Reply to this comment] A pair of Rust kernel modules Posted Sep 13, 2022 10:33 UTC (Tue) by ncm (subscriber, #165) [Link] Or, you could code it in C++, correctly, and be done. Coding it in Rust does not, in fact, guarantee it is correct. Coding C++, you need only choose known-correct primitives to get the same level of assurance. But you would then face rabid, unreasoned hostility from Linus for C++ features you may use in Rust code without approbation; and have your patch summarily rejected with, most likely, a rude remark. The easy way to get C++ code into the kernel is via eBPF, which offers solid support for building from C++, and where there is nothing Linus can do to stop you. And, it is exactly as safe as Rust eBPF, but likely less annoying to code. [Reply to this comment] A pair of Rust kernel modules Posted Sep 13, 2022 11:08 UTC (Tue) by gspr (subscriber, #91542) [ Link] > Or, you could code it in C++, correctly, and be done. I believe that the parent commenter meant that doing exactly this is signficantly harder than it is in Rust. [Reply to this comment] A pair of Rust kernel modules Posted Sep 13, 2022 11:13 UTC (Tue) by milesrout (subscriber, # 126894) [Link] It's far easier than in Rust, because in Rust it's impossible. [Reply to this comment] Please Posted Sep 13, 2022 11:16 UTC (Tue) by corbet (editor, #1) [Link] Can we please try to avoid yet another round of silly language-advocacy postings here? We've all heard it. The article is about people showing real work, albeit at an early stage; let's focus on the actual work. Thank you all. [Reply to this comment] A pair of Rust kernel modules Posted Sep 12, 2022 16:28 UTC (Mon) by NHO (subscriber, #104320) [ Link] Then there's Asahi DRM driver for Apple Silicon. [Reply to this comment] A pair of Rust kernel modules Posted Sep 13, 2022 9:48 UTC (Tue) by josh (subscriber, #17465) [Link ] The NVMe driver was presented again the this week at Plumbers, to enthusiastic reactions: https://twitter.com/josh_triplett/status/ 1569363148985233414 [Reply to this comment] A pair of Rust kernel modules Posted Sep 13, 2022 15:38 UTC (Tue) by MrWim (subscriber, #47432) [ Link] Video: https://www.youtube.com/watch?v=Xw9pKeJ-4Bw&t=8040s [Reply to this comment] A pair of Rust kernel modules Posted Sep 13, 2022 11:12 UTC (Tue) by milesrout (subscriber, # 126894) [Link] > At the end of Hindborg's talk, Paul McKenney asked if there was any available information on the relative bug rates between the C and Rust drivers. Hindborg answered that there have certainly been some bugs; building Rust abstractions around existing C code can be hard to do correctly. That work needs a lot of care and review, but once it works, drivers built on it tend to show few problems. So in other words: oh yes, there are still bugs, but we can just blame them on C. Any issues with interfacing Rust and C? Obviously the fault of C. Even though clearly and logically any bugs introduced by having to interface two languages is only reasonably attributable to the language that came along later and seems intent on forcing its way into the kernel regardless of what anyone else wants. And things like this: > At the end, I asked whether the objective of demonstrating the security benefits of Rust had been achieved; has there been, for example, any fuzz testing of the server? Almeida answered that the Rust-based parsing interface makes a lot of mistakes impossible. No fuzz testing has been done -- the server has only been working for a couple of weeks -- but he will do it. He concluded that he will be interested to see how his server fares in such testing relative to the QEMU implementation. In other words "no we haven't actually tested it but I'm sure the language prevents bugs, they're totally impossible". Yeah right. This is typical of the Rust community: huge promises, no evidence to back them up, all topped off with an "if it compiles it is correct" attitude that totally disregards that there are many more issues other than those the Rust people have decided count as 'safety'. Anyone noticed that 'unsafe' as a general concept has suddenly been redefined to mean 'whatever Rust prevents' (if you don't use `unsafe `, which the Rust for linux code does all over the place)? When people involved in Rust's development realised just before the release of the language that the language was fundamentally unsound, allowing memory leaks, they quietly redefined 'safety' to exclude leak freedom, because they didn't have time to properly fix it before 1.0. This was despite months and years of telling people how wonderful Rust was because it prevented memory leaks, lol. This should come as no surprise from the same group of radicals that redefine words on a daily basis from what everyone in the world understood them to mean, inside and outside technology. 'Master branch' is an (attempted) victim of the same ideology - nothing, not even in the truth, will stand in their way. The entire idea is horrible. I do not want a Rust compiler on my computer, I do not want to need a Rust compiler on my computer to be able to compile the kernel. Rust is 'trusting trust' on steroids, for one thing. It is incredibly slow. Compiling a kernel takes long enough as it is! It's specified as 'whatever rustc does'. It has one implementation, with no other implementations even remotely close to being ready (there are other "implementations", but they are nowhere near complete and cannot do borrow checking, the core feature of the language, so are practically useless.) It is an abhorrently complex language. Nobody has EVER explained what it provides to the kernel that Ada or better static analysis tools for C could not. The ONLY thing Rust has going for it is an incredibly pushy "community". They say "oh it will only be in a few drivers". Yeah, at first. But there's no real point doing it if there's not going to be a lot of it. And that means SLOW compile times and an inability to bootstrap reliably. Anyway, let's be real: Rust is just not going to happen in the kernel. Rust can express legibly only one way of managing memory. Nobody has even managed to represent the Wayland memory model in Rust, and the kernel is far more complicated than that. The one serious attempt to do so failed after months of work because it required writing thousands upon thousands of lines of memory management boilerplate. If you want to represent anything more complex than Box or Rc you need to just about write a thesis. And look at the code itself. There's a huge list of nightly (basically brand new!) compiler features that are required to build it. No way should there be code in the Linux kernel that requires anything other than stable language features. At least one of them is described as "Status: the current design is perma-unstable -- a new RFC is needed. The issue may be split." And generic associated types have been in progress for more than 5 years with no real sign that they're going to be stabilised any time soon. And last I checked, 'async' in Rust is widely considered a failure of design. [Reply to this comment] A pair of Rust kernel modules Posted Sep 13, 2022 13:10 UTC (Tue) by PengZheng (subscriber, # 108006) [Link] Thank you very much for stopping me from reading a Rust book, successfully. I'd better invest that time learning C++20/23. [Reply to this comment] A pair of Rust kernel modules Posted Sep 13, 2022 14:04 UTC (Tue) by reijoslav (guest, #98915) [ Link] The comment you're replying to seems to be pretty trolly. The author has actually been banned from some other sites with the reason "Troll". [Reply to this comment] A pair of Rust kernel modules Posted Sep 13, 2022 14:45 UTC (Tue) by Wol (subscriber, #4433) [Link] Said trolly author has clearly never heard of a Turing Machine, so yes ... Cheers, Wol [Reply to this comment] A pair of Rust kernel modules Posted Sep 13, 2022 15:06 UTC (Tue) by rav (subscriber, #89256) [Link ] > And generic associated types have been in progress for more than 5 years with no real sign that they're going to be stabilised any time soon. Actually, under two hours after your comment, generic associated types were stabilised: https://github.com/rust-lang/rust/pull/96709# issuecomment... [Reply to this comment] A pair of Rust kernel modules Posted Sep 13, 2022 16:12 UTC (Tue) by ssokolow (guest, #94568) [Link ] Anyone noticed that 'unsafe' as a general concept has suddenly been redefined to mean 'whatever Rust prevents' (if you don't use `unsafe`, which the Rust for linux code does all over the place)? When people involved in Rust's development realised just before the release of the language that the language was fundamentally unsound, allowing memory leaks, they quietly redefined 'safety' to exclude leak freedom, because they didn't have time to properly fix it before 1.0. This was despite months and years of telling people how wonderful Rust was because it prevented memory leaks, lol. No, the "leakpocalype" (There's your googleable keyword) was specifically about the realization that the original version of the scoped threads API that got re-added recently was unsound because it didn't account for Arc and a reference cycle allowing you to leak memory and, as such, allowed code not marked with "unsafe" to create pointers that outlived what they pointed to. There was nothing "quietly" about it, and the thing they didn't have time to properly fix before 1.0 was the scoped threads API. It was already clear that preventing memory leaks in the the general case was a Rice's theorem problem. (Among other reasons, it's a good hint it might be when you can "leak memory" in the "same externally observed properties" sense in JavaScript just by losing track of which event handlers you've forgotten to unregister.) It's specified as 'whatever rustc does'. As opposed to "whatever GCC does" like the dialect of C that the Linux kernel requires, which LLVM is still working to perfectly replicate? Nobody has EVER explained what it provides to the kernel that Ada or better static analysis tools for C could not. Ada is more focused on constraining integer types, which requires more runtime support, which makes it a worse fit for interoperating with an existing C kernel. Also, I don't have a citation for this, but I remember reading a comment that it's misleading to look at GCC's platform support and assume that GNAT has proper support for all those platforms. (And then there's the fact that Ada's more Wirth-style syntax feels more alien to the average 21st-century programmer than Rust's "Ocaml in a C++ trench coat" syntax does, and part of the interest in Rust is in heading off the "COBOL programmers get paid a ton because demand is outstripping supply" problem the kernel might face in the coming decades.) As for "better static analysis tools", I've used Splint. Annotating C with the information needed to prove the relevant properties at compile time rapidly makes the code less readable than the equivalent Rust. That's why you generally don't see people just extending C to get the same results. And last I checked, 'async' in Rust is widely considered a failure of design. Citation, please. The people I've seen tend to say that it's an impressive exercise in pushing the boundaries for what can be achieved without tracing garbage collection. [Reply to this comment] A pair of Rust kernel modules Posted Sep 13, 2022 21:02 UTC (Tue) by NYKevin (subscriber, #129325) [Link] How does anyone come to the conclusion that Rust is "supposed to" prevent memory leaks, when Box has a whole convenience function specifically for leaking memory? It boggles the mind. https://doc.rust-lang.org/std/boxed/struct.Box.html#metho... [Reply to this comment] A pair of Rust kernel modules Posted Sep 13, 2022 18:23 UTC (Tue) by lambda (subscriber, #40735) [ Link] I'm sure I shouldn't be replying to the troll, but since other folks reading might be misled by this comment, I figure I should provide some clarifications. > So in other words: oh yes, there are still bugs, but we can just blame them on C. Any issues with interfacing Rust and C? Obviously the fault of C. The fundamental unique feature of Rust is that it enables you to build safe abstractions around unsafe code (we'll get to what "safe" means in a bit). Whether you are interfacing with C, or defining your own primitives in Rust, doing that can be tricky, but once it's done, you have fairly strong guarantees on what can happen in the safe code that uses those abstractions. So, this isn't "blaming bugs on C", but just an acknowledgement that Rust is not a silver bullet (and has never been intended as such), and that the fundamental work of building safe abstractions over unsafe code still requires care. Since there are a large number of abstractions that already exist in the Linux kernel, interoperating with them requires work, and there will be bugs in the process. > In other words "no we haven't actually tested it but I'm sure the language prevents bugs, they're totally impossible". You're attacking a strawman here. No one has ever said that bugs are totally impossible. > Anyone noticed that 'unsafe' as a general concept has suddenly been redefined to mean 'whatever Rust prevents' "Unsafe" in Rust means "can do things that could potentially be undefined behavior" or in other words, could potentially have executions which have no valid behavior under the language model. And as a Turing-complete, general purpose language, Rust is not intended to prevent you from making logic bugs, infinite loops, or resource leaks; it does have a number of design features that help make them more difficult to make (RAII, exhaustive match statements, etc), but it makes a hard distinction between things that it prevents categorically, and things that the language design helps with but doesn't prevent entirely. And this definition is part of a contract between the compiler, standard library, and libraries that provide safe abstractions over unsafe code. There needs to be a definition of what rules such libraries need to follow, and what assumptions they can make, so that you can combine two different libraries which each provide safe abstractions over unsafe code, and the combination itself will continue to be safe. So this particular set of rules are "whatever Rust prevents." As an example, pre-Rust 1.0, there were some API designs that were safe only if it was impossible to leak an object and return to the caller; but of course, reference counted pointers could be set up to cause a cycle and leak an object. There was a long involved discussion of this, with various proposals, but in the end, it was decided that categorically preventing leaks would impose too much burden on language and library design; for instance, it would require a much more complicated system for reference counted objects in order to prevent circular references, and impose similar design burdens many other kinds of data structures. So rather, it was decided that leaking objects wasn't something that would be categorically prevented in safe code, and libraries that provide safe abstractions over unsafe primitives can't depend on an object not being leaked for their safety guarantees. So that's just part of the contract now; there were alternative possibilities for that contract, which would have had different tradeoffs, but this was the tradeoff chosen. Effectively, what is forbidden in safe Rust is anything which could cause objects to be interpreted as the wrong type, accessed when they are not valid, or accessed in overlapping ways in space or time. This means no out of bounds access, use after free, data races (two threads accessing the same memory in ways that no linear interleaving could produce), iterator invalidation, etc. This is a category of bugs which are quite common in C and C++ programs, which are difficult to reason about because they effectively break the model of the programming language, and are quite commonly prone to exploitation by attackers. Leaking an object, on its own, does not lead to arbitrary behavior; it's undesirable, and can lead to resource exhaustion, but in any Turing complete language you could write code which never halts or uses up an arbitrary amount of resources. > because they didn't have time to properly fix it before 1.0 This is not true. It was not the case that "they didn't have time to properly fix it", it's that there is no way to statically prevent leaking of resources in a way that would not be overly burdensome on the API of data structures. > This was despite months and years of telling people how wonderful Rust was because it prevented memory leaks, lol. No one has ever claimed that Rust prevented memory leaks. Reference counted types have existed from the beginning, and everyone has always known that it's possible to create reference counted cycles and leak memory that way. Anyone who thought that Rust was supposed to prevent all memory leaks likely misinterpreted the term "memory safety", which is a term that predates Rust, and refers to preventing unsafe access to objects, not preventing leaking of objects. > Rust is 'trusting trust' on steroids, for one thing. There are at least two independent bootstrap paths currently possible for the Rust compiler; you can start from the original pre-1.0 compiler written in OCaml and bootstrap from release to release that way. Or there is an independent compiler, mrustc, written in C++ which is able to compile much more recent releases of rustc and start the bootstrap chain that way. This provides the diverse double compilation necessary to defeat trusting trust attacks. > Nobody has EVER explained what it provides to the kernel that Ada or better static analysis tools for C could not. Kernel developers are interested in writing kernel code in Rust, not Ada. Ada is far more foreign, and wasn't designed for the same kind of interop with C and encapsulation of unsafe code behind safe interface that Rust was. A lot of people focus on the safety guarantees of Rust, because it has a unique approach to safety that no other language provides. But besides that, it's a modern language with a lot of nice features that many developers appreciate, and with a focus on balancing those safety goals with usability, performance, and interop with existing ecosystems like C. "Better static analysis" can only get you so far. The language level support that Rust provides means that you design your APIs around the type system and lifetimes, which means that code can be statically checked independently and separately; the lack of such guarantees and annotations in C makes it far more difficult to add static checking that is anywhere near as robust as what Rust provides out of the box. It's not for lack of trying; people have been writing static checkers for C and C++ for decades, and yet memory safety errors still cause the majority of security vulnerabilities in codebases written in C and C++. > It's specified as 'whatever rustc does'. It has one implementation, with no other implementations even remotely close to being ready You realize that the same was true of C in the kernel until relatively recently, right? The kernel is not written in standard C; it's written in GCC C. The kernel has a memory model that is different than the standard C memory model. The kernel is now mostly able to be built with clang as well, but only by years of effort of adding GCC features to clang and modifying the kernel to not rely on them in quite as many places. There is a Rust reference, and ongoing efforts on continuing to specify Rust, but this is mostly irrelevant to its usage in the kernel. > There's a huge list of nightly (basically brand new!) compiler features that are required to build it. No way should there be code in the Linux kernel that requires anything other than stable language features. An unstable feature in Rust is much like an implementation-specific feature in C; and the kernel uses plenty of GCC features. It's a way of providing some features without committing to supporting that feature in exactly that form indefinitely; there may be backwards incompatible changes in the future. Implementation-specific features in GCC may be replaced by standardized C features in the future, but that doesn't mean they aren't used in the kernel. The entirety of the internal Linux kernel API is considered unstable, Linux only provides a stable interface to userspace; does that mean that no one writing code in one part of the kernel should depend on code in another part because it's unstable? No, it just means that changes to libraries within the kernel may need to be propagated to usages as well. The same is true of these unstable Rust features; you opt into a few, with the knowledge that you may need to make changes later on when upgrading compilers. It's something you shouldn't use willy-nilly, but using a few features where you may need to make changes in the future is not that big a deal. These aren't being used with abandon; there is a tracking issue explaining all unstable features, and standard library config flags, what they are needed for, how essential they are, whether they're on track to stabilization, etc: https://github.com/Rust-for-Linux/linux/ issues/2 Some of them will be stabilized in Rust, some will be removed from the kernel, and some will be lived with. > (basically brand new!) > generic associated types have been in progress for more than 5 years with no real sign that they're going to be stabilised any time soon. Ok, besides the fact that as someone else pointed out, this feature was agreed to be stabilized within hours of your post (it had already been in final comment period for a while), it's funny that you call these features "basically brand new" while also acknowledging that some of thems have been undergoing serious development for years. > Nobody has even managed to represent the Wayland memory model in Rust, and the kernel is far more complicated than that. The one serious attempt to do so failed after months of work because it required writing thousands upon thousands of lines of memory management boilerplate. If you want to represent anything more complex than Box or Rc you need to just about write a thesis. There are hundred of crates that provide memory management primitives beyond Box and Rc (https://lib.rs/memory-management), plus numerous examples of successful integration of Rust into C and C++ projects with custom memory management, including Firefox. There was one project in which someone attempted to provide a general-purpose rust API over wlroots, a C Wayland library, and eventually decided that it wasn't worth it for their purposes. That doesn't mean it's impossible; just that one person tried one approach, didn't like it, and decided it wasn't what they wanted to spend their time doing. > And last I checked, 'async' in Rust is widely considered a failure of design. So much of a failure of design that it's being used for substantial production usage like an entire rewrite of the Tor client in Rust in two years: https://blog.torproject.org/arti_100_released/ There are gripes that folks have with aspects of async in Rust; but it's actually a very well received, widely used feature, that just happens to be a little bit young and has had some ecosystem growing pains. [Reply to this comment] A pair of Rust kernel modules Posted Sep 13, 2022 19:44 UTC (Tue) by atnot (subscriber, #124910) [ Link] > Nobody has EVER explained what it provides to the kernel that Ada or better static analysis tools for C could not. To quote David Gerard: "could" is a word that means "doesn't" [Reply to this comment] Glad to see this! Posted Sep 13, 2022 13:20 UTC (Tue) by david.a.wheeler (subscriber, # 72896) [Link] I'm very glad to see this. The only real way to have justified confidence that the Rust APIs will work for kernel modules is to try using those APIs. [Reply to this comment] Copyright (c) 2022, Eklektix, Inc. Comments and public postings are copyrighted by their creators. Linux is a registered trademark of Linus Torvalds