[HN Gopher] Futurelock: A subtle risk in async Rust
___________________________________________________________________
Futurelock: A subtle risk in async Rust
This RFD describes our distillation of a really gnarly issue that
we hit in the Oxide control plane.[0] Not unlike our discovery of
the async cancellation issue[1][2][3], this is larger than the
issue itself -- and worse, the program that hits futurelock is
correct from the programmer's point of view. Fortunately, the
surface area here is smaller than that of async cancellation and
the conditions required to hit it can be relatively easily
mitigated. Still, this is a pretty deep issue -- and something that
took some very seasoned Rust hands quite a while to find. [0]
https://github.com/oxidecomputer/omicron/issues/9259 [1]
https://rfd.shared.oxide.computer/rfd/397 [2]
https://rfd.shared.oxide.computer/rfd/400 [3]
https://www.youtube.com/watch?v=zrv5Cy1R7r4
Author : bcantrill
Score : 178 points
Date : 2025-10-31 16:49 UTC (6 hours ago)
(HTM) web link (rfd.shared.oxide.computer)
(TXT) w3m dump (rfd.shared.oxide.computer)
| Sytten wrote:
| I am wondering if there is a larger RFC for Rust to force users
| to not hold a variable across await points.
|
| In my mind futurelock is similar to keeping a sync lock across an
| await point. We have nothing right now to force a drop and I
| think the solution to that problem would help here.
| cogman10 wrote:
| The ideas that have been batted around is called "async drop"
| [1]
|
| And it looks like it's still just an unaddressed well known
| problem [2].
|
| Honestly, once the Mozilla sackening of rust devs happened it
| seems like the language has been practically rudderless. The
| RFC system seems almost dead as a lot of the main contributors
| are no longer working on rust.
|
| This initiative hasn't had motion since 2021. [3]
|
| [1] https://rust-lang.github.io/async-fundamentals-
| initiative/ro...
|
| [2] https://rust-lang.github.io/async-fundamentals-initiative/
|
| [3] https://github.com/rust-lang/async-fundamentals-initiative
| raggi wrote:
| Those pages are out of date, and AsyncDrop is in progress:
| https://github.com/rust-lang/rust/issues/126482
|
| I think "practically rudderless" here is fairly misinformed
| and a little harmful/rude to all the folks doing tons of
| great work still.
|
| It's a shame there are some stale pages around and so on, but
| they're not good measures of the state of the project or
| ecosystem.
|
| The problem of holding objects across async points is also
| partially implemented in this unstable lint marker which is
| used by some projects: https://dev-doc.rust-
| lang.org/unstable-book/language-feature...
|
| You also get a similar effect in multi-threaded runtimes by
| not arbitrarily making everything in your object model Send
| and instead designing your architecture so that most things
| between wake-ups don't become arbitrarily movable references.
|
| These aren't perfect mitigations, but some tools.
| bigstrat2003 wrote:
| In fairness, if you're a layman to the rust development
| process (as I am, so I'm speaking from personal experience
| here) it's _damn near impossible_ to figure out the status
| of things. There tracking issues, RFCs, etc which is very
| confusing as an outsider and gives no obvious place to look
| to find out the current status of a proposal. I 'm sure
| there _is_ a logic to it and that if I spent the time to
| learn it would make sense. But it is really hard to
| approach for someone like me.
| cogman10 wrote:
| > I think "practically rudderless" here is fairly
| misinformed and a little harmful/rude to all the folks
| doing tons of great work still.
|
| That great work is mostly opaque on the outside.
|
| What's been noticeable as an observer is that a lot of the
| well known names associated with rust no longer work on it
| and there's been a large amount of turnover around it.
|
| That manifests in things like this case where work was in
| progress up until ~2021 and then was ultimately
| backburnered while the entire org was reshuffled. (I'd note
| the dates on the MCP as Feb 2024).
|
| I can't tell exactly how much work or what direction it
| went in from 2021 to 2024 but it does look apparent that
| the work ultimately got shifted between multiple
| individuals.
|
| I hope rust is in a better spot. But I also don't think I
| was being unfair in pointing out how much momentum got
| wrecked when Mozilla pulled support.
| raggi wrote:
| The language team tends to look at these kinds of
| challenges and drive them to a root cause, which spins
| off a tree of work to adjust the core language to support
| what's required by the higher level pieces, once that
| work is done then the higher level projects are unblocked
| (example: RPIT for async drop).
|
| That's not always super visible if you're not following
| the working groups or in contact with folks working on
| the stuff. It's entirely fair that they're prioritizing
| getting work done than explaining low level language
| challenges to everyone everywhere.
|
| I think you're seeing a lack of data and trying to use
| that as a justification to fit a story that you like,
| more than seeing data that is derivative of the story
| that you like. Of course some people were horribly
| disrupted by the changes, but language usage also
| expanded substantially during and since that time, and
| there are many team members employed by many other
| organizations, and many independents too.
|
| And there are more docs, anyway:
|
| https://rust-lang.github.io/rust-project-
| goals/2024h2/async.... https://rust-lang.github.io/rust-
| project-goals/2025h1/async.... https://rust-
| lang.github.io/rust-project-goals/2025h2/field-...
| https://rust-lang.github.io/rust-project-
| goals/2025h2/evolvi... https://rust-lang.github.io/rust-
| project-goals/2025h2/goals....
| sunshowers wrote:
| Note that forcing a drop of a lock guard has its own issues,
| particularly around leaving the guarded data in an invalid
| state. I cover this a bit in my talk that Bryan linked to in
| the OP [1].
|
| [1] timestamped: https://youtu.be/zrv5Cy1R7r4?t=1067
| ameliaquining wrote:
| There's an existing lint that lets you prohibit instances of
| specific types from being held across await points:
| https://rust-lang.github.io/rust-clippy/stable/index.html#aw...
| amluto wrote:
| I'm not convinced that this can help in a meaningful way.
|
| Fundamentally, if you have two coroutines (or cooperatively
| scheduled threads or whatever), and one of them holds a lock,
| and the other one is awaiting the lock, and you don't schedule
| the first one, you're stuck.
|
| I wonder if there's a form of structured concurrency that would
| help. If I create two futures and start both of them (in Rust
| this means polling each one once) but do not continue to poll
| both, then I'm sort of making a mistake.
|
| So imagine a world where, to poll a future at all, I need to
| have a nursery, and the nursery is passed in from my task and
| down the call stack. When I create a future, I can pass in my
| nursery, but that future then gets an exclusive reference to my
| future until it's complete or cancelled. If I want to create
| more than one future that are live concurrently, I need to
| create a FutureGroup (that gets an exclusive reference to my
| nursery) and that allows me to create multiple sub-nurseries
| that can be used to make futures but cannot be used to poll
| them -- instead I poll the FutureGroup.
|
| (I have yet to try using an async/await system or a reactor or
| anything of the sort that is not very easy to screw up. My
| current pet peeve is this pattern: data =
| await thingy.read()
|
| What if thingy.read() succeeds but I am cancelled? This gets
| nasty is most programming languages. Python: the docs on when I
| can get cancelled are almost nonexistent, and it's not
| obviously possible to catch the CancelledError such that I
| still have data and can therefore save it somewhere so it's not
| lost. Rust: what if thingy thinks it has returned the data but
| I'm never polled again? Maybe this can't happen if I'm careful,
| but that requires more thought than I'm really happy with.)
| orthecreedence wrote:
| Great read, and the example code makes sense. This stuff can be a
| nightmare to find, but once you do it's like a giant 1000 piece
| puzzle just clicks together instantly.
| bcantrill wrote:
| Indeed. One of the interesting side effects of being a remote
| company that records everything[0] is that we have the instant
| where the "1000 piece puzzle just clicks together" recorded,
| and it's honestly pretty wild. In this case, it was very much a
| shared brainstorming between four engineers (Eliza, Sean, John
| and Dave) -- and there is almost a passing of the baton where
| they start to imagine the kind of scenario that could induce
| this and then realize that those are exactly the conditions
| that exist in the software.
|
| We are (on brand?) going to do a podcast episode on this on
| Monday[1]; ahead of that conversation I'm going to get a clip
| of that video out, just because it's interesting to see the
| team work together to debug it.
|
| [0] https://rfd.shared.oxide.computer/rfd/0537
|
| [1] https://discord.gg/QrcKGTTPrF?event=1433923627988029462
| mycoliza wrote:
| As a member of (Eliza, Sean, John, and Dave), I can second
| that debugging this was certainly an adventure. I'm not going
| to go as far as to say that we _had fun_ , since...you can't
| have a heroic narrative without real struggle. But it was
| certainly rewarding to be in the room for that "a-ha!"
| moment, in which all the pieces really did begin to fit
| together very quickly. It was like the climax of a detective
| story --- and it was particularly well-scripted the way each
| of us contributed a little piece of the puzzle.
| littlestymaar wrote:
| Since you are of of the people working directly on this
| codebase, may I ask you why is _select!_ being used
| /allowed in the first place?
|
| Its footgun-y nature has been known for years (IIRC even
| the first version of the tokio documentation warned against
| that) and as such I don't really understand why people are
| still using it. (For context I was the lead of a Rust team
| working on a pretty complex async networking program and we
| had banned _select!_ very early in the project and never
| regretted this decision once).
| oconnor663 wrote:
| > FAQ: doesn't future1 get cancelled?
|
| I guess cancellation is really two different things, which
| usually happen at the ~same time, but not in this case: 1) the
| future stops getting polled, and 2) the future gets dropped. In
| this example the drop is delayed, and because the future is
| holding a guard,* the delay has side effects. So the future "has
| been cancelled" in the sense that it will never again make
| forward progress, but it "hasn't been cancelled yet" in the sense
| that it's still holding resources. I wonder if it's practical to
| say "make sure those two things always happen together"?
|
| * Technically a Tokio-internal `Acquire` future that owns a queue
| position to get a guard, but it sounds like the exact same bug
| could manifest after it got the guard too, so let's call it a
| guard.
| jacquesm wrote:
| If any rust designers are lurking about here: what made you
| decide to go for the async design pattern instead of the actor
| pattern, which - to me at least - seems so much cleaner and so
| much harder to get wrong?
|
| Ever since I started using Erlang it felt like I finally found
| 'the right way' when before then I did a lot of work with sockets
| and asynchronous worker threads. But even though it usually
| worked as advertised it had a large number of _really_ nasty
| pitfalls which the actor model seemed to - effortlessy - step
| aside.
|
| So I'm seriously wondering what the motivation was. I get why JS
| uses async, there isn't any other way there, by the time they
| added async it was too late to change the fundamentals of the
| language to such a degree. But rust was a clean slate.
| sunshowers wrote:
| Not a Rust designer, but a big motivation for Rust's async
| design was wanting it to work on embedded, meaning no malloc
| and no threads. This unfortunately precludes the vast majority
| of the design space here, from active futures as seen in
| JS/C#/Go to the actor model.
|
| You can write code using the actor model with Tokio. But it's
| not natural to do so.
| lll-o-lll wrote:
| As a curious bystander, it will be interesting to see how the
| Zig async implementation pans out. They have the advantage of
| getting to see the pitfalls of those that have come before.
|
| Getting back to Rust, even if not natural, I agree with the
| parent that the actor model is simply the better paradigm.
| Zero runtime allocation should still be possible, you just
| have to accept some constraints.
|
| I think async looks simple because it looks like writing
| imperative code; unfortunately it is just obfuscating the
| complex reality underlying. The actor model makes things
| easier to reason about, even if it looks more complicated
| initially.
| sunshowers wrote:
| I think you can do a static list of actors or tasks in
| embedded, but it's hard to dynamically spin up new ones.
| That's where intra-task concurrency is helpful.
| shepmaster wrote:
| > But it's not natural to do so.
|
| I tend to write most of my async Rust following the actor
| model and I find it natural. Alice Rhyl, a prominent Tokio
| contributor, has written about the specific patterns:
|
| https://ryhl.io/blog/actors-with-tokio/
| sunshowers wrote:
| Oh I do too, and that's one of the recommendations in RFD
| 400 as well as in my talk. cargo-nextest's runner loop [1]
| is also structured as two main actors + one for each test.
| But you have to write it all out and it can get pretty
| verbose.
|
| [1] https://nexte.st/docs/design/architecture/runner-loop/
| oconnor663 wrote:
| Kind of a tangent, but I think "systems programming" tends to
| bounce back and forth between three(?) different concerns
| that turn out to be closely related:
|
| 1. embedded hardware, like you mentioned
|
| 2. high-performance stuff
|
| 3. "embedding" in the cross-language sense, with foreign
| function calls
|
| Of course the "don't use a lot of resources" thing that makes
| Rust/C/C++ good for tiny hardware also tends to be helpful
| for performance on bigger iron. Similarly, the "don't assume
| much about your runtime" thing that's necessary for bare
| metal programming also helps a lot with interfacing with
| other languages. And "run on a GPU" is kind of all three of
| those things at once.
|
| So yeah, which of those concerns was async Rust really
| designed around? All of them I guess? It's kind of like, once
| you put on the systems programming goggles for long enough,
| all of those things kind of blend together?
| fpoling wrote:
| Rust async still uses a native stack which just a form of
| memory allocator that uses LIFO order. And controlling stack
| usage in the embedding world is just as important as not
| relying on the system allocator.
|
| So its a pity that Rust async design tried so hard to avoid
| any explicit allocations rather than using an explicit
| allocator that embedding can use to preallocate and reuse
| objects.
| zbentley wrote:
| > a native stack [is] just a form of memory allocator
|
| There is a _lot_ riding on that "just". Hardware stacks are
| very, very unlike heap memory allocators in pretty much
| every possible way _other_ than "both systems provide
| access to memory."
|
| Tons and tons of embedded code assumes the stack is,
| indeed, a hardware stack. It's far from trivial to make
| that code "just use a dummy/static allocator with the same
| api as a heap"; that code may not be in Rust, and it's
| ubiquitous for embedded code to not be written with
| abstractions in front of its allocator--why would it do
| otherwise, given that tons of embedded code was written for
| a specific compiler+hardware combination with a specific
| (and often automatic or compiler-assisted) stack memory
| management scheme? That's a bit like complaining that a
| specific device driver doesn't use a device-agnostic
| abstraction.
| tony69 wrote:
| Stack allocation/deallocation does not fragment memory,
| that's a yuge difference for embedded systems and the main
| reason to avoid the heap
| raggi wrote:
| _an answer_ is performance - the necessity of creating
| copyable/copied messages for inter-actor communication
| everywhere in the program _can be_ expensive.
|
| that said there are a lot of parts of a lot of programs where a
| fully inlined and shake optimized async state machine isn't so
| critical.
|
| it's reasonable to want a mix, to use async which can be
| heavily compiler optimized for performance sensitive paths, and
| use higher level abstractions like actors, channels, single
| threaded tasks, etc for less sensitive areas.
| lll-o-lll wrote:
| I'm not sure this is actually true? Do messages _have_ to be
| copied?
| raggi wrote:
| if you want your actors to be independent computation flows
| and they're in different coroutines or threads, then you
| need to arrange that the data source can not modify the
| data once it arrives at the destination, in order to be
| safe.
|
| in a single threaded fully cooperative environment you
| could ensure this by implication of only one coroutine
| running at a time, removing data races, but retaining
| logical ones.
|
| if you want to eradicate logical races, or have actual
| parallel computation, then the source data must be copied
| into the message, or the content of the message be wrapped
| in a lock or similar.
|
| in almost all practical scenarios this means the data
| source copies data into messages.
| gleenn wrote:
| Isn't that something Rust is particularly good at,
| controlling the mutation of shared memory?
| raggi wrote:
| yes
| vlovich123 wrote:
| In Rust wouldn't you just Send the data?
| sapiogram wrote:
| Rust solves this at compile-time with move semantics,
| with no runtime overhead. This feature is arguably why
| Rust exists, it's really useful.
| the__alchemist wrote:
| I'm surprised learning this too. I know the hobby embedded, and
| HTTP-server OSS ecosystem have committed to Async, but I didn't
| expect Oxide would.
| mycoliza wrote:
| We actually _don 't_ use Rust async in the embedded parts of
| our system. This is largely because our firmware is based on
| a multi-tasking microkernel operating system, Hubris[1], and
| we can express concurrency at the level of the OS scheduler.
| Although our service processors are single-core systems, we
| can still rely on the OS to schedule multiple threads of
| execution.
|
| Rust async is, however, very useful in single-core embedded
| systems that _don 't_ have an operating system with
| preemptive multitasking, where one thread of execution is all
| you ever get. It's nice to have a way to express that you
| might be doing multiple things concurrently in an event-
| driven way without having to have an OS to manage preemptive
| multitasking.
|
| [1] https://hubris.oxide.computer/
| jabedude wrote:
| Heh, this is super interesting to hear. Single threaded
| async/concurrent code is so fun and interesting to see.
| I've ran some tokio programs in single threaded mode just
| to see it in action
| the__alchemist wrote:
| Your application needs concurrency. So, the answer is... switch
| your entire application, code style, and libraries it uses into
| a separate domain that is borderline incompatible with normal
| one? And has its own dialects that have their own compatibility
| barriers? Doesn't make sense to me.
| mdasen wrote:
| I'd recommend watching this video:
| https://www.infoq.com/presentations/rust-2019/; and reading
| this: https://tokio.rs/blog/2020-04-preemption
|
| I'm not the right person to write a tl;dr, but here goes.
|
| For actors, you're basically talking about green threads. Rust
| had a hard constraint that calls to C not have overhead and so
| green threads were out. C is going to expect an actual stack so
| you have to basically spin up a real stack from your green-
| thread stack, call the C function, then translate it back. I
| think Erlang also does some magic where it will move things to
| a separate thread pool so that the C FFI can block without
| blocking the rest of your Erlang actors.
|
| Generally, async/await has lower overhead because it gets
| compiled down to a state machine and event loop. Languages like
| Go and Erlang are great, but Rust is a systems programming
| language looking for zero cost abstractions rather than just
| "it's fast."
|
| To some extent, you can trade overhead for ease. Garbage
| collectors are easy, but they come with overhead compared to
| Rust's borrow checker method or malloc/free.
|
| To an extent it's about tradeoffs and what you're trying to
| make. Erlang and Go were trying to build something different
| where different tradeoffs made sense.
| moralestapia wrote:
| Hmm, curious to see if this could happen on JS. I'll reproduce
| the code.
| raggi wrote:
| yes, you can produce similar issues with promise guarded states
| and so on as well, it's a fairly common issue in async
| programming, but can be surprising when it's hidden by layers
| of abstraction / far up/down a call-chain.
| dvt wrote:
| > &mut future1 is dropped, but this is just a reference and so
| has no effect. Importantly, the future itself (future1) is not
| dropped.
|
| There's a lot of talk about Rust's await implementation, but I
| don't really think that's the issue here. After all, Rust doesn't
| guarantee convergence. Tokio, on the other hand (being a library
| that handles multi-threading), _should_ (at least when using its
| own constructs, e.g. the `select!` macro).
|
| So, since the crux of the problem is the `tokio::select!` macro,
| it seems like a pretty clear tokio bug. Side note, I never looked
| at it before, but the macro[1] is absolutely hideous.
|
| [1] https://docs.rs/tokio/1.34.0/src/tokio/macros/select.rs.html
| raggi wrote:
| i forget if this part unwinds to the exact same place, but some
| of this kind of design constraint in tokio stems from the much
| earlier language capabilities and is prohibitive to adjust
| without breaking the user ecosystem.
|
| one of the key advertised selling points in some of the other
| runtimes was specifically around behavior of tasks on drop of
| their join handles for example, for reasons closely related to
| this post.
| oconnor663 wrote:
| There's nothing `select!` could do here to force `future1` to
| drop, because it doesn't receive ownership of `future1`. If we
| wanted to force this, we'd have to forbid `select!` from
| polling futures by reference, but that's a pretty fundamental
| capability that we often rely on to `select!` in a loop for
| example. The blanket `impl<F> Future for &mut F where F: Future
| ...` isn't a Tokio thing either; that's in the standard
| library.
| mycoliza wrote:
| What could `tokio::select!` do differently here to prevent bugs
| like this?
|
| In the case of `select!`, it is a direct consequence of the
| ability to poll a `&mut` reference to a future in a `select!`
| arm, where the future is not dropped should another future win
| the "race" of the select. This is not really a choice Tokio
| made when designing `select!`, but is instead due to the
| existence of implementations of `Future` for `&mut T: Future +
| Unpin`[1] and `Pin<T: Future>`[2] in the standard library.
|
| Tokio's `select!` macro cannot easily stop the user from doing
| this, and, furthermore, the fact that you can do this is
| _useful_ --- there are many legitimate reasons you might want
| to continue polling a future if another branch of the select
| completes first. It 's desirable to be able to express the idea
| that we want to continually poll drive one asynchronous
| operation to completion while periodically checking if some
| other thing has happened and taking action based on that, and
| then continue driving forward the ongoing operation. That was
| precisely what the code in which we found the bug was doing,
| and it is a pretty reasonable thing to want to do; a version of
| the `select!` macro which disallows that would limit its
| usefulness. The issue arises specifically from the fact that
| the `&mut future` has been polled to a state in which it has
| acquired, but not released, a shared lock or lock-like
| resource, and then another arm of the `select!` completes first
| and the body of that branch runs async code that _also_ awaits
| that shared resource.
|
| If you can think of an API change which Tokio could make that
| would solve this problem, I'd love to hear it. But, having
| spent some time trying to think of one myself, I'm not sure how
| it would be done without limiting the ability to express code
| that one might reasonably want to be able to write, and without
| making fundamental changes to the design of Rust async as a
| whole.
|
| [1] https://doc.rust-
| lang.org/stable/std/future/trait.Future.htm... [2]:
| https://doc.rust-lang.org/stable/std/future/trait.Future.htm...
| dap wrote:
| (author here)
|
| Although the design of the `tokio::select!` macro creates ways
| to run into this behavior, I don't believe the problem is
| specific to `tokio`. Why wouldn't the example from the post
| using Streams happen with any other executor?
| OptionOfT wrote:
| Minor nitpick:
|
| > In this case, what's dropped is &mut future1
|
| No, nothing is dropped as the `select!` macro doesn't own the
| future. The reason you'd do this is to be able to poll the future
| again:
|
| https://tokio.rs/tokio/tutorial/select#resuming-an-async-ope...
|
| The `select!` macro cancels the other branches by dropping them
| and dropping a reference in Rust doesn't do anything.
|
| But still, this is an amazing blog-post. It does feel like the
| unfortunate marriage of `&mut branch` not dropping (because not
| owned by `select!`) and futures not advancing on their own.
|
| Things that require quite detailed Rust knowledge.
| DAlperin wrote:
| I think the next sentence clarifies pretty well.
|
| > In this case, what's dropped is &mut future1. But future1 is
| not dropped, so the actual future is not cancelled.
| oconnor663 wrote:
| The author clearly understands these details. I think it's just
| a question of wording: did we "drop a reference (which has no
| effect)" or did we "not drop anything (because references don't
| implement Drop)"?
| singron wrote:
| This sounds very similar to priority inversion. E.g. if you have
| Thread T_high running at high priority and thread T_low running
| at low priority, and T_low holds a lock that T_high wants to
| acquire, T_high won't get to run until T_low gets scheduled.
|
| The OS can detect this and make T_low "inherit" the priority of
| T_high. I wonder if there is a similar idea possible with tokio?
| E.g. if you are awaiting a Mutex held by a future that "can't
| run", then poll that future instead. I would guess detecting the
| "can't run" case would require quite a bit of overhead, but maybe
| it can be done.
|
| I think an especially difficult factor is that you don't even
| need to use a direct await. let future1 =
| do_async_thing("op1", lock.clone()).boxed();
| tokio::select! { _ = &mut future1 => {
| println!("do_stuff: arm1 future finished"); }
| _ = sleep(Duration::from_millis(500)) => { // No
| .await, but both will futurelock on future1.
| tokio::select! { _ = do_async_thing("op2",
| lock.clone()) => {}, _ = do_async_thing("op3",
| lock.clone()) => {}, }; } };
|
| I.e. so "can't run" detector needs to determine that no other
| task will run the future, and the future isn't in the current set
| of things being polled by this task.
| oconnor663 wrote:
| > I wonder if there is a similar idea possible with tokio? E.g.
| if you are awaiting a Mutex held by a future that "can't run",
| then poll that future instead.
|
| Something like this could make sense for Tokio _tasks_. (I don
| 't know how complicated their task scheduler is; maybe it
| already does stuff like this?) But it's not possible for
| futures _within_ a task, as in this post. This goes all the way
| back to the "futures are inert" design of async Rust: You
| don't necessarily need to communicate with the runtime at all
| to create a future or to poll it or to stop polling it. You
| only need to talk to the runtime at the _task_ level, either to
| spawn new tasks, or to wake up your own task. Futures are
| pretty much just plain old structs, and Tokio doesn 't know how
| many futures my async function creates internally, any more
| than it knows about my integers or strings or hash maps.
| mycoliza wrote:
| Yeah, a coworker coming from Go asked a similar question
| about why Rust doesn't have something like the Go runtime's
| deadlock detector. Your comment is quite similar to the
| explanation I gave him.
|
| Go, unlike Rust, does not really have a notion of intra-task
| concurrency; goroutines are the fundamental unit of
| concurrency _and_ parallelism. So, the Go runtime can reason
| about dependencies between goroutines quite easily, since
| goroutines are the things which it is responsible for
| scheduling. The fact that channels are a language construct,
| rather than a library construct implemented _in_ the
| language, is necessary for this too. In (async) Rust, on the
| other hand, tasks are the fundamental unit of parallelism,
| but _not_ of concurrency; concurrency emerges from the
| composition of `Future`s, and a single task is a state
| machine which may execute any number of futures concurrently
| (but not in parallel), by polling them until they cannot
| proceed without waiting and then moving on to poll another
| future until _it_ cannot proceed without waiting. But
| critically, this is _not what the task scheduler sees_ ; it
| interacts with these tasks as a single top-level `Future`,
| and is not able to look inside at the nested futures they are
| composed of.
|
| This specific failure mode can actually _only happen_ when
| multiple futures are polled concurrently _but not in
| parallel_ within a single Tokio task. So, there is actually
| no way for the Tokio scheduler to have insight into this
| problem. You could imagine a deadlock detector in the Tokio
| runtime that operates on the _task_ level, but it actually
| could _never_ detect this problem, because when these
| operations execute in parallel, it actually cannot occur. In
| fact, one of the suggestions for how to avoid this issue is
| to select over spawned tasks rather than futures within the
| same task.
| newpavlov wrote:
| >This goes all the way back to the "futures are inert" design
| of async Rust
|
| Yeap. And this footgun is yet another addition to the long
| list of reasons why I consider the Rust async model with its
| "inert" futures managed in user space a fundamentally flawed
| un-Rusty design.
| Veserv wrote:
| I thought Rust async is a colored stackless coroutine model and
| thus it would be unsafe to continue execution of previously
| executing async functions.
|
| To explain, generally speaking, stackless coroutine async only
| need coloring because they are actually "independent stack"less
| coroutines. What they actually do is that they share the stack
| for their local state. This forces async function execution to
| proceed in LIFO order so you do not blow away the stack of the
| async function executing immediately after which demands state
| machine transforms to be safe. This is why you need coloring
| unlike stackful coroutine models which can execute, yield, and
| complete in arbitrary order since their local state is
| preserved in a safe location.
| treyd wrote:
| > thus it would be unsafe to continue execution of previously
| executing async functions.
|
| There's more nuance than this. You can keep polling futures
| as often as you want. When an async fn gets converted into
| the state machine, yielding is just expressed as the poll fn
| returning as not ready.
|
| So it is actually possible for "a little bit" of work to
| happen, although that's limited and gets tricky because the
| way wakers work ensure that normally futures only get polled
| by the runtime when there's actually work for them to do.
| elchananHaas wrote:
| My view of this is that its closer to the basic 2 lock
| Deadlock.
|
| Thread 1 acquires A. Thread 2 acquires B. Thread 1 tries to
| acquire B. Thread 2 tries to acquire A.
|
| In this case, the role "A" is being played by the front of the
| Mutex's lock queue. Role "B" is being played by the Tokio's
| actively executed task.
|
| Based on this understanding, I agree that the surprising
| behavior is due to Tokio's Mutex/Lock Queue implementation. If
| this was an OS Mutex, and a thread waiting for the Mutex can't
| wake for some reason, the OS can wake a different thread
| waiting for that Mutex. I think the difficulty in this approach
| has to do with how Rust's async is implemented. My guess is the
| algorithm for releasing a lock goes something like this:
|
| 1. Pop the head of the wait queue. 2. Poll the top level
| tokio::spawn'ed task of the Future that is holding the Mutex.
|
| What you want is something like this
|
| For each Future in the wait queue (Front to Back): Poll the
| Future. If Success - Break ???Something if everything fails???
|
| The reason this doesn't work has to do with how futures
| compose. Futures compile to states within a state machine. What
| happens when a future polled within the wait queue completes?
| How is control flow handed back to the caller?
|
| I guess you might be able to have some fallback that polls the
| futures independently then polls the top level future to try
| and get things unstuck. But this could cause confusing behavior
| where futures are being polled even though no code path within
| your code is await'ing them. Maybe this is better though?
| johnisgood wrote:
| Off-topic but that code looks quite... complicated as opposed
| to what I would write in Erlang, Elixir, Go, or even C. Maybe
| it is just me.
| arjie wrote:
| Wow, that makes sense afterwards but I would not have guessed at
| it immediately looking at the code. Very insidious. Great
| blogpost.
| forrestthewoods wrote:
| I feel like I'm pretty good at writing multithreaded code. I've
| done it a lot. As long as you use primitives like Rust Mutex that
| enforce correctness for data access (ie no accessing data without
| the lock) it's pretty simple. Define a clean boundary API and
| you're off to the races.
|
| async code is so so so much more complex. It's so hard to read
| and rationalize. I could not follow this post. I tried. But it's
| just a full extra order of complexity.
|
| Which is a shame because async code is supposed to make code
| simpler! But I'm increasingly unconfident that's true.
| wbl wrote:
| Sadly I'm away from my bookshelf but I think Concurrent ML solved
| this issue.
| octoberfranklin wrote:
| _It's really important to understand what's happening here_
|
| Then maybe you should take a moment to pick more descriptive
| identifiers than future1, future2, future3, do_stuff, and
| do_async_thing. This coding style is atrocious.
| Matthias247 wrote:
| As far as I remember from building these things with others
| within the async rust ecosystem (hey Eliza!) was that there was a
| certain tradeoff: if you wouldn't be able to select on
| references, you couldn't run into this issue. However you also
| wouldn't be able run use select! in a while loop and try to
| acquire the same lock (or read from the same channel) without
| losing your position in the queue.
|
| I fully agree that this and the cancellation issues discussed
| before can lead to surprising issues even to seasoned Rust
| experts. But I'm not sure what really can be improved under the
| main operating model of async rust (every future can be dropped).
|
| But compared to working with callbacks the amount of surprising
| things is still rather low :)
| octoberfranklin wrote:
| > However you also wouldn't be able run use select! in a while
| loop and try to acquire the same lock (or read from the same
| channel) without losing your position in the queue.
|
| No, just have select!() on a bunch of owned Futures return the
| futures that weren't selected instead of dropping them. Then
| you don't lose state. Yes, this is awkward, but it's the only
| logically coherent way. There is probably some macro voodoo
| that makes it ergonomic. But even this doesn't fix the root
| cause because dropping an owned Future isn't guaranteed to
| cancel it cleanly.
|
| For the real root cause:
| https://news.ycombinator.com/item?id=45777234
| mycoliza wrote:
| > No, just have select!() on a bunch of owned Futures return
| the futures that weren't selected instead of dropping them.
| Then you don't lose state.
|
| How does that prevent this kind of deadlock? If the owned
| future has acquired a mutex, and you return that future from
| the select so that it might be polled again, and the user
| assigns it to a variable, then the future that has acquired
| the mutex but has not completed is still not dropped. This is
| basically the same as polling an `&mut future`, but with more
| steps.
| octoberfranklin wrote:
| > How does that prevent this kind of deadlock?
|
| Like I said, it doesn't:
|
| > even this doesn't fix the root cause because dropping an
| owned Future isn't guaranteed to cancel it cleanly.
|
| It fixes this:
|
| > However you also wouldn't be able run use select! in a
| while loop and try to acquire the same lock (or read from
| the same channel) without losing your position in the
| queue.
|
| If you want to fix the root cause, see
| https://news.ycombinator.com/item?id=45777234
| mycoliza wrote:
| Indeed, you are correct (and hi Matthias!). After we got to the
| bottom of this deadlock, my coworkers and I had one of our
| characteristic "how could we have prevented this?"
| conversations, and reached the somewhat sad conclusion that
| actually, there was basically nothing we could easily _blame_
| for this. All the Tokio primitives involved were working
| precisely as they were supposed to. The only thing that would
| have prevented this without completely re-designing Rust 's
| async from the ground up would be to ban the use of `&mut
| future`s in `select!`...but that eliminates a lot of _correct_
| code, too. Not being able to do that would make it pretty hard
| to express a lot of things that many applications might
| reasonably want to express, as you described. I discussed this
| a bit in this comment[1] as well.
|
| On the other hand, it also wasn't our coworker who had written
| the code where we found the bug who was to blame, either. It
| wasn't a case of sloppy programming; he had done everything
| correctly and put the pieces together the way you were supposed
| to. All the pieces worked as they were supposed to, and his
| code seemed to be using them correctly, but the interaction of
| these pieces resulted in a deadlock that it would have been
| very difficult for him to anticipate.
|
| So, our conclusion was, wow, this just kind of sucks. Not an
| indictment of async Rust as a whole, but an unfortunate
| emergent behavior arising from an interaction of individually
| well-designed pieces. Just something you gotta watch out for, I
| guess. And that's pretty sad to have to admit.
|
| [1] https://news.ycombinator.com/item?id=45776868
| octoberfranklin wrote:
| For anybody who wants to cut to the chase, it's this:
|
| > _The behavior of tokio::select! is to poll all branches '
| futures only until one of them returns `Ready`. At that point, it
| drops the other branches' futures and only runs the body of the
| branch that's ready._
|
| This is, unfortunately, doing what it's supposed to do: acting as
| a footgun.
|
| The design of tokio::select!() implicitly assumes it can cancel
| tasks cleanly by simply dropping them. We learned the hard way
| back in the Java days that _you cannot kill threads cleanly all
| the time_. Unsurprisingly, the same thing is true for async
| tasks. But I guess every generation of programmers has to re-
| learn this lesson. Because, you know, actually learning from
| history would be too easy.
|
| Unfortunately there are a bunch of footguns in tokio (and async-
| std too). The state-machine transformation inside rustc is a
| thing of beauty, but the libraries and APIs layered on top of
| that should have been iterated many more times before being
| rolled out into widespread use.
| littlestymaar wrote:
| I genuinely don't understand why people use _select!_ at all
| given how much of a footgun it is.
| octoberfranklin wrote:
| Well the less-footgun-ish alternative would look something
| like a Stream API, but the last time I checked tokio-stream
| wasn't stable yet.
|
| Then you could merge a `Stream<A>` and `Stream<B>` into a
| `Stream<Either<A,B>>` and pull from that. Since you're
| dealing with owned streams, dropping the stream forces _some_
| degree of cleanup. There are still ways to make a mess, but
| they take more effort.
| ....................................
|
| Ratelimit so I have to reply to mycoliza with an edit here:
|
| That example calls `do_thing()`, whose body does not appear
| anywhere in the webpage. Use better identifiers.
|
| If you meant `do_stuff()`, you haven't replaced select!()
| with streams, since `do_stuff()` calls `select!()`.
|
| The problem is `select!()`; if you keep using `select!()` but
| just slather on a bunch of streams that isn't going to fix
| anything. You have to _get rid of select!()_ by replacing it
| with streams.
| mycoliza wrote:
| An analogous problem is equally possible with streams: http
| s://rfd.shared.oxide.computer/rfd/0609#_how_you_can_hit_...
| hitekker wrote:
| Skimming through, this document feels thorough and transparent.
| Clearly, a hard lesson learned. The footnotes, in particular,
| caught my eye
| https://rfd.shared.oxide.computer/rfd/397#_external_referenc...
|
| > Why does this situation suck? It's clear that many of us
| haven't been aware of cancellation safety and it seems likely
| there are many cancellation issues all over Omicron. It's awfully
| stressful to find out while we're working so hard to ship a
| product ASAP that we have some unknown number of arbitrarily bad
| bugs that we cannot easily even find. It's also frustrating that
| this feels just like the memory safety issues in C that we
| adopted Rust to get away from: there's some dynamic property that
| the programmer is responsible for guaranteeing, the compiler is
| unable to provide any help with it, the failure mode for getting
| it wrong is often undebuggable (by construction, the program has
| not done something it should have, so it's not like there's a log
| message or residual state you could see in a debugger or
| console), and the failure mode for getting it wrong can be
| arbitrarily damaging (crashes, hangs, data corruption, you name
| it). Add on that this behavior is apparently mostly undocumented
| outside of one macro in one (popular) crate in the async/await
| ecosystem and yeah, this is frustrating. This feels antithetical
| to what many of us understood to be a core principle of Rust,
| that we avoid such insidious runtime behavior by forcing the
| programmer to demonstrate at compile-time that the code is well-
| formed
| Dagonfly wrote:
| That's a really subtle version of the deadlock described in
| withoutboats FuturesUnordered post [0]
|
| When using "intra-task" concurrency, you really have to ensure
| that none of the futures are starving.
|
| Spawning task should probably be the default. For timeouts use
| tokio::select! but make sure all pending futures are owned by it.
| I would never recommend FuturesUnordered unless you really test
| all edge-cases.
|
| [0] https://without.boats/blog/futures-unordered/
___________________________________________________________________
(page generated 2025-10-31 23:00 UTC)