https://github.com/tokio-rs/tokio-uring/pull/1 Skip to content Sign up Sign up * Why GitHub? Features - + Mobile - + Actions - + Codespaces - + Packages - + Security - + Code review - + Project management - + Integrations - + GitHub Sponsors - + Customer stories- * Team * Enterprise * Explore + Explore GitHub - Learn and contribute + Topics - + Collections - + Trending - + Learning Lab - + Open source guides - Connect with others + The ReadME Project - + Events - + Community forum - + GitHub Education - + GitHub Stars program - * Marketplace * Pricing Plans - + Compare plans - + Contact Sales - + Education - [ ] [search-key] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up Sign up {{ message }} tokio-rs / tokio-uring * Notifications * Star 9 * Fork 1 * Code * Issues 0 * Pull requests 1 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights New issue Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community. Pick a username [ ] Email Address [ ] Password [ ] [ ] Sign up for GitHub By clicking "Sign up for GitHub", you agree to our terms of service and privacy statement. We'll occasionally send you account related emails. Already on GitHub? Sign in to your account Jump to bottom Tokio-uring design proposal #1 Open carllerche wants to merge 9 commits into master base: master from design-doc Open Tokio-uring design proposal #1 carllerche wants to merge 9 commits into master from design-doc +756 -0 Conversation 58 Commits 9 Checks 0 Files changed 1 Conversation @carllerche Copy link Quote reply Member @carllerche carllerche commented Mar 30, 2021 * edited Rendered The RFC proposes a new asynchronous Rust runtime backed by io-uring as a new crate: tokio-uring. The API aims to be as close to idiomatic Tokio, but will deviate when necessary to provide full access to io-uring's capabilities. It also will be compatible with existing Tokio libraries. The runtime will use an isolated thread-per-core model, and many types will be !Send. The repo contains a proof-of-concept implementation. [?] 6 7 7 @carllerche Initial design doc draft Verified This commit was signed with the committer's verified signature. [6180] carllerche Carl Lerche GPG key ID: AA14CE6F061D8F7A Learn about signing commits 44b6e3e @bdonlan bdonlan reviewed Mar 30, 2021 View changes DESIGN.md spawn(async move { ... }); } }) ``` This comment has been minimized. Sign in to view @bdonlan bdonlan Mar 30, 2021 It'd probably be best for there to be a clear model for how we might move an IO object from one thread to another, should such advanced load balancing become required. For example, a webserver might find that a bunch of high bandwidth clients ended up on a single thread, overloading that thread. I'm actually of the opinion that an explicit runtime handle might be the way to deal with this. Consider something like this sketch: struct LocalRuntime { ... } // !Sync fn spawn_new_group + Send>(f: FnOnce(&LocalRuntime)->F) -> GroupHandle; impl LocalRuntime { fn spawn + Send>(f: impl FnOnce(&LocalRuntime)->F) -> SpawnHandle; } impl TcpStream { fn bind_to_runtime<'a>(self, runtime: &'a LocalRuntime) -> LocalTcpStream<'a>; } Here we're spawning a bunch of tasks in a group, which lives on a single runtime. The tokio runtime can migrate this group, if needed (perhaps when instructed by the application); this might involve tracking the uring tasks issued by this group, and either forwarding results cross-thread or cancelling and reissuing those tasks during migration. Within the group, we can avoid synchronization overhead most of the time - ie, when we're not actively in the middle of a migration - as we know that all of our IO object handles are not shared between threads. This comment has been minimized. Sign in to view @carllerche carllerche Apr 6, 2021 Author Member Such a system could be implemented as an additional crate or at the application level. I am inclined to punt for now and explore balancing strategies in a "real app". DESIGN.md Outdated Show resolved Hide resolved @sfackler sfackler reviewed Mar 30, 2021 View changes DESIGN.md Outdated Show resolved Hide resolved @gardnervickers gardnervickers reviewed Mar 31, 2021 View changes DESIGN.md Outdated Show resolved Hide resolved @gardnervickers gardnervickers reviewed Mar 31, 2021 View changes DESIGN.md Outdated Show resolved Hide resolved @Kestrer Kestrer reviewed Mar 31, 2021 View changes DESIGN.md to receive the operation result. Interestingly, the resources, e.g., `TcpListener`, can be `Send` as long as they do not hold operation futures Comment on lines +103 to +104 This comment has been minimized. Sign in to view @Kestrer Kestrer Mar 31, 2021 I think this would prevent tokio-io_uring from being able to register its resource file descriptors with the thread local io_uring instance - doing so could potentially improve performance. This comment has been minimized. Sign in to view @carllerche carllerche Apr 7, 2021 Author Member You are correct. I punted discussing registering files. My gut is that there will be a Registered<_> decorator type maybe that forces the inner type to be !Send. This can be added later though. DESIGN.md Show resolved Hide resolved DESIGN.md impl File { async fn read_at(&self, buf: buf::Slice, pos: u64) -> BufResult< usize>; async fn write_at(&self, buf: buf::Slice, pos: u64) -> BufResult< usize>; This comment has been minimized. Sign in to view @Kestrer Kestrer Mar 31, 2021 Similarly, it should support writing from shared buffers, such as Bytes or Arc<[u8]>. I think using AsRef<[u8]> would actually be the best option here, since it's the most flexible. DESIGN.md ```rust impl TcpStream { async fn close(self) { ... } This comment has been minimized. Sign in to view @Kestrer Kestrer Mar 31, 2021 How does this differ from or relate to AsyncWrite's shutdown? Could these two APIs be unified somehow? This comment has been minimized. Sign in to view @glommer glommer Mar 31, 2021 I don't think this is an "instead" question. For glommio, I always provide a shutdown and/or close functions. However there are situations in which it is simply not possible to call them. One example are traits that contains consuming functions, but there are also situations for instance where an LSM compaction finishes operating on a file stream, but there are still readers (you can have logic to wait for the readers, it is just too complex) The best way is to provide indeed a close function and steer users towards using it, but you'll have to choose between one of the mechanisms for close-on-drop. I heavily prefer async-close-on-drop. The "too-many-files" is not a big problem in practice, because the queueing into the ring is always synchronous, and ultimately you have control over when to dispatch. It's even possible to keep a counter of files that are asked-to-close-but-not-yet-closed, and force io_uring_submit dispatch when the number becomes too big. Calling the system call itself is non-blocking This comment has been minimized. Sign in to view @Kestrer Kestrer Mar 31, 2021 Sorry, I'm not quite sure what you're talking about here - both shutdown and close are asynchronous. Either way, after reading more in to the difference between shutdown and close I finally understand why they are different: shutdown only shuts down the write end of the TcpStream and doesn't take ownership, whereas close is shutdown followed by actually closing the fd. I still think it's a bit confusing to have two AsyncDrop-like mechanisms, but I don't see an obvious way around it. DESIGN.md Outdated Show resolved Hide resolved @glommer glommer reviewed Mar 31, 2021 View changes DESIGN.md Outdated lifecycle to `Ignored` and submits a cancellation request to the kernel. The cancellation request will attempt to terminate the operation, causing it to complete immediately with an error. Cancellation is best-effort; the operation may or may not terminate early. If the operation does complete, the runtime This comment has been minimized. Sign in to view @glommer glommer Mar 31, 2021 That's indeed a good behavior, just a heads up on a situation where you want to not ignore the result: timeouts. That's specially true for linked SQE timeouts (where you send the operation, and automatically link a timeout). In that case you don't want to ignore the result. @carllerche carllerche reviewed Apr 5, 2021 View changes DESIGN.md Outdated Show resolved Hide resolved @carllerche carllerche reviewed Apr 5, 2021 View changes DESIGN.md Show resolved Hide resolved carllerche added 7 commits Apr 6, 2021 @carllerche apply tweaks Verified This commit was signed with the committer's verified signature. [6180] carllerche Carl Lerche GPG key ID: AA14CE6F061D8F7A Learn about signing commits 4239d13 @carllerche Move TcpStream read/write ops to an alternative section Verified This commit was signed with the committer's verified signature. [6180] carllerche Carl Lerche GPG key ID: AA14CE6F061D8F7A Learn about signing commits cfb3334 @carllerche add some use cases Verified This commit was signed with the committer's verified signature. [6180] carllerche Carl Lerche GPG key ID: AA14CE6F061D8F7A Learn about signing commits 8e7219f @carllerche remove todo Verified This commit was signed with the committer's verified signature. [6180] carllerche Carl Lerche GPG key ID: AA14CE6F061D8F7A Learn about signing commits edebcc0 @carllerche add some more alternatives Verified This commit was signed with the committer's verified signature. [6180] carllerche Carl Lerche GPG key ID: AA14CE6F061D8F7A Learn about signing commits 486665e @carllerche expand on Slice uninitialized memory Verified This commit was signed with the committer's verified signature. [6180] carllerche Carl Lerche GPG key ID: AA14CE6F061D8F7A Learn about signing commits 5414dbe @carllerche load-balancing spawn fn Verified This commit was signed with the committer's verified signature. [6180] carllerche Carl Lerche GPG key ID: AA14CE6F061D8F7A Learn about signing commits 0176a1b @quininer quininer reviewed Apr 7, 2021 View changes DESIGN.md assert_eq!(slice.capacity(), 100); ``` A trait argument for reading and writing may be possible as a future This comment has been minimized. Sign in to view @quininer quininer Apr 7, 2021 I think we also need 'static constraints. like https://github.com/ quininer/ritsu/blob/master/src/actions/io.rs#L16 DESIGN.md // ensures there are a minimum number of workers // in the runtime that are flagged as with capacity // to avoid total starvation. current_worker::wait_for_capacity().await; This comment has been minimized. Sign in to view @quininer quininer Apr 7, 2021 I like this. DESIGN.md ## Reading and writing Read and write operations require passing ownership of buffers to the kernel. This comment has been minimized. Sign in to view @quininer quininer Apr 7, 2021 I think fd also needs to pass ownership. DESIGN.md /// /// This is implemented as a new type to implement std::ops::Try once /// the trait is stabilized. type BufResult = (std::io::Result, buf::Slice); This comment has been minimized. Sign in to view @quininer quininer Apr 7, 2021 This seems a bit awkward because users cannot directly use ?. Maybe we need a custom Error type that allows fd and buffer to be take out from it. or let user pass &mut Option, like https:// github.com/quininer/ritsu/blob/master/src/actions/io.rs#L18 This comment has been minimized. Sign in to view @seanmonstar seanmonstar Apr 8, 2021 Member The newtype could implement a method to convert into a Result and drop the buffer on errors, I suppose. let buf = file.read_(0, buf).await.result()?; or something. DESIGN.md Vec(Vec), /// Buffer pool backed buffer. The pool is managed by io-uring. Provided(ProvidedBuf), This comment has been minimized. Sign in to view @quininer quininer Apr 7, 2021 I slightly suspect that ProvidedBuf is not worth considering too early, because I have not observed that it has too much optimization in terms of performance and memory usage, and it is not useful for file, and the handling when the pool is exhausted is tricky. This comment has been minimized. Sign in to view @carllerche carllerche Apr 8, 2021 Author Member Those are good points for registering buffers, but what about for read operations to a buffer pool instead of an explicit buffer, that isn't about performance but about reducing memory overhead. DESIGN.md my_tcp_stream.close().await; ``` The resource must still tolerate the caller dropping it without being explicitly This comment has been minimized. Sign in to view @quininer quininer Apr 7, 2021 In this case, tokio-uring will close the resource in the background, avoiding blocking the runtime. We can be optimistic that closing will not cause congestion, because we also do this under epoll. The drop handler will move ownership of the resource handle to the runtime and submit cancellation requests for any in-flight operation. Once all existing in-flight operations complete, the runtime will submit a close operation. This sounds quite complicated, and maybe it's easier to use reference counting. This comment has been minimized. Sign in to view @carllerche carllerche Apr 8, 2021 Author Member "move ownership" is probably a poor choice of words here. It is a logical transfer of ownership. The value is already stored in the runtime's internal storage. The implementation changes the operation state to Ignored which signals to the runtime that it owns that state now. @pnkfelix pnkfelix reviewed Apr 8, 2021 View changes DESIGN.md socket, to the kernel using the submission queue. The kernel then performs the operation. On completion, the kernel returns the operation results via the completion queue and notifies the process. The `io_uring_enter` syscall flushes the submission queue and acquires any pending completion events. Upon request, This comment has been minimized. Sign in to view @pnkfelix pnkfelix Apr 8, 2021 What does it mean for a specific syscall to "acquire a completion event" ? (i.e. The syscall itself cannot own responsibility for handling that event, right? So is it handing off that responsibility to whomever in user space invoked the syscall ?) This comment has been minimized. Sign in to view @pnkfelix pnkfelix Apr 8, 2021 (Or, I guess since the syscall is allowed to block the thread, it can have that responsibility? But below you say waiting for a minimum number of completion events; not all of the events... so I'm still confused I guess.) This comment has been minimized. Sign in to view @carllerche carllerche Apr 8, 2021 Author Member Acquire at the memory ordering level (think atomic load operation w/ Acquire ordering). @seanmonstar seanmonstar reviewed Apr 8, 2021 View changes DESIGN.md This read function takes ownership of the buffer; however, any pointer to the buffer obtained from a value becomes invalid when the value moves. Storing the buffer value at a stable location while the operation is in-flight should be sufficient to satisfy safety. This comment has been minimized. Sign in to view @seanmonstar seanmonstar Apr 8, 2021 Member It's technically safe to return a new slice or pointer on each call to as_mut(), as bonkers as that may be, so it might need something like unsafe trait TrustedAsMut: AsMut<[u8]> {} This comment has been minimized. Sign in to view @quininer quininer Apr 8, 2021 If we hold ownership of T and guarantee to call as_mut only once, I think this is trusted. This comment has been minimized. Sign in to view @carllerche carllerche Apr 8, 2021 Author Member This is correct, but I am pretty sure that as long as as_mut() is called only once and is kept at a stable location, it should be fine.... but that is also why I am punting @seanmonstar seanmonstar reviewed Apr 8, 2021 View changes DESIGN.md The tokio-uring crate targets use-cases that can benefit from taking advantage of io-uring at the expense of discarding Tokio's portable API. These use cases will also benefit from reduced synchronization overhead and fine-grained control over thread load balancing strategies. This comment has been minimized. Sign in to view @seanmonstar seanmonstar Apr 8, 2021 Member I think not requiring synchronization in a sub-library is a good call. Those wanting finer grained control can require pinning to a thread. What about the possibility of making tokio-uring support ! Send or Send generically, so that if someone did want to use a work-stealing multi-threaded scheduler, they can do so safely without wrapping the types in Mutexs? Or is that basically the best solution anyways? @pnkfelix pnkfelix reviewed Apr 8, 2021 View changes DESIGN.md ### Load-balancing spawn function The tokio-uring crate omits a spawn function that balances tasks across tasks, This comment has been minimized. Sign in to view @pnkfelix pnkfelix Apr 8, 2021 "balances tasks across tasks" -- typo? (did you mean "across threads"?) @carllerche carllerche reviewed Apr 8, 2021 View changes DESIGN.md // Provide the buffer pool to the kernel. This passes // ownership of the pool to the kernel. let pool_token = rt.provide_buffers(MY_POOL, 256, 4 * 4096)?; This comment has been minimized. Sign in to view @carllerche carllerche Apr 8, 2021 Author Member This line is incorrect. rt.provide_buffers(my_pool); @seanmonstar seanmonstar reviewed Apr 8, 2021 View changes DESIGN.md kind: Kind, } enum Kind { This comment has been minimized. Sign in to view @seanmonstar seanmonstar Apr 8, 2021 Member Is this just for concept purposes? Do you plan to instead actually use like a vtable design kind of like in bytes? Otherwise accessing the slice will require a match every single time, which means the performance downsides of SBO. This comment has been minimized. Sign in to view @carllerche carllerche Apr 8, 2021 Author Member Might eventually, but I am not worrying about it now. It could be changed to avoid the branch on deref. This comment has been minimized. Sign in to view @carllerche carllerche Apr 8, 2021 Author Member I mean, eventually, we might "just" use bytes::Bytes. @withoutboats Copy link @withoutboats withoutboats commented Apr 8, 2021 Overall there's a lot here that's similar to ringbahn. I'll leave some notes about the differences. ringbahn is specifically designed to be externally extensible in a number of ways: 1. A user of ringbahn can define how io_uring_enter gets called; this is done via what's called the Drive trait. ringbahn's code just requests from the implementation of Drive a certain number of SQEs, and then expects the drive implementer to somehow call ringbahn's complete function to wake tasks or clean up resources as events are completed. This is how ringbahn could allow both multi-threaded and single-threaded implementations (note that this means there is a Mutex internal to ringbahn, basically around its equivalent to this document's Lifecycle type, though it should be very low contention). Obviously, end users are not expected to implement this, but its designed to allow multiple other frameworks built on top of it to share its code. 2. A user of ringbahn can define their own operations; there's no centralized enums of all the possible operations as appears here in the State enum. This only really requires one modification to the core code: instead of storing the buffers in the State enum, the future types owns any memory associated with the operation & if its dropped, the equivalent of the Ignored enum takes ownership and holds a callback to clean up that state. This is not only beneficial because it the set of primitive ops supported by io-uring is frequently growing, but because it could in theory allow users of ringbahn to define their own complex multi-step "operations" that submit multiple SQEs that are linked together and then tying that state to the completion of the whole chain. 3. ringbahn's equivalent of TcpStream etc is not extensible in terms of buffer strategy (i.e. there's no realistic way to use automatic buffer selection with ringbahn), but it is supposed to be if I were able to work on it again in the future. So rather than enumerating the kinds of buffers, it would ideally support arbitrary buffers that implement some trait to be called at the right times. I'm certain this is possible, my dayjob keeps me away from completing the work. My impression is extensibility in this manner has never been a priority for tokio, so I'm not surprised to see the differences. Its certainly easier if you don't do this. The other big difference is that ringbahn's provided IO handles use AsyncBufRead instead of passing ownership in and out. I think any good implementation should support both (ringbahn doesn't provide the ownership passing API, but I believe it would be straightforward to implement on top of its existing APIs). @Diggsey Diggsey reviewed Apr 8, 2021 View changes DESIGN.md for the duration of their lifecycle. Each runtime thread will own a dedicated submission and completion queue pair, and operations are submitted using the submission queue associated with the current thread. Operation completion futures will not implement `Send`, guaranteeing that they remain on the thread This comment has been minimized. Sign in to view @Diggsey Diggsey Apr 8, 2021 * edited This would have quite a large impact on the ecosystem. Is there a possibility to make the futures be Send, but only incur a significant performance penalty if polled on another thread? For example, the source of the !Send bound appears to be the necessity to access the driver state when polling: https://github.com/tokio-rs/tokio-uring/blob/master/src/driver/op.rs# L15 I could imagine a solution where the Op stores an Arc>. The idea behind MagicCell is that it would perform a runtime Sync check (ie. check that the accessing thread is the same as the thread it was created on) before allowing access to the interior. If this check fails, the Op would do something like this pseudocode: let (tx, rx) = Proxy::new(); let original_op = mem::replace(self, rx); original_thread.send(original_op.and_then(tx)); As long as tasks are fairly sticky, then the overhead would only be incurred for in-progress ops when a task is actually moved. Of course, there's the question of whether the overhead of the Arc and MagicCell would outweigh the benefits. @carllerche fix typo Verified This commit was signed with the committer's verified signature. [6180] carllerche Carl Lerche GPG key ID: AA14CE6F061D8F7A Learn about signing commits 4fd50d0 @Diggsey Copy link @Diggsey Diggsey commented Apr 8, 2021 How do you see this being used? Is the idea that application authors would directly build on top of tokio-uring instead of tokio? If so, would it make sense to split the new public API (and associated execution model) from the underlying uring-based implementation, so that applications are still building against a platform-independent interface? (Even if the only back-end is linux specific right now) AIUI io-uring is basically identical to IO completion ports on windows, with the innovation that operations can be submitted without a system-call, so it's plausible that there would be more than one "backend" that would benefit from this execution model. And it would also be desirable to have a fallback implementation based on mio for platforms that cannot support io-uring or other backend directly. @nikomatsakis nikomatsakis suggested changes Apr 8, 2021 View changes Copy link Quote reply @nikomatsakis nikomatsakis left a comment one quick nit DESIGN.md Linux added a new API, io-uring, which reduces overhead by eliminating most syscalls and mapping memory regions used for byte buffers ahead of time. Early benchmarks comparing io-uring against epoll are promising; a TCP echo client and server show up to [60% improvement][bench]. Though not yet measured, using This comment has been minimized. Sign in to view @nikomatsakis nikomatsakis Apr 8, 2021 This is for C benchmarks, right? Probably just add a note like "TCP echo clients and servers implemented in C show up to 60% improvement" 1 @Kestrer Copy link @Kestrer Kestrer commented Apr 8, 2021 io_uring is similar to IOCP in that they're both completion-based, but it has one major difference that makes it really hard to abstract over both: the threading model. io_uring naturally lends itself to a ring-per-thread model where each thread is self-contained and manages its own I/O. For Rust, this means thread-local runtimes, little synchronization and lots of !Send types. On the other hand IOCP is designed to work in work-stealing systems: there is typically one global IOCP instance that executes all I/O, and wakes up an essentially arbitrary thread when a completion status is received. For Rust, this means a single multithreaded work-stealing runtime, lots of synchronization and Send types. One way to reconcile this difference is to always use a runtime-per-thread model, so on Windows there would be an IOCP instance for each thread (I don't know enough about IOCP to say whether this is a bad idea). Alternatively, there could be one global io_uring instance protected by a mutex, or an io_uring instance for each thread where submission queues are shared with a mutex but completion queues are thread-local. I really don't know which solution is best; they all have disadvantages when run on platforms that don't natively support the model. I think in an ideal world, io_uring would support multiple submission/completion queues on a single io_uring instance so we can use a multithreaded work stealing model everywhere, but that might not ever happen. @xonatius xonatius reviewed Apr 8, 2021 View changes DESIGN.md mem: [u8; 10], } impl AsMut<[u8]> { This comment has been minimized. Sign in to view @xonatius xonatius Apr 8, 2021 Suggested change impl AsMut<[u8]> { impl AsMut<[u8]> for MyUnusableBuf { DESIGN.md ```rust async fn read>(&self, dst: T) { ... } struct MyUnsableBuf { This comment has been minimized. Sign in to view @xonatius xonatius Apr 8, 2021 Suggested change struct MyUnsableBuf { struct MyUnusableBuf { DESIGN.md Internally, byte streams submit read operations using the default buffer pool. Additional methods exist to take and place buffers, supporting zero-copy piping between two byte streams. ```rust my_tcp_stream.fill_buf().await?; let buf: IoBuf = my_tcp_stream.take_read_buf(); // Mutate `buf` if needed here. my_other_stream.place_write_buf(buf); my_other_stream.flush().await?; ``` Comment on lines +444 to +456 This comment has been minimized. Sign in to view @xonatius xonatius Apr 8, 2021 Is there a plan to support IORING_OP_SPLICE or IORING_OP_TEE? This comment has been minimized. Sign in to view @carllerche carllerche Apr 8, 2021 Author Member Yes via inherent methods on relevant types. It should fit into the model proposed here. DESIGN.md required synchronization adds overhead. An earlier article on the Tokio blog includes an overview of various scheduling strategies. The tokio-uring crate targets use-cases that can benefit from taking advantage of io-uring at the expense of discarding Tokio's portable API. These use cases will also benefit from reduced synchronization overhead and fine-grained control over thread load balancing strategies. Comment on lines +680 to +687 This comment has been minimized. Sign in to view @xonatius xonatius Apr 8, 2021 Suggested change required synchronization adds overhead. An earlier article on the Tokio blog includes an overview of various scheduling strategies. The tokio-uring crate targets use-cases that can benefit from taking advantage of io-uring at the expense of discarding Tokio's portable API. These use cases will also benefit from reduced synchronization overhead and fine-grained control over thread load balancing strategies. required synchronization adds overhead. [An earlier article] on the Tokio blog includes an overview of various scheduling strategies. The tokio-uring crate targets use-cases that can benefit from taking advantage of io-uring at the expense of discarding Tokio's portable API. These use cases will also benefit from reduced synchronization overhead and fine-grained control over thread load balancing strategies. [An earlier article]: https://tokio.rs/blog/2019-10-scheduler Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment Reviewers @bdonlan bdonlan @seanmonstar seanmonstar @pnkfelix pnkfelix @glommer glommer @xonatius xonatius @Diggsey Diggsey @sfackler sfackler @gardnervickers gardnervickers @quininer quininer @Kestrer Kestrer @nikomatsakis nikomatsakis Assignees No one assigned Labels None yet Projects None yet Milestone No milestone Linked issues Successfully merging this pull request may close these issues. None yet 13 participants @carllerche @withoutboats @Diggsey @Kestrer @bdonlan @seanmonstar @nikomatsakis @pnkfelix @glommer @xonatius @sfackler @gardnervickers @quininer Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. * (c) 2021 GitHub, Inc. * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.