[HN Gopher] Waiting for many things at once with io_uring
___________________________________________________________________
Waiting for many things at once with io_uring
Author : ashvardanian
Score : 48 points
Date : 2024-11-14 12:04 UTC (4 days ago)
(HTM) web link (mazzo.li)
(TXT) w3m dump (mazzo.li)
| jchw wrote:
| io_uring and Linux's many different types of file descriptors are
| great. I mean, I personally think that the explicit large API
| surface of WinNT is kinda nicer than jamming a bunch of weird
| functionality into files and file descriptors like Linux, but
| when things work, they do show some nice advantages of unifying
| everything to some framework, ill-fitting as it may sometimes be
| (Though now that I say this, it's not like WinNT Objects are
| really any different here, they just offer more advanced baseline
| functionality like ACLs). io_uring and it's ability to tie
| together a lot of pre-existing things in new ways is pretty cool.
| UNIX never really had a story for async operations, something I
| will not fault an OS designed 50 years ago for. However, still
| not having a decent story for async operations today is harder to
| excuse. I've been excited to learn about io_uring. I've learned a
| lot listening to conference talk recordings about it. While it
| has its issues (like the many times it (semi-?)accidentally
| bypassed security subsystems...) it has some really cool and
| substantial benefits.
|
| I'll tell you what I would love to see next: a successor to
| inotify that does not involve opening one zillion file
| descriptors to watch a recursive subtree. I'm sure there are
| valid reasons why it's not easy to just make it happen, but it
| feels like it will be a major improvement in a lot of use cases.
| And in many cases, it would probably fix the dreaded problem of
| users needing to fight against ulimits, especially in text
| editors like VSCode.
|
| I don't have anything of great substance to say about the actual
| subject of the article. It feels a bit late to finally get this
| functionality proper in Linux after NT had it basically forever,
| but any improvement is welcome. Next time I'm doing something
| where I want to wait on a bunch of FDs I will have to try this
| approach.
| trws wrote:
| An inotify replacement that can work at whole FS level (and
| doesn't require root/admin like the existing option) would be
| amazing. To be honest, I don't see a reason it would be hard at
| the whole filesystem or perhaps mount level unless there are
| security ramifications. Restricting it to a subdirectory might
| be tricky though.
| hansvm wrote:
| > inotify
|
| A hack that should be performant enough if properly implemented
| would be a custom FUSE implementation over the directory. As a
| one-off it could just do the callbacks you want done, or as a
| reusable component it could implement the inotify behavior you
| want.
| iknowstuff wrote:
| fanotify?
|
| https://man7.org/linux/man-pages/man7/fanotify.7.html
| jchw wrote:
| It feels like last time I looked into this, fanotify was for
| some reason not suitable for most inotify use cases. Maybe
| this has changed. Would be great news if so.
| jeffbee wrote:
| Some of the things that you cannot wait on using io_uring are
| your kernel actually supporting the feature mentioned in the
| article, io_uring actually working properly, and io_uring solving
| its seemingly bottomless supply of local user exploits. In the
| early days of this feature I was bullish but the way its
| implementation has emitted CVEs has not been a source of joy, and
| now many major Linux operators have banned the API internally.
| Maybe what is needed is a moment of reflection and a scratch
| reimplementation that learns the lessons of io_uring?
| loeg wrote:
| A new from-scratch implementation would suffer from a similar
| problem as early io_uring did (high rate of code change, which
| seems to be what drives security bug rates).
| 4hg4ufxhy wrote:
| Very interesting, but unfortunate there is no example program. I
| guess that is left as exercise for reader, but it's a bit
| daunting for a non systems programmer.
| tux1968 wrote:
| For c there is :
|
| https://git.kernel.dk/cgit/liburing/tree/examples
|
| There's also a minimal Rust example for tokio:
|
| https://github.com/tokio-rs/io-uring/tree/master/examples
| refulgentis wrote:
| It took me many io_uring hello world articles to find out it's
| not really used in production (ex. Android and ChromeOS both
| disable it) because it was, and continues to be, a source of an
| absolutely bonkers outsized # of security issues.
|
| I don't remember much more than that*, but just dropping it here
| because I learned a _ton_ more from reading about that, than my
| Nth io_uring article.
|
| * for example, the article mentioning relevant buffers are shared
| with the system made me want to say "aHA, yes, that's what the
| security articles said was a core issue!" -- but I can't actually
| remember with 100% confidence
| loeg wrote:
| Well, it's not true that it isn't used in production. Google
| has been burned and at least historically did not use it. But I
| know some services at Facebook use it in production.
|
| Yes, historically it was a big source of security bugs. I think
| that has tapered off somewhat as the rate of change slows down.
| hosh wrote:
| Discussion thread in the Erlang community proposing implementing
| io_uring for BEAM, security issues, and a digression comparing it
| to FreeBSD's kqueues
|
| https://erlangforums.com/t/erlang-io-uring-support/765/18?pa...
| ashvardanian wrote:
| Surprisingly, I only came across Francesco's blog this month. I
| stumbled upon the 2021 post "Speeding up atan2f by 50x" while
| searching for others who have to reimplement trigonometry in SIMD
| every other year. I've also enjoyed "Beating the L1 cache with
| value speculation" from the same year, as well as the 2013 Agda
| sorting example.
|
| Highly recommend checking it out: https://mazzo.li/archive.html
___________________________________________________________________
(page generated 2024-11-18 23:00 UTC)