Post B58JGpBLvrezAdAA2y by gabrielesvelto@mas.to
(DIR) More posts by gabrielesvelto@mas.to
(DIR) Post #B57A3Lt4RfEFnu9Wm8 by ska@social.treehouse.systems
2 likes, 0 repeats
So, systemd's NOTIFY_SOCKET readiness notification mechanism has a sort of authentication mechanism by pid: it doesn't like it when the notification is sent from another process than the main one, unless you add the NotifyAccess=all directive. Okay, fair.Except that even with this directive, it still can fail for some reason:if an auxiliary process of the unit sends an sd_notify() message and immediately exits, the service manager might not be able to properly attribute the message to the unit, and thus will ignore it, even if NotifyAccess=all is set for it.Huh? Even if the supervisor wants to check that the MAINPID= given in the message is the relevant unit, it shouldn't need the notifier process to still be around, but, let's ignore that for now. So, it has a barrier mechanism to tell the notifier to stick around until the supervisor has acknowledged it. What is that barrier mechanism? The notifier passes an fd to the supervisor via SCM_RIGHTS in the notification message, and when the supervisor closes that fd, it means it has acked the notification, and the notifier can now exit.That's right, folks: it's a notification that the notification has been received!There are two mistakes here: 1. the supervisor doesn't need the notifier process to be alive when processing the notification, and 2. NOTIFY_SOCKET being a socket means that any process can connect to it so the supervisor needs to authenticate the relevant unit.And that is how a bad design decision cascades into a maze of complexity and inefficiency, and a pain to implement. Unless you're willing to use libsystemd to get access to sd_pid_notify_barrier(), good luck getting this to work right.(systemd people will say this is conspiratorial, but if they wanted to force users to link against libsystemd, they wouldn't behave differently. But I really don't care if it's intentional or not. If it's not intentional, it's just terrible engineering.)What would be a simpler way to handle all this? Have the supervisor run the unit with a pipe from the daemon to the supervisor, not a socket anyone can connect to. Processes that can write to the pipe are the ones that can inherit it, which means they're necessarily part of the same unit. No need for authentication via pid. When the supervisor receives a certain token on the pipe (e.g. a newline), it means that the daemon is ready. In the daemon or any process notifying on its behalf, that spells: write(fd, "\n", 1); and that's it.In other words: the s6 readiness notification protocol, which is exactly what is described in the above paragraph, is as efficient as systemd's synchronization protocol for its notification protocol.The real tragedy is that when you go through systemd with a looking glass, every part of it is this way. It still works, and nobody notices the inefficiencies because modern machines are blazingly fast, but it is. so. bad.(Edit: typos)
(DIR) Post #B57ATulDPSZ1UU90gy by lanodan@queer.hacktivis.me
0 likes, 0 repeats
@ska Mades me wonder how well it works for OpenRC given it supports both styles (although just READY=1 of NOTIFY_SOCKET, which uuhh I guess hints there's more than that… why???).
(DIR) Post #B57HsXpIVyPPHRkHK4 by ska@social.treehouse.systems
1 likes, 0 repeats
@lanodan I assume OpenRC doesn't authenticate the pid that connects to the notification socket, which isn't a problem in practice. If OpenRC wanted to fully harden against evil services notifying readiness for other services, its supervisor would need to add a whole layer of complexity for that.But even then, I don't think it would need a synchronization mechanism, because who cares if the notifier is still around, as long as the main pid is?s6's new s6-notify-fd-from-socket binary, doing the systemd-to-s6 protocol conversion, mitigates the problem by only using autobind with abstract sockets, so any foreign would-be notifier would need to guess the abstract socket name.
(DIR) Post #B57eKPtJoDCLlnK0Tg by hipsterelectron@circumstances.run
1 likes, 0 repeats
@ska the anonymity of pipes had not quite occurred to me as a permissions mechanism
(DIR) Post #B57eOSM0auh2ZAfRuC by navi@social.vlhl.dev
1 likes, 0 repeats
@ska @lanodan we don't auth it at all, and we open one socket per serviceone socket per-service means we don't need to use cgroups to locate the service like systemd does, but since we have cgroups support, we could add authentication against the cgroupexcept that would result on the same "race condition" wrt exiting, if the process exits and is reaped before the NOTIFY_SOCKET helper can auth-check it, then we can't auth-check itso to implement permission control on our NOTIFY_SOCKET helper we would also need to implement BARRIER, :blep:
(DIR) Post #B57h3GrWQPG2hrztdg by ska@social.treehouse.systems
1 likes, 0 repeats
@navi @lanodan If you have one socket per service with an unpredictable name, I don't think you need to authenticate at all. Only the right service knows the socket name, if the name is in a large enough space, you're good; it's all collaborative, if someone is trying to be malicious you have already lost. (You could verify via SO_PEERCRED that the client is root, though, if you want to be really safe.)s6-notify-socket-from-fd uses an abstract socket with autobind. The autobind address space is 220; that should be enough for the kernel to make the socket address unpredictable, and if not, 🤷
(DIR) Post #B57kVkAPMeD3Z26EfA by navi@social.vlhl.dev
0 likes, 0 repeats
@ska @lanodan the name is predictable, but i can change that alright
(DIR) Post #B57kVkOaVvYIH0jYlU by fiore@brain.worm.pink
0 likes, 0 repeats
@navi @lanodan @ska uuid as filenames
(DIR) Post #B57kbxwx24ne34lwHI by lanodan@queer.hacktivis.me
1 likes, 0 repeats
@fiore @ska @navi Well… mktemp is pretty much that
(DIR) Post #B58Fokbf3F7V0aOFnc by ada@zoner.work
0 likes, 0 repeats
@ska@social.treehouse.systems @navi@social.vlhl.dev @lanodan@queer.hacktivis.me i low key disagree. just because if someone is trying to be malicious shouldn't automatically default to "you already lost." giving up acls in this case is just the security by obscurity fallacy, it's the entire reason why you should have it.the only thing that makes me coincide is that openrc sockets are extremely small and narrow surface so implementing it would likely just introduce even worse situations via bugs due to complexity
(DIR) Post #B58FolSTsrN3eP6R84 by ska@social.treehouse.systems
1 likes, 0 repeats
@ada @navi @lanodan someone is on your local machine trying to interfere with the service manager. Whether or not they succeed, you have a problem. It is good to make such an attack difficult, but I don't think it's a good use of resources or dev time to focus on hardening against that. A large space for the socket name is enough.
(DIR) Post #B58FxXWB5DYz1P14Pw by ska@social.treehouse.systems
0 likes, 0 repeats
@lanodan @fiore @navi mktemp is "locally unique", which is enough for this purpose
(DIR) Post #B58FxXudcH84FGSbXk by lanodan@queer.hacktivis.me
0 likes, 0 repeats
@ska @fiore @navi Yeah, plus can be used to create a chmod 0700 directory for further stuff
(DIR) Post #B58Ixz1j6bxGCXfysq by lispi314@udongein.xyz
0 likes, 0 repeats
@ska > What would be a simpler way to handle all this? Have the supervisor run the unit with a pipe from the daemon to the supervisor, not a socket anyone can connect to. Processes that can write to the pipe are the ones that can inherit it, which means they're necessarily part of the same unit.POSIX does guarantee a whole 7 other file descriptors available for such purposes, after all, why not use them?
(DIR) Post #B58IxzURNqvJdb6udk by navi@social.vlhl.dev
0 likes, 0 repeats
@lispi314 @ska fwiw _POSIX_OPEN_MAX (which is the minimum value for the max fds) is 20but honestly i doubt any system now a days has less than at least 1024? if not unlimited (w/o accounting for ulimit)
(DIR) Post #B58IxzhCcP8EHB56X2 by ska@social.treehouse.systems
0 likes, 0 repeats
@navi @lispi314 it's Linux-only, so POSIX limits are extremely conservative. The default is 1024, yes, but that can be changed by the admin, who knows. Would systemd even run with a 20 fd limit? 😁
(DIR) Post #B58IxzqQ48VKjlOStk by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@ska @navi @lispi314 I’ve had fun with very low fd limits on some systems I ported mksh to…
(DIR) Post #B58J307qRqdj3Zvl68 by lanodan@queer.hacktivis.me
0 likes, 0 repeats
@navi @ska @lispi314 Made me check and OPEN_MAX is 256 on Solaris 10 and 11, didn't except that but well you learn everyday.Then UNIX® Conformance Statements wise, seems to be 60000 for HP-UX, 65534 for AIX, RLIM_INFINITY (huh?) for macOS Tahoe (sadly work's macOS testlab machine is currently down and haven't got an AIX or an HP-UX).
(DIR) Post #B58JGpBLvrezAdAA2y by gabrielesvelto@mas.to
0 likes, 0 repeats
@hipsterelectron @ska they are extremely useful. I use the same mechanism within Firefox crash reporting machinery because they guarantee that processes at both ends are what they claim to be, and both ends can immediately see if the other end went away without having to deal with PIDs, waitpid() or anything else for the matter, so they work even across unrelated processes.
(DIR) Post #B58JGpOT969TpJIdUW by ska@social.treehouse.systems
1 likes, 0 repeats
@gabrielesvelto @hipsterelectron pipes as a death detection mechanism in unrelated processes are really underestimated and underused, probably because "unrelated" processes still need a common ancestor to create the pipe. But it's a really useful pattern, and that's only one of the many uses of pipes.
(DIR) Post #B58LHSXrmTuvA47kPo by whynothugo@fosstodon.org
0 likes, 0 repeats
@hipsterelectron @ska Passing around pipes like this (or file descriptors in general) is very very close to how capabilities work in capability-based systems. They are quite pleasant to use.
(DIR) Post #B58LHSiV8wQLh36EzY by hipsterelectron@circumstances.run
0 likes, 0 repeats
@whynothugo @ska it happens to have a very curious analogy to a theory of delegated identities i have been working on for anonymous cryptography
(DIR) Post #B58LHSqIfwf85EkT9E by whynothugo@fosstodon.org
0 likes, 0 repeats
@hipsterelectron @ska I can imagine why you say this, but I'm still curious to hear more.
(DIR) Post #B58LHT0w2PAYcDixiy by hipsterelectron@circumstances.run
0 likes, 0 repeats
@whynothugo @ska i am too! i've been trying to write it out. it originated from the desire to expand signal's sealed sender to cover recipient protection (this is impossible in signal's model due to the need for message routing in plaintext). there is one way to expand this in general through the mechanism of ratcheting https://eprint.iacr.org/2020/148 (awesome paper) but in general very little work has been done on anonymity so the application of identity separation is much less well developed
(DIR) Post #B58LHTElD0EDJ6C0H2 by hipsterelectron@circumstances.run
0 likes, 0 repeats
@whynothugo @ska fds have a useful analogy to keypairs in that the only way to achieve any form of atomicity in the VFS layer is to open up an fd. but i also think every single filesystem is terrible and could fix that without necessarily invoking capabilities
(DIR) Post #B58LHTQoUBrxuTpd3o by ska@social.treehouse.systems
0 likes, 0 repeats
@hipsterelectron @whynothugo The set of filesystem primitives is indubitably the weakest part of the Unix API, it's not on the filesystems themselves - what we need is a well-designed transactional file API, and then various filesystem implementations will follow.That said, the concept of fd is really useful; when Unix people say "everything is a file" they really mean "everything is a file descriptor" and that applies cleanly to a surprising lot of stuff. An fd is a generic handle that has very nice properties (lifetime, configurable sharability, etc.) and can be piggybacked onto by a lot of systems that need the same properties, e.g. capabilities.
(DIR) Post #B58LHTfLc9UmdYdEiO by whynothugo@fosstodon.org
0 likes, 0 repeats
@ska @hipsterelectron One of the more notable missing APIs is one to atomically replace a file."Replace file at path with this other file, unless someone else the given fd no longer refers to it". Or something along these lines.
(DIR) Post #B58LHTnr6WIj3wc1ya by ska@social.treehouse.systems
1 likes, 0 repeats
@whynothugo @hipsterelectron Huh? rename() exists and it's the only reason why the API is even usable. It's the most powerful tool in the box.I have endless complaints about how it's not powerful enough because it cannot replace directories and I have to constantly use symlinks in order to emulate that; but for not-directories, it's definitely there and essential.
(DIR) Post #B59PtokcGNYi0farNg by mia@shrimptest.0x0.st
0 likes, 0 repeats
@ska why can’t linux implement authenticated IPC for fuck’s sake, this is such a clownshow
(DIR) Post #B59Ptp1zDnSAsXijSK by ska@social.treehouse.systems
0 likes, 0 repeats
@mia It does. (getsockopt() with SO_PEERCRED, and you get the uid, gid and pid of the client.) But that's not useful here, because what systemd wants is a mechanism that's restricted to one unit; neither uid nor gid can be used for it, and pid isn't a good identifier since the notifier isn't necessarily the main process in the unit.There's an easy way to provide a communication channel to the unit only: the supervisor creates a pipe beforehand and passes the writing end to the unit. That's what s6 does. But of course, that would be way too simple for systemd, so here we are.
(DIR) Post #B59PtpV3TihoKhJwlU by resuna@ohai.social
0 likes, 0 repeats
@ska @mia Not generally a fan of s6 but it does get a lot of stuff right.
(DIR) Post #B59PtppyDxR5NZ6eMi by ska@social.treehouse.systems
0 likes, 0 repeats
@resuna @mia If you have actionable criticism of s6, I'm always open to hearing it.(But if it's about the interface, I know! and I'm working on it. 😅)
(DIR) Post #B59Ptq5DJHd48qEp7o by resuna@ohai.social
0 likes, 0 repeats
@ska @miaThe scripting language 'execline' is really strange, it's not as strange as the one that nix came up with, but it's pretty strange. I just find it existentially unpleasant to work in.Okay the standard shell is kind of a security nightmare, but couldn't it have used for example a lisp derivative which would be comparatively sane.Or maybe Postscript? Smalltalk?
(DIR) Post #B59PtqHcZ9YOlK2jSq by ska@social.treehouse.systems
0 likes, 0 repeats
@resuna @mia I can't believe I'm still hearing this in 2026, to be honest. Will that misperception ever die?You do not need to write your run scripts in execline. You never have, and you never will. You can write your run scripts in any scripting language you want. You can even make your run "scripts" compiled executables, if you like.Most people use /bin/sh as the interpreter for their run scripts, and it's totally fine.s6 (more accurately, s6-rc) uses execline internally because execline is better than sh for automated script generation, but you never have to see it. You can use s6 and pretend execline doesn't exist. You can even build s6 without execline, it's not recommended, but it's possible. I don't know what more I can do at this point.
(DIR) Post #B59PtqUjmO2tQ0BCuO by resuna@ohai.social
0 likes, 0 repeats
@ska @mia I am supporting a system that was built by other people, so I don't have the luxury of changing the scripting language it uses, and it uses execline.And right now I just Googled it and found the documentation on S6 and it goes on about how execline is a better scripting language than the shell and how horrible the source code to the Shell is and yes I agree Bournegol is insane.Which doesn't exactly read like execline is an internal tool and you don't have to actually touch it.
(DIR) Post #B59Ptqg56D7TzBUGae by ska@social.treehouse.systems
0 likes, 0 repeats
@resuna @mia The documentation is old for the most part and really deserves a full pass, so maybe it's not clear enough that execline is completely optional, but it has always been the case.
(DIR) Post #B59PtqoaaZvQPZT3qq by resuna@ohai.social
0 likes, 0 repeats
@ska @mia It reads more like "Technically you can build without it but this is what we recommend and man it's so much more secure than the shell look how cool it is."
(DIR) Post #B59Ptqwk6GRmorHZYm by mia@shrimptest.0x0.st
1 likes, 0 repeats
@resuna @ska good technical documentation writers are in VERY short supply
(DIR) Post #B5ARWpzrrIS0HQrDOa by ska@social.treehouse.systems
0 likes, 0 repeats
@Kishi Of course OpenRC is better. So is s6. So is literally anything else unless you need very specific features that are only supported by systemd.
(DIR) Post #B5ARWqFotzD94uJxGC by navi@social.vlhl.dev
1 likes, 0 repeats
@ska @Kishi as we keep implementing the few good ideas but with an actual good design, the number of those features only slim downcase 'n point my design for reexport in openrc vs doing `systemctl import-environment` from inside unit files and polluting the env for all services