[HN Gopher] Towards userspaceification of POSIX - part I: signal...
___________________________________________________________________
Towards userspaceification of POSIX - part I: signal handling and
IO
Author : akyuu
Score : 114 points
Date : 2024-08-02 21:07 UTC (1 days ago)
(HTM) web link (www.redox-os.org)
(TXT) w3m dump (www.redox-os.org)
| tom_ wrote:
| Those who understand Unix are condemned to reproduce it exactly.
| Aerbil313 wrote:
| Which is why the next OS might be written by younglings who
| know little to nothing about "modern" OSes and just design from
| first principles, instead of being beholden to a gigantic
| historical architecture debt.
|
| We're still running on kernels written in 1990. Time for an
| upgrade?
| blueflow wrote:
| Communicating using a protocol from the 90ies, using a
| network stack from the 80ies, running an CPU architecture
| from the 70ies.
|
| Being old and being unsuitable for its purpose are
| orthogonal.
| exe34 wrote:
| we still use roads and wheels, which are positively
| antique.
| Aerbil313 wrote:
| I disagree. While some amount of stability is a precursor
| to usability, it does hold us back from progress. I posit
| we hit a [complexity] ceiling and we need to do a clean
| rewrite. Maybe hardware guys in the back can't, but we
| software people surely can.
| jbernsteiniv wrote:
| We still have banks running code even older than that. Do you
| want banks to take a large risk and potentially impact
| millions of people because "old code is bad code"? There is a
| time for an upgrade where things make sense. Not everything
| needs to be re-developed. We discovered fire a very long time
| ago. I don't think we need to retire fire because its so
| ancient.
| eddd-ddde wrote:
| I wholly believe it's impossible to get out of our "local
| minimum" without re developing from zero at some point.
|
| So many of our technologies are just there because they
| solve some problem we created ourselves.
| mananaysiempre wrote:
| > POSIX allows file descriptors to be shared by an arbitrary
| number of processes, after e.g. forks or SCM_RIGHTS transfers
| (even though this use case is most likely very rare, so it's not
| entirely impossible for this state to be moved to userspace).
|
| Normally it's rare, except shell functionality crucially depends
| on it[1].
|
| The way to think about this, IMO, is that while a disk file is
| more or less random-access, when you open it unavoidably turns
| into strange hybrid of a random-access byte array and a
| (seekable) byte stream (classically, the "file description"), and
| _that_ is what the resulting fd and any of its copies point at.
| Seekable streams made some sense in the age of tape drives, but
| nowadays it seems to me that random-access files and non-seekable
| streams have enough of a gulf between them that forcing both to
| appear like the same kind of object is more confusing than
| helpful.
|
| (I don't know what my ideal world would look like, given I still
| like to direct streams from/to files. I still dislike the "file
| description" business rather strongly.)
|
| [1]
| https://utcc.utoronto.ca/~cks/space/blog/unix/DupSharedOffse...
| raggi wrote:
| absolutely right re. shells, there's a lot in core old unix
| that depends on dup and a lot from the 90s/00s that depends on
| dup2 as well, with all the pain in the butt that it comes with
| (for multi-threaded and userspace implementors).
|
| readat and friends are certainly becoming more popular in
| various domains, particularly as they're cheaper in highly
| parallel programs avoiding the locking on seeks, and avoiding
| the stalls on mappings (unless you can afford gigantic / up-
| front & static mappings). At this point what we really need is
| thread independent mapping solutions, a solution to request a
| mapping be bound to a thread, and the ability to also pass that
| around. Mapping will never be free, but if we can avoid whole
| process stalls and giant mapping pressure both of which are
| painfully common today that's a big step forward.
| wahern wrote:
| How would that work in practice? Each thread has their own
| memory map, but with one shared branch where all the normal
| process-global virtual memory is mapped, and a non-shared
| branch for per-thread virtual memory?
|
| Seems like there's the potential for surprise, accidental
| overlapping of non-shared VM mappings. But I guess this is
| purely a performance thing; caveat emptor, etc....
|
| How easy would that be to hack into Linux's existing page
| table structures?
| Veserv wrote:
| That is not how MMUs work, so it would go very poorly.
|
| MMUs expect a tree. You can not give them "Tree A, except
| at Node N substitute a different sub-tree". To have a
| different sub-tree, you must have a different root.
| However, you can share sub-trees amongst different trees.
|
| To do what you are expecting, you would actually want
| multiple _processes_ with their own mappings /tree and you
| would share a substantial common/shared mapping/sub-tree.
| The difficulty there is making sure all the processes are
| aware of the common sub-tree and properly share and
| synchronize with it.
| jart wrote:
| Yeah per thread virtual memory is called a process.
| Within a process the memory manager has to maintain a
| single virtual address space. You could do it with a red
| black tree and a global lock. Or you could use
| potentially cleverer algorithms. Skip lists are friendly
| to scalable concurrency. You could also do a radix tree
| of page tables, similar to what x86-64 processors do
| internally, then have each item be its own atomic long,
| and use some kind of optimistic transactional locking
| strategy when allocating mappings that span multiple
| pages. Personally, I think the rbtree is easiest, since
| it's better to leave memory scalability to malloc(). The
| way I like to make malloc() scale is having an arena for
| each CPU and then indexing them using sched_getcpu().
| immibis wrote:
| The Linux kernel actually doesn't know the difference
| between a thread and a process. The clone syscall (which
| starts a new thread) takes a bitmask specifying what
| should be shared between the parent and child (e.g.
| address space, open files, thread group ID (what most
| people call a process ID))
| wahern wrote:
| > To have a different sub-tree, you must have a different
| root. However, you can share sub-trees amongst different
| trees.
|
| That's what I meant by each thread having their own map
| (i.e. root), and within each map they have a shared
| branch, a branch being a pointer to a subtree. In this
| case, what I had in mind was the shared branch being at
| the root; basically slicing the user land address space
| in half (or 1/radix size...), though, that's probably
| getting too detailed.
|
| Perhaps by "branch" you thought I meant a jump operation
| in code. That's certainly not how most MMUs work, which
| walk a data structure constructed by the OS. But _some_
| MMUs actually fault to OS provided code to resolve an
| address, in which case "branch" in the meaning of
| software code isn't necessarily wrong. (But to be clear,
| it's not what I had in mind.)
| nolist_policy wrote:
| Passing around fd's via unix sockets is important for
| sandboxing in e.g. Chrome and Firefox.
| seeknotfind wrote:
| Ah! Well as a systems programmer, it's not so rare. FDs are
| really not a friendly interface, and I'd love a better
| interface (FDs have their quirks), but as an application
| programmer, it's really sad we don't take advantage of FD
| passing to build more modular programs. For instance on desktop
| operating systems, you pass paths, not capabilities to use
| files, and the norm is giving programs access to read large
| swaths of user data. This is not secure, and it's one of the
| issues well-passed FDs could help solve. In general, despite
| the challenges in making them secure, I'd strongly advocate for
| more modular and integrated applications.
|
| As for the seeking abstraction, it fits well with other
| buffered device driver information streams. Yes, it's a
| complicated and confusing interface, but the key thing is it
| allows you to share an OS/system/hardware level resource
| between multiple programs. We want to take advantage of that.
| It's the abstraction we've got sure, but it's what we do with
| it that counts!
| mananaysiempre wrote:
| I have nothing against FD passing, and indeed agree it's
| unfortunate we don't do more of it. An (almost-)everything-
| is-a-string (shell) language with object-capability powers is
| still something I'd like to figure out someday. The
| Lisp-2-ish way Tcl object systems approach this feels
| interesting, but still a bit off from a real solution.
|
| My reading of TFA was that it's rare for it to be important
| that descriptors sharing a description thus also share a file
| position. And shell-like redirection use cases really are the
| only case I can think of where that's important.
|
| > As for the seeking abstraction, it fits well with other
| buffered device driver information streams.
|
| I don't think I understand what you're getting at here. My
| point was that having some objects (fds, whatever) support
| {fstat, read/write} only and others {fstat, pread/pwrite,
| mmap} only would get rid of the confusing user-visible notion
| of "file description". Obviously I don't expect this to
| happen, but it's still nice to dream.
| tbrownaw wrote:
| > _My reading of TFA was that it's rare for it to be
| important that descriptors sharing a description thus also
| share a file position._
|
| It reads like this was an assumption rather than an
| observation.
|
| > _And shell-like redirection use cases really are the only
| case I can think of where that's important._
|
| .xsession-errors , or really anything of that nature where
| a process tree shares an error log file on stderr.
| pcwalton wrote:
| On desktop Unix, D-Bus provides a friendlier interface to
| send file descriptors than sendmsg.
| mananaysiempre wrote:
| Going straight to D-Bus feels excessive. Here's an 85-line
| file I had lying around that should cover most cases of FD
| passing: https://paste.rs/6FBFS.c.
| folmar wrote:
| There are a few existing libraries like libancillary that
| would do this for you and provide some level of OS
| compatibility.
| astrobe_ wrote:
| APIs are/were designed for completeness more than
| friendliness. Speaking of _sendmsg_ , the whole BSD socket
| API is plain horrible; it only takes a couple of uses to
| realize that you never want to use it directly again; you
| either make your own library on top of it, or a class, or
| whatever form of code reuse the language deems appropriate.
| mananaysiempre wrote:
| > the whole BSD socket API is plain horrible; it only
| takes a couple of uses to realize that you never want to
| use it directly again
|
| That was my initial impression as well, but recently I've
| had to use it again and surprisingly did not find as bad
| as I remembered. Except, indeed, for the fd-passing
| experience, for which see my wrapper elsewhere in the
| thread (also other sideband stuff, but how often do you
| really need SCM_CREDENTIALS?).
|
| The syscall/kernel-ABI people seem to love it as well--I
| remember reading an article that praised it for remaining
| so stable over its lifetime. I think these are actually
| two sides of the same coin: BSD sockets essentially layer
| a second ABI on top of C function invocations. It's a tad
| more specific than generic ioctl-ish (selector, payload),
| but not that much, and the farther away you are from the
| happy path of send()/recv(), the closer it is to that
| (and the more extension capability the kernel programmer
| wants, and the more misery the userland programmer
| feels).
|
| The Unix approach of exposing syscalls from libc
| essentially directly was a nice thought, but the sockets
| API feels like a reductio ad absurdum of it.
| immibis wrote:
| Only because it wraps sendmsg in a nicer API at the
| language level - something you could also do with raw
| sendmsg.
| saghm wrote:
| > As for the seeking abstraction, it fits well with other
| buffered device driver information streams. Yes, it's a
| complicated and confusing interface, but the key thing is it
| allows you to share an OS/system/hardware level resource
| between multiple programs.
|
| As someone who has only dabbled in OS-level programming but
| recently had a use case that the seek interface seemed to
| work well for (parsing a file format that heavily used
| offsets to reference other parts of the data), I'm super
| curious about what you think the "complicated and confusing"
| parts of the interface are. (To be clear, I'm not doubting
| you; I'm asking because I suspect that my understanding might
| be more surface-level than I thought and there are probably
| some pitfalls that I might not be aware of!) Offhand, the
| only parts that seemed potentially confusing to me are the
| mix of signed and unsigned integers depending on the offset
| type (not sure if this was specific to the Rust
| implementation, but it used signed integers for relative
| offsets and unsigned for absolute offsets, which makes sense
| but maybe isn't something people would expect) and the fact
| that it's valid to seek past the end of a file (which I
| didn't need for my use case), but are there other subtleties
| that I didn't think of?
| thinkharderdev wrote:
| Not the OP but the complicated part to me is just that the
| fd has a global cursor which makes concurrent access
| require synchronization. The rust std::fs::File API at
| least makes this clear through mutability requirements but
| I imagine in other languages this either can cause a lot of
| bugs or requires a more complicated API to surface the
| functionality safely.
| 4lDO2 wrote:
| Rust does however implement the IO traits for `&File` as
| well (shared), and IIRC also implements `try_clone` which
| is the dup equivalent.
| the8472 wrote:
| > but as an application programmer, it's really sad we don't
| take advantage of FD passing to build more modular programs.
|
| For that programming languages need better unix socket (with
| SCM_RIGHTS) and directory-handle (openat & co) support. And
| of course windows does things differently so getting a
| portable abstraction would be difficult.
| EPWN3D wrote:
| On Darwin, you can wrap a file descriptor wrapped in a Mach
| port right. That is leveraged pretty heavily on Apple's
| platforms for exactly these reasons.
| zzo38computer wrote:
| My idea of operating system design is you do pass
| capabilities; actually, a message can only pass capabilities
| and byte sequences, and all I/O must use that interface.
| Furthermore, "proxy capabilities" are possible; i.e. a
| program can make up its own capabilities and send them to
| other capabilities it has access to, and then use those
| capabilities that it had made up to receive messages and do
| something with them (such as forward them to other
| capabilities; this is a simple case that can be used for
| logging or for revocable capabilities, but more complicated
| uses are possible). Actually, my operating system design does
| not have file paths.
|
| This makes it more secure than how UNIX is doing it (if it is
| designed properly; the way to do this is to avoid making the
| interface too complicated, since a complicated interface
| would also increase the complexity of the more advanced uses
| of proxying), as well as more flexible (and allows
| modularity). It is possible to then allow to read only a part
| of a file, or to decompress (or compress) automatically
| without the application program knowing about the
| compression, or to log accesses, etc.
|
| However, for seeking it will require a message containing a
| command to request to seek the file, since "seeking" is not a
| function known to the operating system, and there is no
| system call for "seeking"; the system calls are sending and
| receiving messages, waiting for objects, discarding
| capabilities, and creating new proxy capabilities.
|
| But, the seeking command and others would be standardized and
| defined in the operating system specification, so that
| programs can use them, even though the system call interface
| does not directly have such a command, and proxies can handle
| messages in whatever way they want to (since they are just
| arbitrary byte sequences and/or capability passing).
|
| (My own operating system design also allows
| "userspaceification of POSIX"; the kernel is not POSIX, but a
| compatibility layer (for at least much of POSIX, but maybe
| not all of it necessarily) can be made in user space if it is
| desired.)
| 4lDO2 wrote:
| Generally this aligns with what the Redox kernel is
| currently transitioning into, with a few limitations in
| order to retain compatibility for the (quite larger number)
| of applications we "need" to support.
|
| > My idea of operating system design is you do pass
| capabilities; actually, a message can only pass
| capabilities and byte sequences, and all I/O must use that
| interface.
|
| File descriptors and capabilities are very similar, and
| Redox already uses file descriptors, which are handled by
| scheme daemons, for most interfaces in general. I'm working
| on a _virtual memory-based capability_ RFC, on top of which
| the POSIX file table can be implemented in redox-rt
| (userspace) without forcing (some) POSIX semantics onto
| capabilities.
|
| > Actually, my operating system design does not have file
| paths.
|
| We're eventually going to switch fully to the openat*
| family of path calls, which would generalize the open
| syscall into a scheme call that sends a capability
| reference (dirfd), a path, and returns a capability.
|
| > However, for seeking it will require a message containing
| a command to request to seek the file, ...
|
| But that the cursor to be stored either by the client
| (requiring extensive messaging for each non-absolute IO
| call), by the server (requiring unnecessary state since
| almost all positioned IO nowadays is random-access), or
| currently, in the kernel. I'd like this state to be stored
| in userspace, as the article mentions, but this will first
| require assessing whether it would be feasible to break
| compatibility for this, or at least "performant
| compatibility".
|
| > the kernel is not POSIX, but a compatibility layer (for
| at least much of POSIX, but maybe not all of it
| necessarily) can be made in user space if it is desired.
|
| This is exactly what Redox is transitioning to: a kernel
| that's not necessarily Unix-like, with the bulk of POSIX
| logic implemented in userspace (redox-rt).
| Cloudef wrote:
| Linux has had plan9 namespaces for long time which you can use
| from userspace as well. With this you can give a process
| isolated view of the filesystem, network etc ..
|
| Unfortunately not enabled on AWS lambda runner
| <https://github.com/aws/aws-lambda-base-images/issues/143>
| XorNot wrote:
| I don't know that I see the practical difference here: a
| seekable stream and a random access byte array are one very
| thin abstraction away from each other, with the important
| caveat that a stream can represent a byte array which is being
| appended to while you're using it.
| layla5alive wrote:
| That's true in serial computations, but not true in
| concurrent ones - the abstraction now includes mutual
| exclusion in the stream case, and does not require it in the
| array case.
| mananaysiempre wrote:
| As long as only one thread of execution is using it, yes,
| there's essentially no difference.
|
| When it's shared between several, though, seekable streams
| become annoying (and difficult to impossible to use
| correctly). When the sharing spans process boundaries, we get
| the well-known bane of microkernel Unices that TFA describes
| --you need every file position for every Unix process to be
| (at least potentially) tracked by a single systemwide "Unix
| server". That's a lot of pain for not a lot of win.
|
| My point was that I can't think of any non-niche thing that's
| naturally a _seekable_ stream--they're usually either
| nonseekable or outright random-access. Tape was the natural
| example when the API was invented, but it is niche now.
| XorNot wrote:
| But if the stream represents a file that can receive
| writes, then one way or another you need to keep track of
| the order in which writes and reads happen - i.e. you need
| locks - which means you need global state.
|
| If the access to a file was truly read only, then trivially
| every reader can just have its own seekable FD tracking
| position locally (or permissions to access the underlying
| object and open new ones).
| mananaysiempre wrote:
| From the point of view of the application, yes, you need
| to coordinate. But that's a concern for the application.
| From the point of view of the kernel / device server /
| however your microkernel OS works, requests for disk I/O
| arrive in some order, probably update some cache pages or
| whatnot, then go onto disk's queue. That's arguably
| global, but it feels like a logical extension of the fact
| that you only have a single physical disk. However you're
| going to multiplex it, something like that is still going
| to happen, and it has little to do with Unix.
|
| What is not a natural extension of the whole thing is
| that, even on an otherwise completely quiescent system,
| int fd1 = open("foo", O_RDWR), fd2 = dup(fd1);
|
| behaves _very_ differently from int fd1 =
| open("foo", O_RDWR), fd2 = open("foo", O_RDWR);
|
| (imagine these fds are then passed out to other processes
| or whatnot).
|
| Even with O_RDONLY, these are still not at all the same.
| Witness the epitome of CLI design that is OpenSSL /s :
| { openssl x509 -out
| "/etc/swanctl/x509/$1.pem" while openssl
| x509 -out "$t" 2>/dev/null; do
| fp=$(openssl x509 -in "$t" -noout -md5 -fingerprint |
| sed 's/.*=//; s/://g') mv -f --
| "$t" /etc/swanctl/x509ca/$fp.pem done
| } < "/etc/ssl/uacme/$1/cert.pem"
|
| This is how you pick apart a PEM cert bundle using
| OpenSSL: the shell spawns openssl, which reads a single
| PEM block from stdin, does its dirty deeds with it, and
| leaves stdin pointing to the next one, ready to be
| consumed by the next instance of itself that's yet to be
| spawned by the shell. You can't do that with a per-
| process file position, trivially or otherwise.
|
| Returning to the application side, imagine you're making
| a concurrent B-tree. If one thread wants to write out
| page X and the other page Y, they have presumably already
| used some locking to make sure that'll leave the data
| structure consistent, so they're free to just issue their
| pwrite()s and let them happen in whatever order. On the
| other hand, if all they have is write() and seek(), they
| have to hit a global lock even if X and Y are completely
| unrelated.
| 4lDO2 wrote:
| I'm aware of this example, this was discussed earlier on HN
| [1]. I think it would be reasonable to (try to) enforce
| O_APPEND in such scenarios, in which case libc might internally
| open a pipe (configured to passively or actively be flushed to
| the underlying file, non-concurrently). A pipe would also be
| more reliable (and secure, though programs in shell pipelines
| are usually trusted), since it's no longer possible for
| isolated programs to change the global cursor, which could
| overwrite other processes' written data.
|
| [1] https://news.ycombinator.com/item?id=38009458
|
| Also, I feel like you are misquoting me, by not including the
| whole sentence (judging from the subcomments). I implied shared
| fds _with shared cursors_ are probably a rare use cases, not
| shared fds in general (as you explained later on). Shared fds,
| and especially the ability to send fds, are obviously very
| fundamental for Unix-like systems, and for moving towards
| capability-oriented security.
| mananaysiempre wrote:
| Yeah, cutting the sentence that way evidently did not work
| out, sorry. For posterity:
|
| > This cursor unfortunately cannot be stored in userspace
| without complex coordination, since POSIX allows file
| descriptors to be shared by an arbitrary number of processes,
| after e.g. forks or SCM_RIGHTS transfers (even though this
| use case is most likely very rare, so it's not entirely
| impossible for this state to be moved to userspace).
| layer8 wrote:
| When you write "file description", do you mean "file
| descriptor"?
| 4lDO2 wrote:
| File descriptors and file descriptions are _not_ the same
| thing. Descriptors are references to descriptions, along with
| some metadata, and multiple descriptors can point to the same
| description.
| layer8 wrote:
| > multiple descriptors can point to the same descriptor.
|
| You mean to the same description?
|
| And is this the same as what is referred to as "open file
| description" in the open(2) man page, or are there also
| other "file descriptions"?
| 4lDO2 wrote:
| Yes, and yes.
| mananaysiempre wrote:
| Crucially, no.
|
| "File descriptions" (or "open file descriptions";
| classically, struct file[1]) are what file descriptors
| designate. The whole hubbub is because a file description is
| not just a device number, an inode number, and a set of
| already-checked permissions; they are all that _and_ a
| position in the file, and that position gets shared along
| with the rest of the description when you dup(), fork(), or
| sendmsg() the file descriptor--but not if you open() the same
| file again, not even[2] via /proc/self/fd.
|
| As I've said, I think the most helpful way to think about
| this is that a file description is a seekable stream object
| on top of the underlying random-access disk file, and a
| reference to that stream object is what you're passing around
| when you're handling (hah) file descriptors.
|
| [1] https://www.tuhs.org/cgi-
| bin/utree.pl?file=V7/usr/sys/sys/fi...
|
| [2] https://blog.gnoack.org/post/proc-fd-is-not-dup/
| tbrownaw wrote:
| Would shared pipes not have approximately the same issue wrt file
| positions being shared state? A process group (cron job, systemd
| service) sharing a pipe for error output is probably a bit more
| common than directly having an on-disk file.
| 4lDO2 wrote:
| Pipes lack a file cursor, but they still need to store the
| fcntl file description flags, so yes. This is emulated by the
| Redox kernel for old schemes, by inserting fcntl calls between
| the legacy read/write calls that don't allow passing the offset
| and flags. However, I'd argue that the fcntl flags are even
| more client-oriented than the cursor, say, O_NONBLOCK. Should
| it really be possible for isolated processes to set each
| others' nonblock flag (unless using preadv2/pwritev2 which is
| non-POSIX)?
| Animats wrote:
| Why emulate POSIX signal semantics? Those are a holdover from the
| single-thread UNIX era. Events should be delivered by threads or
| async calls. Signals should be reserved for exceptions, things
| you don't return to, like segfaults.
| 4lDO2 wrote:
| The rationale was discussed in-depth earlier on LWN
| (https://lwn.net/Articles/982186/).
|
| Signals are useful as a software analogue to hardware
| interrupts, even though they are probably too low-level for
| most application use cases. They are for example used by the
| Golang runtime to preempt threads AFAIU, which wouldn't
| otherwise be possible non-cooperatively. Intel even included an
| ISA extension called user-IPIs, which is similar to signals.
| up2isomorphism wrote:
| This is very typical rust project. Trying to change the old world
| while tightly following the old world in a very unimaginative
| way.
|
| Maybe too much memory safety is bad for creativity?
| throwaway984393 wrote:
| I love how "secure" just translates to "memory safety" now.
| Apparently OpenBSD could have just been a basic Rust kernel and
| stopped thinking about all the other security mechanisms they
| have
| techbrovanguard wrote:
| maybe too little memory safety means your wit overflows back to
| 0?
| tredre3 wrote:
| Rust overflows in the same way (granted it does provide
| convenient built-in wrappers to check for overflows).
| actionfromafar wrote:
| It's the way of the world. Once memory safety has properly
| infiltrated most of the world, we can move on to (re-)discover
| other kinds safety.
___________________________________________________________________
(page generated 2024-08-03 23:01 UTC)