▲ 128 points
back
67 comments
> POSIX allows file descriptors to be shared by an arbitrary number of processes, after e.g. forks or SCM_RIGHTS transfers (even though this use case is most likely very rare, so it’s not entirely impossible for this state to be moved to userspace).
Normally it’s rare, except shell functionality crucially depends on it[1].
The way to think about this, IMO, is that while a disk file is more or less random-access, when you open it unavoidably turns into strange hybrid of a random-access byte array and a (seekable) byte stream (classically, the “file description”), and that is what the resulting fd and any of its copies point at. Seekable streams made some sense in the age of tape drives, but nowadays it seems to me that random-access files and non-seekable streams have enough of a gulf between them that forcing both to appear like the same kind of object is more confusing than helpful.
(I don’t know what my ideal world would look like, given I still like to direct streams from/to files. I still dislike the “file description” business rather strongly.)
[1] https://utcc.utoronto.ca/~cks/space/blog/unix/DupSharedOffse...
Ah! Well as a systems programmer, it's not so rare. FDs are really not a friendly interface, and I'd love a better interface (FDs have their quirks), but as an application programmer, it's really sad we don't take advantage of FD passing to build more modular programs. For instance on desktop operating systems, you pass paths, not capabilities to use files, and the norm is giving programs access to read large swaths of user data. This is not secure, and it's one of the issues well-passed FDs could help solve. In general, despite the challenges in making them secure, I'd strongly advocate for more modular and integrated applications.
As for the seeking abstraction, it fits well with other buffered device driver information streams. Yes, it's a complicated and confusing interface, but the key thing is it allows you to share an OS/system/hardware level resource between multiple programs. We want to take advantage of that. It's the abstraction we've got sure, but it's what we do with it that counts!
My idea of operating system design is you do pass capabilities; actually, a message can only pass capabilities and byte sequences, and all I/O must use that interface. Furthermore, "proxy capabilities" are possible; i.e. a program can make up its own capabilities and send them to other capabilities it has access to, and then use those capabilities that it had made up to receive messages and do something with them (such as forward them to other capabilities; this is a simple case that can be used for logging or for revocable capabilities, but more complicated uses are possible). Actually, my operating system design does not have file paths.
This makes it more secure than how UNIX is doing it (if it is designed properly; the way to do this is to avoid making the interface too complicated, since a complicated interface would also increase the complexity of the more advanced uses of proxying), as well as more flexible (and allows modularity). It is possible to then allow to read only a part of a file, or to decompress (or compress) automatically without the application program knowing about the compression, or to log accesses, etc.
However, for seeking it will require a message containing a command to request to seek the file, since "seeking" is not a function known to the operating system, and there is no system call for "seeking"; the system calls are sending and receiving messages, waiting for objects, discarding capabilities, and creating new proxy capabilities.
But, the seeking command and others would be standardized and defined in the operating system specification, so that programs can use them, even though the system call interface does not directly have such a command, and proxies can handle messages in whatever way they want to (since they are just arbitrary byte sequences and/or capability passing).
(My own operating system design also allows "userspaceification of POSIX"; the kernel is not POSIX, but a compatibility layer (for at least much of POSIX, but maybe not all of it necessarily) can be made in user space if it is desired.)
Generally this aligns with what the Redox kernel is currently transitioning into, with a few limitations in order to retain compatibility for the (quite larger number) of applications we "need" to support.
> My idea of operating system design is you do pass capabilities; actually, a message can only pass capabilities and byte sequences, and all I/O must use that interface.
File descriptors and capabilities are very similar, and Redox already uses file descriptors, which are handled by scheme daemons, for most interfaces in general. I'm working on a _virtual memory-based capability_ RFC, on top of which the POSIX file table can be implemented in redox-rt (userspace) without forcing (some) POSIX semantics onto capabilities.
> Actually, my operating system design does not have file paths.
We're eventually going to switch fully to the openat* family of path calls, which would generalize the open syscall into a scheme call that sends a capability reference (dirfd), a path, and returns a capability.
> However, for seeking it will require a message containing a command to request to seek the file, ...
But that the cursor to be stored either by the client (requiring extensive messaging for each non-absolute IO call), by the server (requiring unnecessary state since almost all positioned IO nowadays is random-access), or currently, in the kernel. I'd like this state to be stored in userspace, as the article mentions, but this will first require assessing whether it would be feasible to break compatibility for this, or at least "performant compatibility".
> the kernel is not POSIX, but a compatibility layer (for at least much of POSIX, but maybe not all of it necessarily) can be made in user space if it is desired.
This is exactly what Redox is transitioning to: a kernel that's not necessarily Unix-like, with the bulk of POSIX logic implemented in userspace (redox-rt).
You've roughly described how fuchsia[1] is designed.
[1]: http://fuchsia.dev
> As for the seeking abstraction, it fits well with other buffered device driver information streams. Yes, it's a complicated and confusing interface, but the key thing is it allows you to share an OS/system/hardware level resource between multiple programs.
As someone who has only dabbled in OS-level programming but recently had a use case that the seek interface seemed to work well for (parsing a file format that heavily used offsets to reference other parts of the data), I'm super curious about what you think the "complicated and confusing" parts of the interface are. (To be clear, I'm not doubting you; I'm asking because I suspect that my understanding might be more surface-level than I thought and there are probably some pitfalls that I might not be aware of!) Offhand, the only parts that seemed potentially confusing to me are the mix of signed and unsigned integers depending on the offset type (not sure if this was specific to the Rust implementation, but it used signed integers for relative offsets and unsigned for absolute offsets, which makes sense but maybe isn't something people would expect) and the fact that it's valid to seek past the end of a file (which I didn't need for my use case), but are there other subtleties that I didn't think of?
Not the OP but the complicated part to me is just that the fd has a global cursor which makes concurrent access require synchronization. The rust std::fs::File API at least makes this clear through mutability requirements but I imagine in other languages this either can cause a lot of bugs or requires a more complicated API to surface the functionality safely.
I have nothing against FD passing, and indeed agree it’s unfortunate we don’t do more of it. An (almost-)everything-is-a-string (shell) language with object-capability powers is still something I’d like to figure out someday. The Lisp-2-ish way Tcl object systems approach this feels interesting, but still a bit off from a real solution.
My reading of TFA was that it’s rare for it to be important that descriptors sharing a description thus also share a file position. And shell-like redirection use cases really are the only case I can think of where that’s important.
> As for the seeking abstraction, it fits well with other buffered device driver information streams.
I don’t think I understand what you’re getting at here. My point was that having some objects (fds, whatever) support {fstat, read/write} only and others {fstat, pread/pwrite, mmap} only would get rid of the confusing user-visible notion of “file description”. Obviously I don’t expect this to happen, but it’s still nice to dream.
> My reading of TFA was that it’s rare for it to be important that descriptors sharing a description thus also share a file position.
It reads like this was an assumption rather than an observation.
> And shell-like redirection use cases really are the only case I can think of where that’s important.
.xsession-errors , or really anything of that nature where a process tree shares an error log file on stderr.
I think "(almost-)everything-is-a-string (shell) language" is not the way to do it; the command shell can be designed in a better way. But, my intention of design of the command shell programming language of a operating system is that it would have object-capability powers, too (and will be called "Command, Automation, and Query Language").
> but as an application programmer, it's really sad we don't take advantage of FD passing to build more modular programs.
For that programming languages need better unix socket (with SCM_RIGHTS) and directory-handle (openat & co) support. And of course windows does things differently so getting a portable abstraction would be difficult.
On Darwin, you can wrap a file descriptor wrapped in a Mach port right. That is leveraged pretty heavily on Apple's platforms for exactly these reasons.
On desktop Unix, D-Bus provides a friendlier interface to send file descriptors than sendmsg.
Going straight to D-Bus feels excessive. Here's an 85-line file I had lying around that should cover most cases of FD passing: https://paste.rs/6FBFS.c.
APIs are/were designed for completeness more than friendliness. Speaking of sendmsg, the whole BSD socket API is plain horrible; it only takes a couple of uses to realize that you never want to use it directly again; you either make your own library on top of it, or a class, or whatever form of code reuse the language deems appropriate.
Only because it wraps sendmsg in a nicer API at the language level - something you could also do with raw sendmsg.
I'm aware of this example, this was discussed earlier on HN [1]. I think it would be reasonable to (try to) enforce O_APPEND in such scenarios, in which case libc might internally open a pipe (configured to passively or actively be flushed to the underlying file, non-concurrently). A pipe would also be more reliable (and secure, though programs in shell pipelines are usually trusted), since it's no longer possible for isolated programs to change the global cursor, which could overwrite other processes' written data.
[1] https://news.ycombinator.com/item?id=38009458
Also, I feel like you are misquoting me, by not including the whole sentence (judging from the subcomments). I implied shared fds with shared cursors are probably a rare use cases, not shared fds in general (as you explained later on). Shared fds, and especially the ability to send fds, are obviously very fundamental for Unix-like systems, and for moving towards capability-oriented security.
Yeah, cutting the sentence that way evidently did not work out, sorry. For posterity:
> This cursor unfortunately cannot be stored in userspace without complex coordination, since POSIX allows file descriptors to be shared by an arbitrary number of processes, after e.g. forks or SCM_RIGHTS transfers (even though this use case is most likely very rare, so it’s not entirely impossible for this state to be moved to userspace).
Linux has had plan9 namespaces for long time which you can use from userspace as well. With this you can give a process isolated view of the filesystem, network etc ..
Unfortunately not enabled on AWS lambda runner <https://github.com/aws/aws-lambda-base-images/issues/143>
absolutely right re. shells, there's a lot in core old unix that depends on dup and a lot from the 90s/00s that depends on dup2 as well, with all the pain in the butt that it comes with (for multi-threaded and userspace implementors).
readat and friends are certainly becoming more popular in various domains, particularly as they're cheaper in highly parallel programs avoiding the locking on seeks, and avoiding the stalls on mappings (unless you can afford gigantic / up-front & static mappings). At this point what we really need is thread independent mapping solutions, a solution to request a mapping be bound to a thread, and the ability to also pass that around. Mapping will never be free, but if we can avoid whole process stalls and giant mapping pressure both of which are painfully common today that's a big step forward.
How would that work in practice? Each thread has their own memory map, but with one shared branch where all the normal process-global virtual memory is mapped, and a non-shared branch for per-thread virtual memory?
Seems like there's the potential for surprise, accidental overlapping of non-shared VM mappings. But I guess this is purely a performance thing; caveat emptor, etc....
How easy would that be to hack into Linux's existing page table structures?
That is not how MMUs work, so it would go very poorly.
MMUs expect a tree. You can not give them “Tree A, except at Node N substitute a different sub-tree”. To have a different sub-tree, you must have a different root. However, you can share sub-trees amongst different trees.
To do what you are expecting, you would actually want multiple processes with their own mappings/tree and you would share a substantial common/shared mapping/sub-tree. The difficulty there is making sure all the processes are aware of the common sub-tree and properly share and synchronize with it.
I don't know that I see the practical difference here: a seekable stream and a random access byte array are one very thin abstraction away from each other, with the important caveat that a stream can represent a byte array which is being appended to while you're using it.
As long as only one thread of execution is using it, yes, there’s essentially no difference.
When it’s shared between several, though, seekable streams become annoying (and difficult to impossible to use correctly). When the sharing spans process boundaries, we get the well-known bane of microkernel Unices that TFA describes—you need every file position for every Unix process to be (at least potentially) tracked by a single systemwide “Unix server”. That’s a lot of pain for not a lot of win.
My point was that I can’t think of any non-niche thing that’s naturally a seekable stream—they’re usually either nonseekable or outright random-access. Tape was the natural example when the API was invented, but it is niche now.
But if the stream represents a file that can receive writes, then one way or another you need to keep track of the order in which writes and reads happen - i.e. you need locks - which means you need global state.
If the access to a file was truly read only, then trivially every reader can just have its own seekable FD tracking position locally (or permissions to access the underlying object and open new ones).
That's true in serial computations, but not true in concurrent ones - the abstraction now includes mutual exclusion in the stream case, and does not require it in the array case.
When you write “file description”, do you mean “file descriptor”?
Crucially, no.
“File descriptions” (or “open file descriptions”; classically, struct file[1]) are what file descriptors designate. The whole hubbub is because a file description is not just a device number, an inode number, and a set of already-checked permissions; they are all that and a position in the file, and that position gets shared along with the rest of the description when you dup(), fork(), or sendmsg() the file descriptor—but not if you open() the same file again, not even[2] via /proc/self/fd.
As I’ve said, I think the most helpful way to think about this is that a file description is a seekable stream object on top of the underlying random-access disk file, and a reference to that stream object is what you’re passing around when you’re handling (hah) file descriptors.
[1] https://www.tuhs.org/cgi-bin/utree.pl?file=V7/usr/sys/sys/fi...
File descriptors and file descriptions are not the same thing. Descriptors are references to descriptions, along with some metadata, and multiple descriptors can point to the same description.
> multiple descriptors can point to the same descriptor.
You mean to the same description?
And is this the same as what is referred to as “open file description” in the open(2) man page, or are there also other “file descriptions”?
Passing around fd's via unix sockets is important for sandboxing in e.g. Chrome and Firefox.
Those who understand Unix are condemned to reproduce it exactly.
Which is why the next OS might be written by younglings who know little to nothing about "modern" OSes and just design from first principles, instead of being beholden to a gigantic historical architecture debt.
We're still running on kernels written in 1990. Time for an upgrade?
We still have banks running code even older than that. Do you want banks to take a large risk and potentially impact millions of people because "old code is bad code"? There is a time for an upgrade where things make sense. Not everything needs to be re-developed. We discovered fire a very long time ago. I don't think we need to retire fire because its so ancient.
I wholly believe it's impossible to get out of our "local minimum" without re developing from zero at some point.
So many of our technologies are just there because they solve some problem we created ourselves.
Communicating using a protocol from the 90ies, using a network stack from the 80ies, running an CPU architecture from the 70ies.
Being old and being unsuitable for its purpose are orthogonal.
I disagree. While some amount of stability is a precursor to usability, it does hold us back from progress. I posit we hit a [complexity] ceiling and we need to do a clean rewrite. Maybe hardware guys in the back can't, but we software people surely can.
Why emulate POSIX signal semantics? Those are a holdover from the single-thread UNIX era. Events should be delivered by threads or async calls. Signals should be reserved for exceptions, things you don't return to, like segfaults.
The rationale was discussed in-depth earlier on LWN (https://lwn.net/Articles/982186/).
Signals are useful as a software analogue to hardware interrupts, even though they are probably too low-level for most application use cases. They are for example used by the Golang runtime to preempt threads AFAIU, which wouldn't otherwise be possible non-cooperatively. Intel even included an ISA extension called user-IPIs, which is similar to signals.
Would shared pipes not have approximately the same issue wrt file positions being shared state? A process group (cron job, systemd service) sharing a pipe for error output is probably a bit more common than directly having an on-disk file.
Pipes lack a file cursor, but they still need to store the fcntl file description flags, so yes. This is emulated by the Redox kernel for old schemes, by inserting fcntl calls between the legacy read/write calls that don't allow passing the offset and flags. However, I'd argue that the fcntl flags are even more client-oriented than the cursor, say, O_NONBLOCK. Should it really be possible for isolated processes to set each others' nonblock flag (unless using preadv2/pwritev2 which is non-POSIX)?
This is very typical rust project. Trying to change the old world while tightly following the old world in a very unimaginative way.
Maybe too much memory safety is bad for creativity?
It's the way of the world. Once memory safety has properly infiltrated most of the world, we can move on to (re-)discover other kinds safety.
maybe too little memory safety means your wit overflows back to 0?
Rust overflows in the same way (granted it does provide convenient built-in wrappers to check for overflows).
I love how "secure" just translates to "memory safety" now. Apparently OpenBSD could have just been a basic Rust kernel and stopped thinking about all the other security mechanisms they have
quality post from `throwaway984393`.