[HN Gopher] NFS > FUSE: Why We Built Our Own NFS Server in Rust
___________________________________________________________________
NFS > FUSE: Why We Built Our Own NFS Server in Rust
Author : ylow
Score : 90 points
Date : 2023-09-19 17:58 UTC (5 hours ago)
(HTM) web link (about.xethub.com)
(TXT) w3m dump (about.xethub.com)
| MayeulC wrote:
| Anyway, interesting idea. As another comment pointed out, I wish
| there was an easy way to mount NFS and 9p shares as a user.
|
| I've been toying with 9p+Wireguard lately, as an alternative to
| SMB/Webdav/sftp/etc for sharing files with friends over the
| Internet. Wireguard+NFS works well enough, but it's relatively
| hard to integrate in userspace. Toying with rust-9p[1], I
| obtained acceptable performance. I still have to experiment with
| putting Wireguard in userspace as well (probably with Tailscale's
| implementation), but I come back to the difficulty of mounting 9p
| and NFS shares as a user. I also wish it was simpler to export
| directories over both protocols, on an unprivileged port, with
| access rights limited by those of the user.
|
| Regarding the article content, I find 9p to be even simpler and
| probably more ubiquitous than NFS, though I don't know if MacOS
| and Windows natively support it.
|
| [1]: https://github.com/pfpacket/rust-9p
| chatmasta wrote:
| If you haven't seen it, Cloudflare also has an implementation
| of Wireguard in userspace, called Boringtun [0], written in
| Rust and successfully deployed to millions of iOS and Android
| devices running the 1.1.1.1 app.
|
| For reference, another Rust project that depends on Boringtun
| is Onetun [1], which uses it to encrypt packets sent over a
| virtual smoltcp interface. I imagine you could follow a similar
| approach to integrating rust-9p with Boringtun, and you
| wouldn't need to leave the Rust ecosystem (whereas you might
| face more obstacles integrating it with Tailscale's wgengine,
| which is written in Go).
|
| [0] https://github.com/cloudflare/boringtun
|
| [1] https://github.com/aramperes/onetun
| mato wrote:
| > Anyway, interesting idea. As another comment pointed out, I
| wish there was an easy way to mount NFS and 9p shares as a
| user.
|
| Add e.g. foo.bar.com:/path /net/foo.bar.com nfs to /etc/fstab
| with options=user ?
|
| > Regarding the article content, I find 9p to be even simpler
| and probably more ubiquitous than NFS, though I don't know if
| MacOS and Windows natively support it.
|
| Windows certainly does not. MacOS I don't know, but if you mean
| "natively" as "stock install without third-party software",
| then probably not.
| mato wrote:
| > Windows certainly does not.
|
| Actually, scratch that, comments below suggest that WSL2 uses
| 9P for file sharing with the host, so it's possible that a
| version of Windows with the WSL2 bits installed will support
| 9P.
| renewiltord wrote:
| This is clever, considering the constraints. We use a fork of
| catfs write now to have a read through cache over NFS, but we
| could, in theory use a similar flow as here to do it and this
| will be a good example. Nice. Part of the problem, though, is
| that we currently use extended attributes, and I think you need
| NFSv4 plus a modern Linux to get those across NFS.
| ylow wrote:
| Doing some quick googling, I see
| https://lwn.net/Articles/353831/ which sounds like an xattr
| protocol extension to nfsv3. I do not know the extent to which
| it is supported though...
| teddyh wrote:
| NFSv _3_? Wouldn't NFSv4 be more appropriate?
| LeoPanthera wrote:
| NFS4 is a huge ugly bloated mess of a protocol, and they keep
| adding _more_ stuff to it.
|
| NFS3 is small (I wouldn't say "elegant" but by comparison...),
| works, and doesn't have anything wrong with it.
| chungy wrote:
| NFSv4 has sane, good locking out of the box. Implementing
| lockd on an NFSv3 server is a big world of hurt.
| notacoward wrote:
| Then again, if you're using file locking on a distributed
| filesystem at non-trivial scale you're in for a big world
| of hurt anyway. Source: I was a developer and maintainer of
| exactly such a beast for over a decade.
| hedora wrote:
| Is there some way to configure NFSv4 not to silently
| corrupt data or surface transient network errors to
| userspace?
|
| (Correct NFSv3 implementations block forever, which is the
| best of three bad alternatives.)
|
| https://access.redhat.com/solutions/1179643
| chungy wrote:
| NFSv4 should never do that, and I've never seen it
| happen.
| ylow wrote:
| NFSv4 is quite a bit more complicated and is also stateful
| which makes it a bit more challenging to implement. NFSv3 is a
| good starting point, and is still supported on all the major
| platforms. Certainly, extending with a v4 implementation would
| be great future work.
| mprovost wrote:
| Version 3 will be supported until the end of time. Version 4
| was an evolutionary dead end that never took off or saw much
| adoption.
| chungy wrote:
| Version 4 has been implemented in and widely used in Linux,
| FreeBSD, Solaris, Microsoft Windows...
|
| I'm not sure what value of "much adoption" you're using,
| but I think it's had great adoption.
| mac-chaffee wrote:
| Funnily enough, "the end of time" for NFSv3 is in about
| 15-83 years, depending on whether the implementation uses
| signed or unsigned 32-bit integers for timestamps:
| https://lwn.net/Articles/717076/
| mprovost wrote:
| True, but I define end of time as a 32-bit integer number
| of seconds from 1970.
| EdSchouten wrote:
| Also keep in mind that NFSv4 is mostly adds necessary
| complexity. Not accidental complexity.
|
| For example, NFSv4 supports accessing files that have been
| unlinked, but still have one or more open file descriptors.
| NFSv3 does not. It needs to resort to a technique named
| 'silly renames'.
|
| http://linux-nfs.org/wiki/index.php/Server-side_silly_rename
| duped wrote:
| You will have to throw away almost all of your code for
| NFSv4, it's a completely different protocol. One of the
| advantages of 4.x is that you don't need the portmapper or
| mount protocols, you just need to know the hostname/port of
| the server.
| [deleted]
| peoplearepeople wrote:
| This is interesting, is it possible to mount NFS as a client in
| an unprivileged user namespace on linux?
| lukax wrote:
| On Gnome you can use `gio mount` otherwise afaik no.
| minroot wrote:
| "I love files"
|
| Who doesn't?
| barryrandall wrote:
| People waiting on a backup process to restore over a billion
| tiny files.
| fragmede wrote:
| Uncompressed DPX video files. For those, a SAN exporting a
| block device (iSCSI, AoE) is preferable.
| mato wrote:
| This is awesome. NFS is very much undervalued as a cross-platform
| interface for talking to a (not necessarily remote) filesystem.
|
| ~20 years ago I was using SFS (Self-certifying file system),
| which was a SSH-like Internet-usable TOFU filesystem, using NFS
| as the "backend" for talking to the host OS. The site has since
| disappeared, but is archived at
| https://web.archive.org/web/20080330201843/http://www.fs.net....
| notacoward wrote:
| Gluster did this about a decade ago, though not in Rust. Actually
| twice - home-grown NFSv3 and Ganesha-based NFSv4. At Facebook we
| used the NFSv3 version to run dozens of clusters serving tens or
| perhaps hundreds of thousands of machines. My own involvement
| with that piece was minimal (I was a maintainer of Gluster
| overall for a while) but it worked pretty well for those users
| who needed a real mountable filesystem with real files instead of
| vaguely file-like objects that let you down every time you try to
| step beyond whole-file read and write.
| ylow wrote:
| Nice! I did look at Ganesha when I was exploring other NFS
| server implementation I could perhaps borrow and simplify. It
| looked great, but pretty much all implementations are too
| "real-filesystem" for me to convert. I needed something which I
| can make a very simple virtual filesystem interface over, and
| so wrote my own.
| ylow wrote:
| (Love the Gluster project btw. Thanks for maintaining it!)
| notacoward wrote:
| Thanks! I'm no longer a maintainer, or even working in
| tech, but still glad to share my experiences if folks might
| find them useful. I'm Obdurodon most places, @hachyderm.io
| on Mastodon, if you want to hit me up any time.
| vamega wrote:
| NFS seems to be a good fit for their use cases, however I don't
| believe that NFS supports an `inotify` like interface which makes
| it less useful for usecases like virtual filesystems for
| developer environments.
|
| Maybe newer NFS revisions allow for the NFS server to notify
| clients of changes like this, which would be really open things
| up here.
| hathawsh wrote:
| Very good point. Is there anything out there that's like NFS
| but also supports inotify? By "like NFS" I mean a remote
| filesystem protocol (not a block-level protocol) that works
| across platforms and can support nearly all filesystem
| features.
| znpy wrote:
| Did i read correctly, nfs v3?
|
| IIRC the statelessness of nfs was largely considered a design
| mistake, and there usually are a large number of hacks to
| overcome the consequences.
|
| So much so that nfsv4 moved to a stateful protocol...
| ylow wrote:
| I don't know that its considered a "design mistake". It is not
| without flaws, but it does make it extremely simple to
| implement. Its a very acceptable point as a tradeoff between
| simplicity and performance. Will certainly love to implement
| NFS4, or even SMB3 at some point though.
| rajatarya wrote:
| Repo link: https://github.com/xetdata/nfsserve
| EdSchouten wrote:
| For people who are interested in doing something similar in Go,
| some time ago I implemented a generic VFS that can be exposed
| both via FUSE and NFSv4.
|
| It's part of Buildbarn, a distributed build cluster for Bazel,
| but it can also easily be used outside that context.
|
| Details: https://github.com/buildbarn/bb-
| adrs/blob/master/0009-nfsv4....
|
| My recommendation to the authors would be to use NFSv4 instead of
| NFSv3. No need to mess around with that separate MOUNT protocol.
| Its semantics are also a lot closer to POSIX.
| ylow wrote:
| Very nice. The mount protocol is pretty minimal, though
| different OS clients seem to use it differently. The NFSv3
| chattiness is definitely a thing, but since its all localhost
| its not "too bad". v4 will be nice and is something I will love
| to try to implement. Its quite a bit more complex though.
| ylow wrote:
| Another possibility I am interested in is SMB3. (SMB4 is
| rather nasty). Similarly, supported by most operating
| systems, and does not look too horrifying to implement (The
| protocol documentation is really really hard to read though).
| And has the advantage that it does not require Windows Pro
| edition.
| ComputerGuru wrote:
| Another option is plan9; Windows uses the p9 protocol to mount
| host/guest directories and drives for IO in WSL.
| Karrot_Kream wrote:
| Wow I didn't know that!
| https://superuser.com/questions/1749690/what-is-this-weird-p...
| that is really interesting.
| ylow wrote:
| Hi! Author here. Happy to answer any questions!
| ReactiveJelly wrote:
| > You simply need to implement the vfs::NFSFileSystem trait.
|
| Wait, what......
|
| .. So not only is it an easy-to-run Rust NFS server and client
| (I gave up on NFS some years ago because I couldn't figure out
| how to run Samba) but it's also meant to use as a _substitute_
| for FUSE in stacks like SSHFS or an app wanting to serve debug
| data as files?
|
| Awesome!
| ylow wrote:
| Exactly. You build the server-side (as a FUSE alternative),
| and just use the OS's NFS client to mount it.
| inferiorhuman wrote:
| Sweet! I've been monkeying around with writing an HFS
| interpreter (in rust ofc). Unfortunately 'fuser' is less
| than ideal on a Mac. This could be an interesting
| alternative.
| eqvinox wrote:
| I don't think this includes a client, in fact the post argues
| that not having to implement the client because it already
| exists in the kernel was a major benefit of the approach...
| lukax wrote:
| This library is amazing. Also very happy to see such a
| permissive license.
|
| Btw, why did you decide to use NFS instead of WebDAV? WebDAV
| should be easier to implement and is also supported on all
| platforms.
| ylow wrote:
| I have not looked at WebDAV closely. I was looking for
| something that is already supported in all the major
| operating systems without additional libraries. I don't
| believe WebDAV is commonly supported out of the box?
| eqvinox wrote:
| WebDAV on Linux works over _drumrull_ FUSE :D
| lukax wrote:
| WebDAV is supported by Windows, MacOS and Linux. On all
| platforms you can mount it without admin/sudo (`net use` on
| Windows, `mount_webdav` on macOS and `gio mount` on Linux).
| Here is official Go package[1] using which you can
| implement your own WebDAV server (you only need to
| implement your FS interface).
|
| [1] https://pkg.go.dev/golang.org/x/net/webdav
| ylow wrote:
| Interesting. I do not know much about webdav. Great to
| learn something new! Will take a look.
| eqvinox wrote:
| Have you run any POSIX compliance checks against it? In
| particular, how are you handling assignment of inode numbers
| (fileid in NFS speak) to "virtual" files and making sure those
| stay reasonably consistent? (I vaguely remember this being a
| bit of a tripwire)
|
| Also did you look at NFSv4 / what made you decide to go with
| NFSv3? My (superficial) impression was that NFSv4 did a lot of
| simplifying and throwing out legacy stuff (e.g. rpcbind), but I
| don't know too many details on this...
|
| [edit: NFSv4 answered on parent /
| https://news.ycombinator.com/item?id=37575304 ]
|
| FWIW I think NFSv4 would be a rewrite rather than an extension
| :D
| mprovost wrote:
| NFSv4 is a completely different protocol and basically
| doesn't share anything with v2/v3. While technically it's
| still implemented using Sun RPC, it only has two procedures:
| NULL and COMPOUND. Version 3 is actually an RPC protocol,
| with a different procedure for each filesystem operation
| (READ, WRITE, CREATE, ...). One of the genius moves that Sun
| made with NFS was to distribute the protocol files (rpcgen)
| which autogenerate stubs (in C) making it easy to implement
| servers and clients. Nothing like that exists for v4.
| EdSchouten wrote:
| Look at RFC 7531. XDR definitions for NFSv4 exists, and
| they can also be used to generate C bindings if you want.
| mprovost wrote:
| Sure, running it through rpcgen generates bindings for
| both RPCs, NULL and COMPOUND: /// /*
| /// * Remote file service routines /// */
| /// program NFS4_PROGRAM { /// version
| NFS_V4 { /// void ///
| NFSPROC4_NULL(void) = 0; /// ///
| COMPOUND4res ///
| NFSPROC4_COMPOUND(COMPOUND4args) = 1; ///
| /// } = 4; /// } = 100003;
| ylow wrote:
| I tested by uh... building a "mirrorfs" and compiling this
| project in it :-) (cargo does stress it pretty hard). I have
| not looked for POSIX compliance testers. Is there one you can
| point me to?
|
| NFSv4 did throw out a lot of the legacy stuff, but also added
| a bunch of other state information like locks and delegation
| which is quite a bit more annoying to implement. Technically
| cos this is meant for "localhost-mounting-only", delegation
| should be easy to build (since we are expecting only exactly
| 1 mount). But v3 had far fewer APIs to implement, and so is a
| good starting point.
| eqvinox wrote:
| There's https://github.com/google/file-system-stress-
| testing but in this case I believe it would mostly stress-
| test the NFS client rather than your code... and I have
| never tried to use it, no idea if it even does anything
| useful.
|
| I know from lore that there are compliance-test suites for
| the POSIX file system API, but I believe those are
| commercial products :(
| ianlevesque wrote:
| xfstests is very thorough and reusable beyond xfs.
| EdSchouten wrote:
| Note that there's absolutely no requirement to support more
| advanced features such as locking and delegations. Even if
| a client offers to accept a delegation, a server may just
| deny/ignore it as part of OPEN.
| zeotroph wrote:
| If I may ask, is this mirrorfs available somewhere? The
| demofs from nfsserve seems to create and server a pure in-
| memory fs tree if I read the sources correctly.
| [deleted]
| formerly_proven wrote:
| I guess these are used as a soft mount, considering that if you
| terminate the application or it crashes a hard mount would just
| hang indefinitely. That's probably fine for a read-only snapshot,
| most problems with soft mounts are around writes.
| loeg wrote:
| I wouldn't recommend "hard" mounts to anyone.
| formerly_proven wrote:
| Well you trade system lockups with sometimes getting IO
| errors under load, which one is better ... _shrug_
| andrewstuart wrote:
| So it's a local file system driver accessed via NFS? Is that
| right? I get that NFS is on one side, but what's on the other?
| ylow wrote:
| This NFS Server on one side, and the OS's builtin NFS client on
| the other.
| [deleted]
| lukax wrote:
| It appears as a network drive on all major operating systems
| and can be accessed as any other local filesystem (through
| kernel syscalls).
| iroddis wrote:
| FUSE is used to present a file system interface of something
| that isn't a file system. E.g. present an LDAP directory as a
| file system.
|
| Writing FUSE drivers is pretty frustrating, though, and not
| well supported by all OSs.
|
| My takeaway is that this project discards FUSE in favour of
| making an NFS server that can be easily extended to do the same
| thing: present non-file system systems as mountable nfs remote
| file systems.
|
| It's a neat idea, and does avoid any extra kernel drivers, etc.
| A safer abstraction at the cost of adding a network stack and
| NFS's sometimes wonky effects (especially with hard mounts).
| Snarwin wrote:
| Page is completely unreadable in Firefox for Android.
| DistractionRect wrote:
| No, it's that dark reader doesnt play nice with the background
| texture on that site. Perfectly readable with dark reader
| disabled.
___________________________________________________________________
(page generated 2023-09-19 23:02 UTC)