[HN Gopher] Reading NFS at 25GB/s using FIO and libnfs
___________________________________________________________________
Reading NFS at 25GB/s using FIO and libnfs
Author : tarasglek
Score : 100 points
Date : 2021-11-30 14:23 UTC (8 hours ago)
(HTM) web link (taras.glek.net)
(TXT) w3m dump (taras.glek.net)
| ninkendo wrote:
| Any word on the iops numbers vs the in-kernel NFS client? The
| throughput is impressive but IME it ends up being the stat/fd
| activity of NFS clients that's the the limiting factor (try
| running `ls -l` in an NFS directory with lots of files in it,
| even worse if there are lots of symlinks involved.)
| diamondlovesyou wrote:
| Presumably, with async you could queue all the `stat`-ops after
| the dir walk at once, leaving the total latency in the ball
| park of "dir walk"+"ping for file stat"+"ping remaining symlink
| stats if present". But I don't think `ls` does this.
|
| Otherwise, yeah, you incur network latency on every file, plus,
| as you say, symlink "pings" if those are present. So its "dir
| walk"+"number of files"*"ping"+"symlinks"*"ping", which adds
| up.
|
| Batching high-latency ops is one of the only cases where I like
| async.
| mprovost wrote:
| NFS version 3 already has the READDIRPLUS operation which
| returns the directory contents and all of the stats together
| in one call to improve performance for this case. Sometimes
| the kernel client doesn't use READDIRPLUS and falls back to
| issuing a bunch of requests - usually because it's being
| super conservative about security (as if there is such a
| thing in NFS). It's also not that straightforward for a
| utility like ls to tell the kernel that it's reading both the
| directory and the stats for each file - they're separate
| system calls and the kernel has to figure out what the
| program is trying to do and optimise.
|
| I wrote a tool that issues raw READDIRPLUS requests to list a
| directory:
|
| https://rawgit.com/mprovost/NFStash/master/man/nfsls.8.html
| tarasglek wrote:
| Usually going via kernel nfs client will use up more memory
| bandwidth. I would expect lower per-client numbers. From what
| I've read you go from 3 memcopies on userspace to 4 with kernel
| nfs.
|
| I haven't yet instrumented memory bandwidth on my amd machines,
| but it feels like I'm at the limit.
| tarasglek wrote:
| In this case my job was to generate a synthetic workload to
| drive max bandwidth. So I didn't have to worry about metadata.
|
| Metadata is more complicated to model, need realistic directory
| structures etc.
|
| https://www.spec.org/sfs2014/ (2020 version) is a good test of
| a metadata-heavy workloads, but isn't open source like fio :(.
| mprovost wrote:
| The nice thing about NFS (v3) is that it's stateless, so you
| can just keep doing the same thing over and over. Messing
| around with metadata sucks but once you get the filehandle of
| the file that you want to read or write then you don't need
| it anymore.
|
| I wrote a suite of tools that does all this and dumps the NFS
| transactions as JSON, maybe it can be useful:
|
| https://github.com/mprovost/NFStash
| JoshTriplett wrote:
| I've tried to drive NFS to reasonable levels of performance in
| the past, and the bottleneck has never seemed like storage or
| network or the NFS server; it always seems like the combination
| of the built-in NFS client in the Linux kernel, the
| implementation of filesystem semantics, and the behavior of
| common workloads ends up making NFS _much_ slower than network
| throughput. I 'm impressed with these benchmarks and the approach
| to collecting them, but I'm wondering if the Linux kernel NFS
| client can get anywhere close to the theoretical limits.
|
| I've tried this with a _read-only_ NFS server, across an AWS
| multi-gigabit connection, and I still found that I couldn 't get
| anywhere near this level of performance for the workload of "make
| -j$(nproc)" in a Linux kernel tree. As a quick baseline, some
| numbers from the last time I tested this, with a c5.12xlarge (48
| CPU) client and server: a defconfig local build was 40s, and a
| defconfig build with a Linux kernel tree in read-only NFS (with a
| tmpfs overlay on top for writability) was 6m55s. That's a 10x
| slowdown. System stats during the build showed 4-5MBps net recv
| and net send, and 1-4MBps disk write.
|
| Is there some well-known method to getting reasonable performance
| out of off-the-shelf NFS servers and clients?
| geertj wrote:
| (PM-T for Amazon EFS, AWS's native NFSv4.1 file system)
|
| Performance turning NFS is difficult mostly because the
| information on how to do it isn't readily available. The two
| things that most people run into:
|
| - 'Close to open' cache consistency. In practical terms this
| means that open() is a round trip to the server to validate any
| data that might be cached already (unless you use delegations),
| write() goes into the page cache (as a writeback cache), and
| close() flushes all dirty data. Building a kernel tree reading
| and creating tons of small files, each of which requires two
| serial round trips over the network. Compare that to a local fs
| where neither open(O_CREAT), write() or close() actually go to
| disk and therefore run at memory speed (unless you use things
| like O_DIRECT or fsync/fdatasync()).
|
| - Per-TCP flow throughput limitations. On the AWS network the
| per-flow limit is 5 Gbit in general and 10 Gbit within a
| placement groups. To work around this, people use the
| 'nconnect' mount option. (which does not work currently with
| EFS). Local networks might have different limitations, but
| single TCP streams will typically always have some bw limit
| lower than the physical network bandwidth. I believe that this
| (very cool!) fio plugin works around this by using multiple
| connections.
|
| The actual data write latency of NFS servers isn't terribly
| different from local file systems.
|
| Today, the best way to get the most performance out of NFS is
| to either use large files and/or keep files open, or use high
| concurrency. By default, the 4.1 client will issue up to 64
| concurrent requests, which can be increased by increasing the
| 'max slots' NFS kernel module parameter. In your example of a
| kernel build, you could -j much higher than the number of CPUs
| because the compile jobs will be IO bound on reading input and
| writing output. This will amortize the round trips over more
| threads, and in theory (barring any other bottlenecks) reduce
| your build times.
| JoshTriplett wrote:
| Thank you _very_ much for the response!
|
| > - 'Close to open' cache consistency. In practical terms
| this means that open() is a round trip to the server to
| validate any data that might be cached already (unless you
| use delegations), write() goes into the page cache (as a
| writeback cache), and close() flushes all dirty data.
| Building a kernel tree reading and creating tons of small
| files, each of which requires two serial round trips over the
| network. Compare that to a local fs where neither
| open(O_CREAT), write() or close() actually go to disk and
| therefore run at memory speed (unless you use things like
| O_DIRECT or fsync/fdatasync()).
|
| That definitely sounds like a concern for writable NFS
| filesystems, but I was benchmarking reads to a read-only NFS
| mount.
|
| Related: Is there some option I can pass to make it clear
| that the data _on the server_ will never change and thus no
| possible write-to-read or close-to-open consistency issues
| can arise?
|
| > - Per-TCP flow throughput limitations. On the AWS network
| the per-flow limit is 5 Gbit in general and 10 Gbit within a
| placement groups. To work around this, people use the
| 'nconnect' mount option. (which does not work currently with
| EFS). Local networks might have different limitations, but
| single TCP streams will typically always have some bw limit
| lower than the physical network bandwidth. I believe that
| this (very cool!) fio plugin works around this by using
| multiple connections.
|
| Interesting! I've never seen the per-flow limit mentioned
| before. Is that documented somewhere?
|
| I'd be concerned about that if I were getting anywhere
| _close_ to that limit, but I was experiencing 4-5MBps network
| throughput. It seemed like individual file operations (like
| stat) were taking an excessive amount of time.
|
| > In your example of a kernel build, you could -j much higher
| than the number of CPUs because the compile jobs will be IO
| bound on reading input and writing output.
|
| I'm writing output to a local tmpfs (via overlayfs), not to
| NFS. And I'd love to tune the NFS setup to the point that
| reads (and stats) from NFS aren't causing a 10x slowdown.
| fefe23 wrote:
| My takeaway from this is: you think your company is great. No
| other information was transferred to me in this article.
|
| I learned nothing about FIO, libnfs, NFS, or your patch.
|
| No feature comparison with other efforts. No benchmark comparison
| with other efforts.
|
| I wasted my time here.
| guenthert wrote:
| Hmmh, there might be room for improvement, but I appreciate the
| article and the link here to it. It's been a while that I tried
| to get the most out of a NFS server and was unaware that it is
| still used for high performance applications. 25GB/s surely
| impressed me.
| stingraycharles wrote:
| As someone who recently spent two months trying to squeeze
| the highest perf out of a 160GBit NFS cluster, I'd say the
| article lacks a _lot_ of detail.
|
| But it's good to know there are ways to get much more perf
| out of it.
| 0xdky wrote:
| I did something similar (~2015) but using the kernel NFS client
| and having multiple mounts to the same volume using different IP
| addresses.
|
| Using vectored IO and spreading across multiple connections
| greatly improved throughout. However, metadata operations cannot
| be parallelized easily without application side changes.
|
| In more modern kernels, NFS supports 'nconnect' mount option to
| open multiple network connections for a single mount. I wonder if
| the approach of using libnfs for multiple connections is even
| required.
|
| https://github.com/0xdky/iotrap
| gjs278 wrote:
| oh nice. I just gave nconnect a try and things do appear to be
| faster for a remote mount I use.
| guenthert wrote:
| "To make use of multiple NICs I needed multiple NFS connections
| per NIC."
|
| What? I presume "multiple NFS connections per host." was meant.
| Not sure what those NFS connections are supposed to be though.
| NFSv3 is (stateless) request/response protocol on top of TCP/IP
| connections. NFSv4 introduced sessions, but here NFSv3 was used,
| wasn't it?
| gtirloni wrote:
| TL;DR; Author integrated libnfs in the fio benchmark tool.
| mayli wrote:
| Thanks, however I come to this post after reading the full
| blog.
| guenthert wrote:
| "This means that to establish multiple connections one must do
| something terrible like requiring NFS server to have multiple
| IPs"
|
| Oh the horror.
___________________________________________________________________
(page generated 2021-11-30 23:01 UTC)