smbd bottleneck on Gig network? reddit / freebsd ---------------------------------------------------------------------- edthesmokebeard · 2023-03-12 00:49 I'm writing large files over a gig network to a FreeBSD machine with a 4-disk RAIDZ2. I'm getting about 36 MB/s, and 'top' shows smbd at or near 25% CPU (4 core machine, 8GB RAM). I don't quite know yet if this is a ZFS issue, a Samba issue, or a FreeBSD issue. 'gstat' shows very short, infrequent bursts of utilization. Is there any way to make this faster? ---------------------------------------------------------------------- NL_Gray-Fox · 2023-03-12 02:12 25% on a 4 core... I think you have your problem there. > The short version is that smbd is not multithreaded https://www.samba.org/samba/docs/old/Samba3-Developers- Guide/architecture.html ---------------------------------------------------------------------- edthesmokebeard · 2023-03-12 02:35 Still? That links to a 26 year old document. ---------------------------------------------------------------------- NL_Gray-Fox · 2023-03-12 02:49 I think it's multithreaded, but only when writing multiple files (e.g. 1 file per core). You should be able to see this when writing a single large file. I can check maybe tonight. ---------------------------------------------------------------------- antiduh · 2023-03-12 03:07 I remember reading about this. Things might have changed with smb4. ---------------------------------------------------------------------- papageek · 2023-03-12 05:43 raiidz2 write will be as fast as the slowest disk. 8gb ram is a bit low as well. ---------------------------------------------------------------------- rykolepl · 2023-03-12 07:51 as fast as slowest disk times multiplied by number of drives - 2 ---------------------------------------------------------------------- KookyWait · 2023-03-12 08:26 Is anything (client or server) configured to use compression? May wish to see how turning it off affects performance. ---------------------------------------------------------------------- Far_Asparagus1654 · 2023-03-12 09:50 25% on a 4 core system is a bit of a yellow flag. Some process being "slow" on N cores when %CPU=100/N is nearly always due to a single core being maxed out. ---------------------------------------------------------------------- timvandijknl · 2023-03-12 12:58 >I don't quite know yet if this is a ZFS issue, a Samba issue, or a FreeBSD issue. Are you using `vfs objects = zfsacl` for the shares that are on ZFS datasets in your samba config? How fast is the transfer between ZFS datasets? Try the following tweaks and see if it makes a lot of difference in your setup: `sysctl vfs.zfs.prefetch.disable=1` `sysctl vfs.zfs.txg.timeout=5` `sysctl vfs.zfs.resilver_min_time_ms=5000` Also, 25% CPU usage for smbd is insignificant to irrellevant. It simply means that it doesn't need more CPU because something else is throttling the whole process; most likely blocking IO. Check your memory usage `zpool iostat` instead. ---------------------------------------------------------------------- shadow0rm · 2023-03-12 13:14 raidz2 is 1x write gain, 2x read gain.... just FYI ---------------------------------------------------------------------- edthesmokebeard · 2023-03-12 13:49 I am doing ZFS native compression but I'd expect to see cpu burn in other areas then. ---------------------------------------------------------------------- KookyWait · 2023-03-12 14:24 What is the maximum sustained write speed you've observed to the datasets with compression enabled? My suspicion is that this is likely what's causing your I/O to be CPU bound. Doubly so if there's other layers (e.g. SMB itself) that are also using compression. A walltime profile of the samba process could confirm it. devel/valgrind can be installed and used ("callgrind" mode) for this purpose. ---------------------------------------------------------------------- edthesmokebeard · 2023-03-12 14:30 About 36 MB/s - via the network. I have no means of writing that much data locally; it would be read off the same drives. ---------------------------------------------------------------------- KookyWait · 2023-03-12 14:33 I'm not sure I follow why you'd expect to see "CPU burn in other areas" - compression is a pure-CPU task, but reading and writing from disk is an I/O task. It is easy for sustained compression to be a CPU- limited task. Another way of testing this hypothesis w/o profiling would be to do 'zfs set compression=gzip-1' on the dataset and see if the performance improves (assuming you're not already using gzip-1; default is gzip-6 iirc). (EDIT: or just try writing to a non-compressed dataset, for shits and giggles) ---------------------------------------------------------------------- SuperDavers · 2023-03-12 15:18 I have that problem too but on a Linux box. 😕 ---------------------------------------------------------------------- Rufus_Tuesday · 2023-03-12 17:37 Have you tried NFS instead? ---------------------------------------------------------------------- KookyWait · 2023-03-13 13:43 >Also, 25% CPU usage for smbd is insignificant to irrellevant 25% CPU usage on a 4-core machine tells me a core is pegged and that the workload is CPU-bound, which is why I asked about compression settings in my answer. ---------------------------------------------------------------------- timvandijknl · 2023-03-13 15:22 Now i'm mostly curious where they got the 25% CPU from. If i do `top` on my machine and then run for example `clamav`, it peaks at 89.77% on the `wcpu` column, yet only uses core 4. So that's more than 25% 🤔 ---------------------------------------------------------------------- pramsky · 2023-03-14 17:51 Are you on Samba3 or Samba4 ? Can you paste and link your smb.conf ? Have you tried running iperf3 to see what bandwidth you get from the network link ? If you rsync a large file, ex a 1Gb file to the server, what bandwidth do you get ? ---------------------------------------------------------------------- shadeland · 2023-03-15 18:18 Did you use ashift=12 when you created the pool? Most drives today will have very poor write performance unless you specify ashift=12. ---------------------------------------------------------------------- edthesmokebeard · 2023-03-16 22:57 No, but ashift is 12. I wouldn't characterize 36 MB/s as "very poor write performance". ---------------------------------------------------------------------- shadeland · 2023-03-16 23:05 If they're large files and sequential writes, 36 MB/s is pretty poor. Most drives should get you at least 200 MB/s even in raid2z. Gigabit Ethernet will max you out at about 115 MB/s, but if you use NBit or 10 Gigabit you should be able to hit 200 MB/s on any hard drives that are relatively recent. I've got a bunch of WD white drives which I think are 5800 RPM and I still hit about 200 MB/s or so.