Appalling performance on external USB SMR drive - misconfiguration? reddit / freebsd ---------------------------------------------------------------------- OtherJohnGray · 2022-05-05 07:09 I'm trying to do an Rsync backup to an external Seagate ST4000DM004 barracuda SMR drive, and am seeing write throughput drop to 32kb/s once the CMR cache is full. Using the same drive on Linux and copying the same set of files, the performance issue doesn't happen, it seems to maintain over 15mb/s write throughput long after the CMR cache is exhausted. I'm wondering if I have done something to misconfigure the partition or filesystem, or am somehow managing to do random I/O rather than sequential via Rsync. I tried with both UFS/GELI and ZFS without encryption and with ashift=12, but got exactly the same result. geom disk list is reporting the drive as 512b sector and 4096b stripe. I set up the UFS/GELI partition with: gpart destroy -F /dev/da1 gpart create -s gpt /dev/da1 gpart add -a 1m -t freebsd-ufs da1 geli init -s 4096 /dev/da1p1 geli attach da1p1 newfs -j /dev/da1p1.eli mkdir /mnt/backup mount /dev/da1p1.eli /mnt/backup rsync -avx /data /mnt/backup EDIT: It’s looking like it’s just UFS with journaling causing massive SMR write amplification. ZFS is working great now, it’s just that with UFS -j, once I terminate a large copy that had been running slow for an hour or so, it seems to leave the disk in a bad state for 20min or more afterwards even across reboots - I’m guessing because of what’s still in the CMR cache. Any operation that writes to the disk at all has multi-second latency during that time. I initially tested with ZFS immediately after hitting problems with UFS, while the disk was still running slow, which made it look like ZFS had the problem too. I’ve been able to reproduce that as well. I’ll try UFS -U tomorrow to confirm whether it’s just the journaling. EDIT 2: newfs -U still exhibits the same problem (on top of GELI, fwiw) ---------------------------------------------------------------------- AntoineDarquier · 2022-05-05 07:56 I would recommend formatting USBs in ZFS instead of UFS. Here are the instructions: [http://dalvikplanet.blogspot.com/2020/03/how-to-format- pendrive-with-zfs.html](http://dalvikplanet.blogspot.com/2020/03/how- to-format-pendrive-with-zfs.html) It works very well for me. I have my desktop and netbook that use FreeBSD + ZFS. And I have a notebook with Ubuntu 20.04 on it. For example, on the Ubuntu 20.04 notebook, the USB is automatically mounted at startup. So I can take the USB out of the Ubuntu notebook and transfer some data from the FreeBSD systems, and then, when I plug it back into the Ubuntu notebook and boot the notebook, the ZFS file system of the USB is automatically mounted and available. Very convenient. ---------------------------------------------------------------------- OtherJohnGray · 2022-05-05 08:02 I still got the performance problem using zfs though - ashift 12, recordsize 128kb default. ---------------------------------------------------------------------- AntoineDarquier · 2022-05-05 08:28 I know, you mentioned it in your original post. I mean you should give it a try with the instructions in my link. You use special options that may have a negative impact. With my SanDisk I get 15MB/s write speed, which is the maximum for USB 2.0 So I would first completely overwrite your USB with zeros so you are sure that there is no corruption. And then I would give it a try with the exact steps in my link. It may not solve your problem, but it would be strange, as I'm not experiencing any performance issues with this configuration. ---------------------------------------------------------------------- PreviousWay · 2022-05-05 08:55 Try without journaling (newfs -j) . ---------------------------------------------------------------------- OtherJohnGray · 2022-05-05 08:57 Ah! Good point. That’s not going to play well with SMR. Is UFS journaling much different to EXT4 journaling? ---------------------------------------------------------------------- eigelp · 2022-05-05 10:57 ZFS is known to have issues with SMR drives, though I’m not sure whether that only applies to arrays or single disks as well. ---------------------------------------------------------------------- OtherJohnGray · 2022-05-05 11:16 It’s weird, these do better with ZFS on Linux as well. I wonder if they are causing problems with some lower level block device driver or usb driver? ---------------------------------------------------------------------- gldisater · 2022-05-05 12:24 USB 2 can do 35MB/s. ---------------------------------------------------------------------- vermaden · 2022-05-05 12:34 I currently copy me 3+ TB of data from my local SSD drive with ZFS on GELI to USB 3.0 SMR 5TB 2.5 drive with ZFS on GELI. Also with `rsync(1)`. After copying for several hours I get constant 45-50MB/s of speed on FreeBSD. % df -g /media/wd Filesystem 1G-blocks Used Avail Capacity Mounted on wd/data 4527 2007 2520 44% /media/wd Here is the `iostat(1)` output for every 10 minutes (600 seconds). % iostat 600 tty ada0 ada1 da0 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 21 352 19.0 26 0.5 21.1 60 1.2 7.1 4 0.0 1 0 1 0 99 1 69 73.8 395 28.5 23.3 78 1.8 97.6 242 23.1 9 2 10 1 77 1 1 513 120 59.9 29.5 39 1.1 742 65 46.8 4 8 17 2 69 0 2 615 94 56.6 19.1 22 0.4 751 68 49.8 1 3 14 1 82 0 0 561 106 57.9 17.9 20 0.4 760 70 52.0 1 2 14 1 82 0 0 1015 57 56.8 18.4 16 0.3 769 68 50.9 1 3 15 1 81 0 0 1017 57 56.3 18.5 16 0.3 757 68 50.6 1 3 14 1 81 0 1 752 72 53.0 16.6 23 0.4 765 67 50.1 1 1 13 0 85 0 0 1014 51 50.1 16.5 21 0.3 723 68 48.3 1 1 13 0 86 0 0 1012 51 50.2 19.8 18 0.3 743 68 49.2 1 1 12 0 86 0 0 1013 48 47.2 19.0 20 0.4 715 66 45.9 1 1 12 0 86 0 0 1011 44 43.8 19.8 17 0.3 684 64 42.4 1 1 11 0 88 0 0 1011 49 48.8 19.2 17 0.3 738 66 47.6 1 1 13 0 86 2 0 1011 52 51.0 38.9 44 1.7 764 65 48.7 8 1 14 0 78 0 0 1015 56 55.6 29.3 36 1.0 799 68 53.1 6 1 15 0 78 0 0 991 57 55.3 37.7 55 2.0 791 68 52.2 9 1 16 0 75 0 8 1013 52 51.3 35.3 41 1.4 757 66 48.6 8 1 14 0 76 1 0 1009 57 56.5 35.9 38 1.3 763 69 51.4 9 1 14 0 76 0 0 1009 59 57.7 21.0 22 0.4 783 68 52.3 2 1 13 0 84 0 2 958 63 58.7 19.0 26 0.5 791 68 52.7 2 1 14 0 83 0 0 1008 59 58.2 19.7 29 0.6 776 67 51.1 8 1 17 0 73 0 0 1009 54 53.4 37.7 28 1.0 774 66 49.8 11 1 16 0 71 0 0 1015 55 54.8 19.0 26 0.5 778 67 50.8 8 1 16 0 75 0 0 1010 55 54.6 19.8 26 0.5 774 67 50.8 8 1 16 0 75 1 96 256 222 55.5 28.0 30 0.8 773 66 50.1 12 1 17 0 70 0 19 1010 54 53.0 25.6 38 0.9 776 67 50.9 6 1 14 0 78 1 19 1012 54 53.3 42.7 71 3.0 774 67 50.6 11 1 15 0 73 This is my 'ZFS Tuning' on that SMR drive if that helps: # zpool history wd History for 'wd': 2022-05-04.23:25:06 zpool create wd /dev/da0p1.eli 2022-05-04.23:26:00 zfs set compression=zstd wd 2022-05-04.23:26:06 zfs set atime=off wd 2022-05-04.23:26:15 zfs set recordsize=1m wd This is the `rsync(1)` command I use: % rsync --modify-window=1 -l -t -r -D -v -S -H --force --progress --no-whole-file --numeric-ids --delete-before /files/ /media/wd/files/ Maybe your disk is just broken? I also use the Seagate 5TB SMR drives (ST5000LM000) as described here: https://vermaden.wordpress.com/2019/04/03/silent-fanless-freebsd- server-redundant-backup/ ... and they behave the same with 30-55 MB/s speed. ---------------------------------------------------------------------- AntoineDarquier · 2022-05-05 12:42 I would like to clarify: **1.** I'm talking about write speeds, not reads **2.** I am not talking about the theoretical speeds, but what the maximum speed of the best USB 2.0 sticks is in practice. I myself have never been able to write faster than +- 15 MB/s on a SanDisk USB 2.0 device, on any operating system/hardware. [https://www.photoflashdrive.com/help-center/what-is-the-speed-of-a- flash-drive/](https://www.photoflashdrive.com/help-center/what-is-the- speed-of-a-flash-drive/) The **read speed** on a USB 2.0 is much faster than the write speeding ranging from **15MB/s to 30+ MB/s**. [https://arstechnica.com/civis/viewtopic.php?t=483757](https://arstech nica.com/civis/viewtopic.php?t=483757) Our office bought some new Simpletech USB 2.0 256MB flash drives. I checked the Simpletech website and they state their drive is "USB 2.0 Embedded" which "operates at a write speed of up to **7.5MB/sec**". [https://forums.tomshardware.com/threads/speed-of-a-usb-3-0-drive-on- a- usb-2-0-port.2366399/](https://forums.tomshardware.com/threads/speed- of-a-usb-3-0-drive-on-a-usb-2-0-port.2366399/) I would like to upgrade my USB 2.0 4GB flash drive which I benchmarked to write at only 4.4MB/s. But I see that **the max write speed of other USB 2.0 flash drives is below 15MB/s** (the USB 2.0 standard allows for a max transfer speed of 60MB/s). The fact that I'm still downvoted even though I'm one of the most knowledgeable people out there. What stupid things should I say before I get an upvote? ---------------------------------------------------------------------- PreviousWay · 2022-05-05 12:45 I don't know much about linux. Thing is UFS will still have soft- updates by default iirc and having both of them gave me issues in the past. But I haven't used UFS in a while. You can check with tunefs -p /dev/ and what's enabled. ---------------------------------------------------------------------- OtherJohnGray · 2022-05-05 12:51 Thanks. Yeah that’s what I would have expected, and it’s roughly what I get on linux on a different machine. I wonder if it’s a hardware problem on the FreeBSD machine. I guess I should try another one of those drives I have and try different USB ports. ---------------------------------------------------------------------- lenzo1337 · 2022-05-05 15:06 Do you know what the external usb controller IC is? I had some issues with cruddy controllers before. You could also check the disk with smartmontools from pkgs/ports and just see what bus it shows up under using camcontrol. That's just where I would go to try stuff first. ---------------------------------------------------------------------- FrakenTurtle · 2022-05-06 13:13 Off-Topic Reply as the OP was asking about an external USB based Spinner ---------------------------------------------------------------------- FrakenTurtle · 2022-05-06 13:24 Check hdparm under linux and see if something is being set differently as the only thing I can see from the geom is the 512b sector - it should be a 4k/n with 512e and that might be the problem. ---------------------------------------------------------------------- OtherJohnGray · 2022-05-06 13:30 It’s looking like it’s just UFS with journaling causing massive SMR write amplification. ZFS is working great now, it’s just that UFS -j seems to leave the disk in a bad state for 20min or more afterwards even across reboots - I’m guessing because of what’s still in the CMR cache. I initially tested with ZFS immediately after hitting problems with UFS, while the disk was still running slow, which made it look like ZFS had the problem too. I’ve been able to reproduce that as well. I’ll try UFS -U tomorrow to confirm whether it’s just the journaling. ---------------------------------------------------------------------- lispstudent · 2022-05-06 16:42 +1 Thank you, very useful info. ---------------------------------------------------------------------- OtherJohnGray · 2022-05-07 01:36 Trying it your way worked fine. It seems to be just UFS, either with or without journaling. ---------------------------------------------------------------------- OtherJohnGray · 2022-05-07 01:43 newfs -U gives same problem, and on a different machine with another freshly unboxed drive of the same model. ZFS works fine after all though. ---------------------------------------------------------------------- vermaden · 2022-05-07 08:54 You can also more 'fine tune' that ZFS with these: - limitparallelism of writes to SMR drives **# sysctl vfs.zfs.spa_allocators=1** - limit or disable TRIM operations for SMR drives - - limit to 1 instead of default 64 (also good for slower/cheaper SSDs) **# sysctl vfs.zfs.vdev.trim_max_active=1** - - disable TRIM for enitre ZFS pool **# zpool autotrim=off pool** ---------------------------------------------------------------------- OtherJohnGray · 2022-05-07 09:29 Those are good tips, thanks! ---------------------------------------------------------------------- grahamperrin · 2022-05-11 19:13 > … UFS will still have soft-updates by default iirc … I think not. ---------------------------------------------------------------------- grahamperrin · 2022-05-11 19:14 > Is UFS journaling much different to EXT4 journaling? FreeBSD bug [261944 – FFS/UFS: explain soft update journaling (SUJ) in documentation and in tunefs(8)](https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=261944) ---------------------------------------------------------------------- grahamperrin · 2022-05-11 19:17 > without journaling (newfs -j) . For FreeBSD 13.0-RELEASE, not a good idea. FreeBSD bug [256746 – fsck_ffs completely broken for background fsck](https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256746) ---------------------------------------------------------------------- grahamperrin · 2022-05-11 19:19 > ZFS works fine after all though. If you like, mark your post: [answered](https://i.imgur.com/pTA9UeG.png) ---------------------------------------------------------------------- OtherJohnGray · 2022-05-11 22:25 Thank you! ---------------------------------------------------------------------- OtherJohnGray · 2022-05-11 22:25 done ---------------------------------------------------------------------- grahamperrin · 2022-05-12 01:24 > limitparallelism of writes to SMR drives > > `# sysctl vfs.zfs.spa_allocators=1` % sysctl vfs.zfs.spa_allocators sysctl: unknown oid 'vfs.zfs.spa_allocators' % > limit or disable TRIM operations for SMR drives > >* limit to 1 instead of default 64 (also good for slower/cheaper SSDs) > > `# sysctl vfs.zfs.vdev.trim_max_active=1` Are you sure that 64 is a default? % sysctl vfs.zfs.vdev.trim_max_active vfs.zfs.vdev.trim_max_active: 2 % ---------------------------------------------------------------------- grahamperrin · 2022-05-12 01:50 > … the instructions: http://dalvikplanet.blogspot.com/2020/03/how-to- format-pendrive-with-zfs.html … Instructing readers to change `kern.geom.debugflags` seems unnecessarily risky, especially without advice to revert the change. Giving an entire device to ZFS is normally not recommended. >> … check the the geometry of da0 wasn't altered: >> >> `gpart show -lp da0` >> >> `gpart: No such geom: da0.` Teaching people to work **without** a GPT seems peculiar. The post seems to teach unnecessary use of force. >> 14) Check pool while it's not imported. Notice that now instead of da0 displays "label/zfs": >> >> `zpool import` >> >> pool: tank How can the name be *tank* when the preceding step changed the name of the pool? ---- And so on …