Listing a directory (and entering) on ZFS takes a few seconds the first time? (re: sleep/idle?) reddit / selfhosted ---------------------------------------------------------------------- sofakng · 2023-11-17 20:06 I have a few ZFS ZPOOLs setup: NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT backup 1.81T 1.41M 1.81T - - 0% 0% 1.00x ONLINE - storage 21.8T 7.47T 14.3T - - 0% 34% 1.00x ONLINE - tank 87.3T 19.2T 68.1T - - 0% 22% 1.00x ONLINE - ...but when I enter a directory ("cd /tank") it takes about two seconds or longer. When I perform a directory listing it's the same result. However, after it "warms up" then everything is instant. What might be causing this? I'm using a Norco RPC-4220 enclosure with LSI/Avago 9211-8i cards in IT mode. The drives are WD Ultrastar DC 550s (on one array) and WD Reds (different array) and are connected to the backplane using SATA and then cards are connected using SAS (I think?). I don't think the drives themselves are spinning down because they don't seem to support it? hdparm -B /dev/disk/by-id/ata-WDC_WD40EFZX-68AWUN0_WD-WX12DB0LN94R /dev/disk/by-id/ata-WDC_WD40EFZX-68AWUN0_WD-WX12DB0LN94R: APM_level = not supported The only thing I can think of is that my server's RAM is 70% full which might be causing some problems? ---------------------------------------------------------------------- ethansky · 2023-11-17 21:05 The contents probably aren't cached, so it has to pull the directory info from the drives, and now that it's sitting in cache, subsequent `ls` will be more responsive. ---------------------------------------------------------------------- sofakng · 2023-11-17 21:07 Do you know if a 'special vdev' will increase the speed and have other benefits? ---------------------------------------------------------------------- ethansky · 2023-11-17 21:11 Yes, that was actually one of the things I specifically wanted when recently built my NAS. Got three of the 58GB P1600X optane SSDs and put them in a three-way mirror for special vdev. Viewing the contents of my datasets via SMB and NFS are pretty much instant despite it being a raidz2 across six disks. ---------------------------------------------------------------------- aetherspoon · 2023-11-17 23:37 Out of curiosity, do you have everything in /tank? As in tens of thousands of directories or files directly in the root versus a tree of directories? ---------------------------------------------------------------------- sofakng · 2023-11-17 23:57 Does your motherboard have 3x M.2 slots or are you using a PCIe adapter? Also, what's the purpose of the three-way mirror? For speed or redundancy? (I'll have to research more about these special vdevs... for example, if you lose the special vdev do you essentially lose all your data because the indexes \[?\] are gone?) ---------------------------------------------------------------------- sofakng · 2023-11-17 23:58 No there are only around 10 directories in the root and maybe 50 or so under that, etc. ---------------------------------------------------------------------- ethansky · 2023-11-18 00:32 > Does your motherboard have 3x M.2 slots or are you using a PCIe adapter? I'm using the [ASUS Hyper M.2 x16 PCIe 3.0 expansion card](https://www.newegg.com/asus-model-hyper-m-2-x16-card-v2-pci- express-to-4-x-m-2/p/N82E16815293043?Item=N82E16815293043), which requires PCIe 4x4x4x4 bifurcation or 4x4x4 bifurcation support on your motherboard. I got lucky on my b450 board as it supports 4x4x4 bifurcation. There are expansion cards out there that have a PLX chip, which is basically a fancy PCIE switch that lets the mobo not need bifurcation. > Also, what's the purpose of the three-way mirror? For speed or redundancy? (I'll have to research more about these special vdevs... for example, if you lose the special vdev do you essentially lose all your data because the indexes [?] are gone?) Redundancy in case of drive failure, technically there is a speed benefit but it doesn't really matter when you're talking about SSDs for special vdev. The special vdev stores the metadata for your filesm, so things like the "what" the file is and "where" it is and "how" much storage it takes, etc. If that special vdev dies, then all your files are gone because all the information about what and where the files are gone. These two videos from Wendel are pretty good at explaining the special vdev and why optane is such a good choice for it ([video 1](https://www.youtube.com/watch?v=mD6i2toN7lE)) (https://www.youtube.com/watch?v=QI4SnKAP6cQ).