Grep beats locate if you have a large enough locate-DB reddit / freebsd ---------------------------------------------------------------------- vogelke · 2023-03-07 06:24 TIL -- grep/ugrep(1) on spinning rust beats locate(1) on an SSD, if your locate DB is big enough. I tried a few tests comparing [u]grep to locate. My FreeBSD box was built by iXsystems in 2020: Machine class: amd64 CPU Model: Intel(R) Xeon(R) E-2124 CPU @ 3.30GHz No. of Cores: 4 Memory: 16248MiB Drives: 2 WD Gold WD2005FBYZ Enterprise-class drives Capacity: 2TB each Speed: 7,200RPM On-Board Cache: 128MB I use ZFS, and my locate DB is built using GNU find/locate tools on a Samsung 240Gb SSD: me% locate --version locate (GNU findutils) 4.9.0 Copyright (C) 2022 Free Software Foundation, Inc. me% ls -lF /var/locate/locatedb -rw-r--r-- 1 root mis 97823563 06-Mar-2023 08:16:30 /var/locate/locatedb I expanded it to spinning rust -- /archive is mounted on a WD Gold drive. me% time locate /etc/passwd /dist/x86_64-oracle6.10-linux/etc/passwd ... /src/www/purl/dist-2.15-src/etc/passwd 3.75s user 0.02s system 99% cpu 3.770 total me% mkdir -p /archive/tmp/locate me% locate / > /archive/tmp/locate/today me% time ugrep -a /etc/passwd /archive/tmp/locate/today /dist/x86_64-oracle6.10-linux/etc/passwd ... /src/www/purl/dist-2.15-src/etc/passwd 0.21s user 0.26s system 99% cpu 0.471 total If you look for a string that's on every line, there's no advantage to using ugrep compared to anything else. Use "-a" to treat everything like text, or ugrep can skip output and lie to you about the results: me% time ugrep -a / /archive/tmp/locate/today | wc -l 8618511 5.33s user 0.42s system 91% cpu 6.310 total me% time locate / | wc -l 8618511 6.22s user 0.02s system 99% cpu 6.238 total ugrep finishes faster when looking for a pattern that doesn't match every line, which makes sense since it avoids ~650 Mbytes of I/O. I wanted to stop caching from skewing the results. AFAIK, the only reliable way to clear the filesystem cache on FreeBSD is to either reboot or remount the drive: root# df /archive Filesystem 1M-blocks Used Avail Capacity Mounted on tank/archive 355119 185057 170062 52% /archive root# zfs umount tank/archive root# zfs mount tank/archive me% time ugrep -a /etc/passwd /archive/tmp/locate/today /dist/x86_64-oracle6.10-linux/etc/passwd ... /src/www/purl/dist-2.15-src/etc/passwd 0.25s user 0.22s system 99% cpu 0.470 total Don't leave grep out: me% grep -V grep (GNU grep) 3.8 Copyright (C) 2022 Free Software Foundation, Inc. me% time grep -a /etc/passwd /archive/tmp/locate/today /dist/x86_64-oracle6.10-linux/etc/passwd ... /src/www/purl/dist-2.15-src/etc/passwd 0.26s user 0.33s system 99% cpu 0.596 total After running umount/mount, grep times were virtually identical: 0.35s user 0.25s system 99% cpu 0.593 total In case it's not obvious, I really like ugrep. ---------------------------------------------------------------------- antiduh · 2023-03-07 09:22 Why do you use gnu locate and grep on a Freebsd machine? ---------------------------------------------------------------------- antiduh · 2023-03-07 09:32 > ugrep finishes faster when looking for a pattern that doesn't match every line, which makes sense since it avoids ~650 Mbytes of I/O. I'm not sure this is true. If you're searching for the string '/etc/passwd', then when you're reading bytes/chars from the file to find matches, the most you can skip when having a non-match is 11 bytes, the length of that string. Given that you're going to read whole pages from the disk every time you do a read, you're not going to save yourself the IO. The smartest thing you could do, which ugrep does do, is concurrent IO - while looking for matches in a batch of pages, already be buffering the next batch of pages. ugrep is multi threaded and has many workers, which speeds things up dramatically. I'd suspect that gnu locate is single threaded which accounts for most of the performance difference. I don't know if you've been careful enough with your results to show that disk caching did not affect things. To be rigorous, you'd have to drop caches with every single comparison, and show that the performance was different with cold caches vs hot caches. Also, it'd be useful to compare apples to apples more - you're running locate on your ssd and ugrep on your rust. Just get the numbers for both on the ssd and both on the rust so we have a good basis for comparison. ---------------------------------------------------------------------- fromwithin · 2023-03-07 09:58 As opposed to what? ---------------------------------------------------------------------- antiduh · 2023-03-07 12:49 Why wouldn't you use the ones that came with freebsd? lappy(~) > which locate /usr/bin/locate lappy(~) > which grep /usr/bin/grep lappy(~) > grep --version grep (BSD grep, GNU compatible) 2.6.0-FreeBSD ---------------------------------------------------------------------- fromwithin · 2023-03-07 13:12 Oh I see.The OP is using GNU versions instead of the BSD versions. I thought there must have been some locate and grep alternative that I didn't know about. ---------------------------------------------------------------------- vogelke · 2023-03-11 02:18 > Why do you use gnu locate and grep on a Freebsd machine? For several years, I maintained fileservers at work which ran FreeBSD, Linux, and Solaris-10/11. I wanted completely consistent behavior for backups, etc. so I built the same versions of GNU find, locate, xargs, and grep on all my boxes. Things like "find ... -printf" are just too useful to do without. ---------------------------------------------------------------------- vogelke · 2023-03-11 02:51 > I don't know if you've been careful enough with your results to show > that disk caching did not affect things. You were right, and it wasn't just the caching; I had compression enabled and that made a HUGE difference in the amount of actual I/O taking place. me% ls -l -rw-r--r-- 1 vogelke mis 97829175 Mar 7 20:00 locatedb -rw-r--r-- 1 vogelke mis 654937692 Mar 7 19:27 today me% du -sm * 40 locatedb 86 today > Just get the numbers for both on the ssd and rust... The best part about ZFS is being able to quickly test things like this. I created two datasets, one for SSD and one for rust: root# zfs create -o compression=off -o primarycache=none \ -o secondarycache=none -o mountpoint=/bench-disk \ tank/bench root# zfs create -o compression=off -o primarycache=none \ -o secondarycache=none -o mountpoint=/bench-ssd \ zpool/bench Resulting properties for both: PROPERTY VALUE SOURCE compression off local primarycache none local secondarycache none local I copied the locatedb and expanded file lists to both and ran tests 20 times each. I love the repeat command in ZSH: #!/usr/local/bin/zsh cd /bench-disk/test printf "\n%s\n" "$(date)" pwd echo; repeat 20 do time /usr/bin/grep -a -c /etc/passwd today; done echo; repeat 20 do time grep -a -c /etc/passwd today; done echo; repeat 20 do time locate -c -d locatedb /etc/passwd; done echo; repeat 20 do time ugrep -a -c /etc/passwd today; done echo; repeat 20 do time ag --nocolor -a -c /etc/passwd today; done cd /bench-ssd/test printf "\n%s\n" "$(date)" pwd echo; repeat 20 do time /usr/bin/grep -a -c /etc/passwd today; done echo; repeat 20 do time grep -a -c /etc/passwd today; done echo; repeat 20 do time locate -c -d locatedb /etc/passwd; done echo; repeat 20 do time ugrep -a -c /etc/passwd today; done echo; repeat 20 do time ag --nocolor -a -c /etc/passwd today; done printf "\n%s\nDONE\n" "$(date)" exit 0 Cliff-notes version: I tossed the largest and smallest values for each run before calculating mean and median times. The weird thing -- running ag (Silver Searcher) apparently did some caching somewhere, but I haven't found out where yet. That's why I ran it last -- running ugrep followed by /usr/bin/grep again didn't change the /usr/bin/grep results, but *everything* got faster after running ag. Thu Mar 9 03:03:34 EST 2023 /bench-disk/test /usr/bin/grep -a -c /etc/passwd today (median=15.274 mean=15.285) grep -a -c /etc/passwd today (median=8.254 mean=8.256) locate -c -d locatedb /etc/passwd (median=5.384 mean=5.394) ugrep -a -c /etc/passwd today (median=6.545 mean=6.540) ag --nocolor -a -c /etc/passwd today (median=0.193 mean=0.193) ----------------- Thu Mar 9 03:15:34 EST 2023 /bench-ssd/test /usr/bin/grep -a -c /etc/passwd today (median=17.099 mean=17.082) grep -a -c /etc/passwd today (median=6.809 mean=6.806) locate -c -d locatedb /etc/passwd (median=4.299 mean=4.298) ugrep -a -c /etc/passwd today (median=4.384 mean=4.384) ag --nocolor -a -c /etc/passwd today (median=0.203 mean=0.203) Full results: an asterisk means this line was excluded when calculating stats. I can't put the complete output here, it's too big -- locate wins without compression or caching. I included the ag results because they're weird. Thu Mar 9 03:03:34 EST 2023 /bench-disk/test ag --nocolor -a -c /etc/passwd today (median=0.193 mean=0.193) * 0.13s user 0.27s system 6% cpu 6.764 total 0.12s user 0.07s system 99% cpu 0.193 total * 0.13s user 0.06s system 99% cpu 0.192 total 0.16s user 0.03s system 99% cpu 0.192 total 0.14s user 0.05s system 99% cpu 0.192 total 0.16s user 0.03s system 99% cpu 0.193 total 0.14s user 0.06s system 99% cpu 0.193 total 0.15s user 0.04s system 99% cpu 0.192 total 0.13s user 0.06s system 99% cpu 0.192 total 0.13s user 0.06s system 99% cpu 0.193 total 0.15s user 0.05s system 99% cpu 0.192 total 0.14s user 0.05s system 99% cpu 0.193 total 0.15s user 0.04s system 99% cpu 0.192 total 0.14s user 0.06s system 99% cpu 0.193 total 0.15s user 0.04s system 99% cpu 0.192 total 0.11s user 0.08s system 99% cpu 0.193 total 0.17s user 0.02s system 99% cpu 0.192 total 0.14s user 0.05s system 99% cpu 0.193 total 0.11s user 0.08s system 99% cpu 0.192 total 0.13s user 0.06s system 99% cpu 0.193 total Thu Mar 9 03:15:34 EST 2023 /bench-ssd/test ag --nocolor -a -c /etc/passwd today (median=0.203 mean=0.203) * 0.13s user 0.27s system 9% cpu 4.397 total 0.15s user 0.05s system 99% cpu 0.204 total * 0.14s user 0.06s system 99% cpu 0.203 total 0.16s user 0.05s system 99% cpu 0.204 total 0.14s user 0.06s system 99% cpu 0.204 total 0.12s user 0.08s system 99% cpu 0.203 total 0.16s user 0.04s system 99% cpu 0.204 total 0.15s user 0.05s system 99% cpu 0.203 total 0.15s user 0.05s system 99% cpu 0.203 total 0.14s user 0.06s system 99% cpu 0.203 total 0.15s user 0.06s system 99% cpu 0.207 total 0.13s user 0.07s system 99% cpu 0.203 total 0.10s user 0.10s system 99% cpu 0.203 total 0.14s user 0.06s system 99% cpu 0.203 total 0.12s user 0.09s system 99% cpu 0.203 total 0.16s user 0.04s system 99% cpu 0.203 total 0.12s user 0.08s system 99% cpu 0.203 total 0.15s user 0.05s system 99% cpu 0.203 total 0.13s user 0.07s system 99% cpu 0.203 total 0.14s user 0.06s system 99% cpu 0.203 total Thu Mar 9 03:26:34 EST 2023 DONE