FreeBSD 12/13 NVMe RAID1 boot reddit / freebsd ---------------------------------------------------------------------- void64 · 2021-08-25 02:00 Looking at building a few new systems using two Intel Optane NVMe drives. Eons ago software RAID gmirror got the job done. Today is ZFS boot on a mirror vdev of two NVMe SSD drives a good practice? I know it might be overkill, but for this purpose that's all we need for storage as these servers will be write intensive, but need some form of redundancy should an SSD fail. Speaking of fail. How does failure detection work with ZFS and a mirrored vdev? Does ZFS/NVMe have a predictive failure like SMART where ZFS can safely take the disk offline and be swapped? Is hot swap NVMe a thing in FreeBSD now ? ---------------------------------------------------------------------- oratpart · 2021-08-25 03:39 zfs mirrored nvme boot is delicious and you should definitely do it. zfs is built to distrust hardware and is really really good at fault tolerance. Today, I demonstrated breaking and rebuilding multiple mirrors in a pool by yanking disks out of a running production (backup) server. I had absolutely no fear there would be more than a few seconds of time to online the disks. With a pair if Optanes, that time will come in something like 365 years if you’re really cruel to them. Let us know how it works out. I’ve never hot swapped nvme but I don’t see why that would be a problem on FreeBSD. Hopefully someone else can address that. ---------------------------------------------------------------------- meatmechdriver · 2021-08-25 05:53 I’m working on some NVMe support right now, mostly reservations and set features support plumbed through camcontrol. I believe NVMe hotswap works, though I think some retimer cards have some problems. I’ve not been involved on that aspect yet, so take that with a big grain of salt. ---------------------------------------------------------------------- rykolepl · 2021-08-25 06:08 > Is hot swap NVMe a thing in FreeBSD now ? If you have your SSDs as PCI-e cards in the back of your server, I doubt that you will be able hot-swap them. And it's not limitation of freebsd, in almost all servers I worked with, you need to remove riser before replacing pcie card, and by removing riser you disconnect all pcie devices. PCI hotplug is supported on freebsd: https://wiki.freebsd.org/PCIHotplug but I don't know how usable/stable it is. With that said: https://forums.freebsd.org/threads/nvme-hot-swap- working.69045/ according to this thread, you could hot-swap NVME U.2 drives. I'm not sure if you can get optanes in U.2 format. I have never used U.2 drives, so I can't test that. Also, I agree with what /u/oratpart said. To add to this, I was once running server from one 120GB. I could go from one 120GB disk, to 2 250GB disks in raid1 without single reboot. Also, this server had only 2 SATA ports, so it was add disk to raid, remove old disk, add second new disk. ZFS is awesome and I can recommend using it. ---------------------------------------------------------------------- void64 · 2021-08-25 09:54 Hot swap isnt a huge issue, so long as I can offline/remove the drive in ZFS first then shutdown the box to replace if I have to. ---------------------------------------------------------------------- void64 · 2021-08-25 09:56 Optanes most certainly do have a u.2 model. Thats what I was thinking, not PCI-e cards. ---------------------------------------------------------------------- void64 · 2021-08-25 10:03 365 years? Heh. I did see they have 100 DWPD models now, thats pretty insane. Even the 10 DWPD models are more than enough. We are not even close to that, just a stat collecting box updating about 25,000 RRDs every five minutes until the end of eternity. ---------------------------------------------------------------------- DependentVegetable · 2021-08-25 12:20 ZFS for sure. Very reliable, vetted, documented and actively developed. Not to mention you get a boat load of very handy features (snapshots, zfs diff, multiple boot environments etc etc) as for disk failure, it will punt a disk if there are too many errors. If you have a stricter level of tolerance, you can always script your own via devd events. a few weeks ago, I had a disk in a special vdev fail. The first errors started Aug 17 10:55:48 backup42 kernel: ahcich0: is 00000000 cs 08000000 ss 00000000 rs 08000000 tfd 80 serr 00000000 cmd 0000db17 Aug 17 10:55:48 backup42 kernel: (ada0:ahcich0:0:0:0): SETFEATURES ENABLE WCACHE. ACB: ef 02 00 00 00 40 00 00 00 00 00 00 Aug 17 10:55:48 backup42 kernel: (ada0:ahcich0:0:0:0): CAM status: Command timeout Aug 17 10:55:48 backup42 kernel: (ada0:ahcich0:0:0:0): Error 5, Periph was invalidated and zfs tossed it out on its own at a few min later. Aug 17 10:57:07 backup42 ZFS[78548]: vdev state changed, pool_guid.... I could watch any number of metrics and fire on that if need be, but I have found in the field, zfs makes choices that work for me. As I dont have hot swap, a power off, replace drive, and then a simple replace. "zpool history" is very handy to show exactly what was done # zpool history zbackup2 | grep replace 2021-08-17.16:02:35 zpool replace zbackup2 18010240422847967786 /dev/ada0p2 ---------------------------------------------------------------------- system-user · 2021-08-27 05:14 I have optanes in U.2 running in ZFS mirror for the OS on HardenedBSD 13, connected to a NVME HBA that supports hot swap; works great and as expected. ---------------------------------------------------------------------- grahamperrin · 2021-08-28 05:46 > … Does ZFS/NVMe have a predictive failure like SMART … NVMe alone: there's S.M.A.R.T., however I don't know about its value for *prediction*. this drive was failing horribly with fewer than a hundred hours' use.