Post AZ0f38csFWVfFoDx68 by chromakode@mastodon.social
 (DIR) More posts by chromakode@mastodon.social
 (DIR) Post #AZ0f36QOQxL0QWMxMm by chromakode@mastodon.social
       2023-08-23T00:48:48Z
       
       2 likes, 3 repeats
       
       Yesterday morning, I pulled open my laptop to send a quick email. It had a frozen black screen, so I rebooted it, and… oh crap.My 2-year-old SSD had unceremoniously died.This was a gut punch, but I had an ace in the hole. I'm typing this from my restored system on a brand new drive.In total, I lost about 10 minutes of data. Here's how. (Spoilers: #zfs #zrepl)
       
 (DIR) Post #AZ0f38csFWVfFoDx68 by chromakode@mastodon.social
       2023-08-23T00:49:21Z
       
       0 likes, 0 repeats
       
       I don’t back up my drives, I replicate them.Last winter, I set up my first serious home network storage. Part of this project was setting up periodic backups of the computers I do creative work on. After surveying the options, one approach stood out: ZFS incremental replication.One of the flagship features of ZFS is the ability to take efficient point-in-time snapshots while it’s running. You can then send only the changed data to other machines...
       
 (DIR) Post #AZ0f3AcEqrBpQPwTNw by chromakode@mastodon.social
       2023-08-23T00:51:27Z
       
       0 likes, 0 repeats
       
       To automate taking snapshots and sending them to my NAS, I’m using a really cool piece of software called zrepl (by @problame). I configured it to snapshot and send my entire filesystem every 10 minutes.Since the snapshots are incremental, this is fine to run in the background on my home network to keep the replica up to date. The last run took 14 seconds to transfer and sent about 64 MiB.
       
 (DIR) Post #AZ0f3BmuUfJv3nwVeq by chromakode@mastodon.social
       2023-08-23T00:51:51Z
       
       0 likes, 0 repeats
       
       @problame Restoring the system was a learning process, and unfortunately quite manual. I let the 625 GiB ZFS receive operation run overnight.My snapshots are encrypted by the original computer (this is cool because the NAS can’t read them!). So I also needed to restore the encryption “wrapper key” to be able to use the backups.Not gonna lie, it was pretty terrifying until I had my first confirmation I could decrypt the data.
       
 (DIR) Post #AZ0f3D5NfTgn5Nam5Q by chromakode@mastodon.social
       2023-08-23T00:52:41Z
       
       0 likes, 0 repeats
       
       @problame To rebuild my system, I followed the OpenZFS guide for setting up a filesystem from scratch via Ubuntu 22.04 live USB:https://openzfs.github.io/openzfs-docs/Getting%20Started/Ubuntu/Ubuntu%2022.04%20Root%20on%20ZFS.html#step-4-system-configurationThis was a priceless resource for getting back up and running. It’d intimidated me in the past, but it’s *so* thorough, and I learned a ton going through the process. This is the best hand-on guide I’ve seen for modern partitioning and chrooting in a Debian environment.
       
 (DIR) Post #AZ0f3EGPHy6Sjrl5ua by chromakode@mastodon.social
       2023-08-23T00:53:05Z
       
       0 likes, 0 repeats
       
       @problame The end result was a beautiful moment: my laptop booted back up to right where I’d left it. Even my browser tabs restored my unfinished work from the previous night.There’s this classic series of Chromebook ads from 12 years ago where computers are repeatedly destroyed in elaborate ways, and the host picks up a new machine and picks up where they left off, with no data lost:https://www.youtube.com/watch?v=lm-Vnx58UYo
       
 (DIR) Post #AZ0f3FbMJYSOt8ZLCy by chromakode@mastodon.social
       2023-08-23T00:56:01Z
       
       0 likes, 0 repeats
       
       That ad has been in my imagination for over a decade. I finally achieved my dream of having a similar disaster recovery plan. And it worked!Setting ZFS up initially had a really high starting cost: it took a full filesystem swap. Maintaining it takes fairly knowledge-heavy and manual processes. But it certainly has unique benefits.This is the first time I can recall losing an SSD in over 15 years of using them. It was fantastic luck that I’d set up replication before my first one failed. 😇
       
 (DIR) Post #AZ0f3Gju5Gt0PvZgAK by chromakode@mastodon.social
       2023-08-23T00:56:17Z
       
       0 likes, 0 repeats
       
       Btw, if you’re curious, the offending drive was a WD_BLACK SN850 from my original Framework order. I’d heard unsettling stories on the Framework forums of this drive spontaneously dying or becoming unbootable. I guess it was my turn to roll some unlucky numbers.Amazon shipped me a new SK Hynix P41 SSD and a Sabrent NVMe enclosure in about 3 hours yesterday, which was phenomenal. I usually try not to order tech from there if I can avoid it, but credit where credit’s due.
       
 (DIR) Post #AZ0w4L9AsexFb6UWIK by tyil@fedi.tyil.nl
       2023-08-23T06:36:55.312Z
       
       0 likes, 0 repeats
       
       @chromakode@mastodon.social "Here's how" .
       
 (DIR) Post #AZ1zZmrOO5pHXuovZI by lamp@kitty.haus
       2023-08-23T18:50:58.279688Z
       
       0 likes, 0 repeats
       
       @chromakode @kkarhan i've had more ssd failures than hdd computers in a pc... (1 vs 0)like i've had weird things happen with hdd's but they've just kept on working on and on for years, decadesssd just spontaneously dies like this and I lost a bunch of stuff :(