Post AWgrnEPYNQxEQ5yFu4 by felsqualle@corteximplant.com
(DIR) More posts by felsqualle@corteximplant.com
(DIR) Post #AWgrn9W6ZeDxExzuvA by felsqualle@corteximplant.com
2023-04-05T16:24:03Z
0 likes, 1 repeats
One backup is no backup.Had to learn this the hard way on a new side-project today.This is what happend: I have a (pretty beefy) machine running Proxmox with ZFS for VM storage. Because I have to work with huge amount of data, I had to go with traditional hard drives instead of SSDs.ZFS was performing really bad because the HDDs couldn't keep up when the host kernel flushed the ZFS cache, effectively stalling the VM whenever the VM had to perform read requests.Easy solution!: Add another hard drive. Make backup. Erase zpool. Reimport backup as .qcow2 or .raw. Done.I did that. Created the update. No issue. Minutes later, destroyed the zpool and started reimporting the backup.Which failed - the backup hard drive that was okay MINUTES ago just died. Many read errors, nothing to recover.Thankfully, the data can be re-created from scratch, but I lost about one week of CPU power and multiple hours of manually fixing metadata.Yikes. Ensure your backups are redundant - don't rely on one drive. Not even for a simple task that should be finished in less than 30 minutes.
(DIR) Post #AWgrnAHxhiVNdOO8W0 by emanuel@schleuss.online
2023-06-14T18:30:34Z
0 likes, 0 repeats
@felsqualle i would even argue local secondary backup is still no backup, especially if it's "connected". Hacker attack, fire, water, anything can wipe a local backup. Always store some of them off-site. Thank god for 18TB drives - i grew up on 20MB HDDs š
(DIR) Post #AWgrnBSzKCv3HsYSLA by felsqualle@corteximplant.com
2023-04-05T16:29:55Z
0 likes, 0 repeats
By the way, this is the very first time that I ever had a data loss.
(DIR) Post #AWgrnEPYNQxEQ5yFu4 by felsqualle@corteximplant.com
2023-04-05T23:48:02Z
0 likes, 0 repeats
And Iām back in business. The system is running again with a clean install š»