Post ANGNhGC8pl25A8A8e0 by wook@fosstodon.org
(DIR) More posts by wook@fosstodon.org
(DIR) Post #ANG7WGbrLvltzrvb9M by urusan@fosstodon.org
2022-09-05T16:00:08Z
0 likes, 1 repeats
I've only seen one definite random bit flip and one potential random bit flip in my storage over my lifetime. I've been scanning for them for about the last 10-15 years.
(DIR) Post #ANG7pu6mimd40Ckyoa by urusan@fosstodon.org
2022-09-05T16:04:12Z
0 likes, 0 repeats
Specifically, I do backups using rsync and I generate an itemized list (with -i) before the actual backup (using -n to prevent it from executing).I can see when data that shouldn't be changing changes while it was just sitting around, because I get a spurious change report.
(DIR) Post #ANG9OK2YuhH4EEl1Bg by neglesaks@mstdn.io
2022-09-05T16:21:42Z
0 likes, 0 repeats
@urusan I do something similar but not as detailed as you have, and while I cannot attest to its accuracy, I think that over the past ten years, on a variety of magnetic storage, I've only encountered about a handful of unexplaibed minor data or filesystem corruptions on a data blob of ~2TB.Considering that my data backup practises used to be quite shoddy, I'm happy that even consumer data storage is somewhat robust. 😅
(DIR) Post #ANGNhGC8pl25A8A8e0 by wook@fosstodon.org
2022-09-05T19:01:50Z
0 likes, 0 repeats
@urusan .. care to share? :)
(DIR) Post #ANGQPqqy6E5NVTeKNk by urusan@fosstodon.org
2022-09-05T19:32:26Z
0 likes, 0 repeats
@wook I'm not sure what more there is to share. I'd be happy to share a more detailed description, but I'm not sure what kind of detail you'd be interested in.
(DIR) Post #ANGbyGT0TTJiHs3NnU by jkaniarz@mastodon.gamedev.place
2022-09-05T21:41:52Z
0 likes, 0 repeats
@urusan I had a SATA controller turn to the dark side one time. It would occasionally but rarely corrupt a block before writing. This caused trouble for RAID because there were no read errors reported by the drive and so there was no way of knowing which block was the bad one. Fortunately I was using RAID-Z which writes additional checksums.
(DIR) Post #ANGvEPHDNyNHNAwspU by wook@fosstodon.org
2022-09-06T01:17:44Z
0 likes, 0 repeats
@urusan .. thx. Wondering how you compare the 2 lists and notify of issues..
(DIR) Post #ANHMScSbrYyf4uHQn2 by urusan@fosstodon.org
2022-09-06T06:22:48Z
0 likes, 0 repeats
@wook So the basic idea is that I have a copy of my home area on an external hard drive. Let's call it /media/backup. After plugging it in and mounting it, I run:```rsync -arin --del /home/ /media/backup/ > changes.txt```This won't do the backup, but will produce changes.txt, which lists the changes to be done to get the two directories exactly the same again.
(DIR) Post #ANHMq2pVzK3JWQAE52 by urusan@fosstodon.org
2022-09-06T06:27:03Z
0 likes, 0 repeats
@wook The format is specific to rsync and you can find the full information on the rsync manpage (under the section for -i/--itemize-changes). However, it's pretty straightforward to tell at a glance which files were added, deleted, and modified in some way.It's also easy to jump over whole sections in practice. It doesn't matter what browser cache file changes happened.If the changes look good, you can execute the backup with:```rsync -ari --del /home/ /media/backup/ > changes.txt```
(DIR) Post #ANHNMdhESHGn7ypeFs by urusan@fosstodon.org
2022-09-06T06:32:56Z
0 likes, 0 repeats
@wook The options are:-a "archive" This turns on a bunch of random options that ensure that the metadata on your files don't get messed up.-r "recursive" This does a recursive copy.-i "itemized-list" This outputs an itemized list of changes-n "dry run" This makes it calculate the changes without making them.--del "delete" By default, rsync just adds and modifies files. This can be useful in some cases, but I want to delete old files.Also useful: -x Don't cross filesystem boundaries.
(DIR) Post #ANHNnURWxz1YFRI7RQ by urusan@fosstodon.org
2022-09-06T06:37:46Z
0 likes, 0 repeats
@wook Another important note: you should always run the `sync` command after a local rsync. This command ensures that all data is really written to disk.A sync will also happen if you shut down the external drive safely (sometimes also called eject). If you just yank it out though, it might not be done writing.`sync` should also be used after calling on the dark magic of the `dd` command.
(DIR) Post #ANHOIBxviOfZv05Me0 by urusan@fosstodon.org
2022-09-06T06:43:21Z
0 likes, 0 repeats
@wook If you're not familiar with rsync in general, also be aware that it's a great way to do remote file transfers via ssh (and in fact this is what it was really designed to do, and my use of rsync as a diff and local copy mechanism is...not the main use case, although it supports it very well).This will push files to another machine securely:```rsync -ari /local/path user@host:/remote/path```You can pull files just by flipping local and remote.
(DIR) Post #ANHtRt8DgwASyr0BjU by wook@fosstodon.org
2022-09-06T12:32:28Z
0 likes, 0 repeats
@urusan .. excellent info, thanks!