2026-08-22 Rambling about backups ================================= A set of winding posts on social ended up in this blog post. Yet another reason I dislike Apple. Well, I've had many reasons, and this one is not a new reason, it just manages to bite me in a new way. Apple uses the HFS+ filesystem. It stores filenames in a Unicode encoding using normal form D (NFD). Everybody else apparently uses normal form C (NFC). This matters because when you meet a non-ASCII character that has multiple representations as bytes, NFD and NFC use different bytes. As an example, consider the humble heavy metal a, ä. NFC uses ä LATIN SMALL LETTER A WITH DIAERESIS NFD uses COMBINING DIAERESIS plus a LATIN SMALL LETTER A Anyway, this just sets the scene. On a Linux system the ext4 filesystem says that filenames are just bytes. Nobody cares about encodings. So whatever tool was used to generate the filename, its encoding determines the bytes used. For example, I buy music on Bandcamp and the filenames contain some bytes. I copy them to an external drive. This external drive, sadly, is mounted by an Apple laptop using HFS+. It's smart. It recognises the bytes. It's UTF-8 NFC! It saves them as NFD. Because it's smart. Since I'm also smart, I keep the music files on an SD card of my little laptop, and on the SD card of the music server, and probably in a few other places. All of these filesystems are not smart. The filenames are just bytes that happen to encode the names using UTF-8 NFC. I am not smart. After a while I wonder whether I forgot to sync new files to the external drive and I run a tool. I might use rsync, for example. And of course it thinks that all the remote files with NFD encoded filenames should be deleted and all the local files with NFC encoded filenames should be added. But the content of the files is identical. Actually, you say, what about the --iconv option to rsync. It does just that, convert the filenames from one encoding to another. Ah, I say, Apple doesn't like GNU and so they have a version of rsync that doesn't support this option. And next we're into self-compiling a new rsync, or installing Homebrew, and at that point I'll say, see, yet another reason I dislike Apple. All of this to say that I decided to change how backups work and now I have a separate borg backup of the card to the disk where all my laptops are backed up to. I no longer sync the SD card to the external disk managed by the Apple laptop. That means, of course, that space requirements change. The important part is in the last two lines: The card uses 417G (the fact that we can have 1T cards still amazes me to no end). The backup disk has 457G available. This doesn't feel great, to be honest. # df -h Filesystem Size Used Avail Use% Mounted on udev 3.8G 0 3.8G 0% /dev tmpfs 791M 1.9M 789M 1% /run /dev/mmcblk2p2 115G 84G 25G 78% / tmpfs 3.9G 300K 3.9G 1% /dev/shm tmpfs 3.9G 12K 3.9G 1% /tmp none 1.0M 0 1.0M 0% /run/credentials/systemd-journald.service /dev/mmcblk2p1 448M 273M 147M 66% /boot none 1.0M 0 1.0M 0% /run/credentials/serial-getty@ttymxc1.service none 1.0M 0 1.0M 0% /run/credentials/getty@tty1.service tmpfs 791M 40K 791M 1% /run/user/1000 /dev/mmcblk0p1 938G 417G 474G 47% /run/media/alex/Card Vault /dev/sda1 916G 413G 457G 48% /mnt/backup Smart people have thought long and hard about strange questions like "what is information" and sadly I am not one of them. What little I know is this: if the data has obvious repetition, it can be compressed. If it looks like random noise, it cannot be compressed. That's why you can't keep zipping the zip files. Similarly, almost all media files are encoded and compressed such that there's is only very little additional compression can do for you. So now we get to file systems and backups. What if you had repetitions across files? If you have a system that stores files like an archive or a filesystem, you could take advantage of that. If you just look at whole files, you can do this already: if file A and file B have the same content, you could replace A with a hard link to B. Nobody would notice a difference. Well, except if you open either file and make changes to it. Depending on the tool you might change both files, or you might rename A to a backup file and write a new copy, so now A's backup and B are "the same file" and the new A is actually new. rdfind is a tool that helps you do this. As for backups, if you use something like rsync, you can create snapshots: if A hasn't changed between the last backup and this backup, just use a hardlink from this/A to last/A. And so, finally, to borg backup. Here, as I understand it, files are split into "chunks" and each chunk is deduplicated. So if files A and B each get split into two chunks and the first chunk of A is the same as the first chunk of B, then only one copy of that chunk is saved and the other is replaced by sort of "hardlink" or pointer. And now, finally, I can get to the damn point of this overly long post. I backed up a directory full of PDFs and MP3 files, mostly. And I would have thought that their internal compression makes it pretty hard to find matching chunks in other files unless the whole file is the same. But who knows. Look at the output: Duration: 9 hours 13 minutes 5.08 seconds Number of files: 85011 Utilization of max. archive size: 0% ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 446.85 GB 437.70 GB 285.23 GB All archives: 3.15 TB 2.65 TB 727.71 GB First, I'm not surprised that compression only gets us from 447G to 438G. But deduplication gets us down to 285G!? I have 150G of duplicate MP3s lying around? What is going on! Although to be fair, and going back to the original problem that sent me on this long adventure, @mathew@universeodon.com says that HFS+ is obsolete and modern Macs use APFS, which doesn't normalise; and Apple uses openrsync. But then again, in this situation I wouldn’t care – my problem is the normalisation. #Backup #Apple 2026-08-23. As it turns out, I was able to convert my external HFS+ drive to APFS using Apple's disk utility. This took very little time and worked in-place even though the external drive is encrypted. I must confess that this was a surprisingly positive thing. So now I'm using Krusader to compare directories. Wish me luck. Krusader sees a difference in a file called "Ion Constantinescu (drîmba) & Marin Cotoanță (cobză) - Folklore Dances from Romania, ca. 1950s - 06 Floricica Olteneasca.mp3", presumably because of letters like î and ă being represented differently. I am confused. There still is a difference? Remote: ❮ ssh bombias Last login: Sun Aug 23 12:22:15 2026 from 2a02:168:4822:0:3ac8:2533:50e5:9730 claudia@ae278fa2-d762-46b4-b521-e6c0398e3281 ~ % ls /Volumes/Extern/Audio/Canary\ Records/*Folklore*/Ion* | sed 's/.*\///' | od -c 0000000 I o n C o n s t a n t i n e s 0000020 c u ( d r 303 256 m b a ) & M 0000040 a r i n C o t o a n 310 233 304 203 0000060 ( c o b z 304 203 ) - F o l k l 0000100 o r e D a n c e s f r o m 0000120 R o m a n i a , c a . 1 9 5 0000140 0 s - 0 6 F l o r i c i c 0000160 a O l t e n e a s c a . m p 3 0000200 \n 0000201 Locally: ❮ ls Music/Audio/Canary\ Records/*Folklore*/Ion* | sed 's/.*\///' | od -c 0000000 I o n C o n s t a n t i n e s 0000020 c u ( d r 303 256 m b a ) & M 0000040 a r i n C o t o a n 310 233 304 203 0000060 ( c o b z 304 203 ) - F o l k l 0000100 o r e D a n c e s f r o m 0000120 R o m a n i a , c a . 1 9 5 0000140 0 s - 0 6 F l o r i c i c 0000160 a O l t e n e a s c a . m p 3 0000200 \n 0000201 The encoding is the same. So what's going on? Just to confirm, I mounted the external drive using sshfs and used the same directory. Same problem. This time, I'm just comparing one directory but the same file shows up as being different. This making me suspicious. Copy the remote file: ❮ rsync -ai bombias:/Volumes/Extern/Audio/Canary\ Records/Canary\ Records\ -\ Folklore\ Dances\ from\ Romania-\ ca.\ 1950s/Ion\ Constantinescu\ \(drîmba\)\ \&\ Marin\ Cotoanță\ \(cobză\)\ -\ Folklore\ Dances\ from\ Romania,\ ca.\ 1950s\ -\ 06\ Floricica\ Olteneasca.mp3 . >f+++++++++ Ion Constantinescu (drîmba) & Marin Cotoanță (cobză) - Folklore Dances from Romania, ca. 1950s - 06 Floricica Olteneasca.mp3 Diff locally: ❮ diff Music/Audio/Canary\ Records/Canary\ Records\ -\ Folklore\ Dances\ from\ Romania-\ ca.\ 1950s/Ion\ Constantinescu\ \(drîmba\)\ \&\ Marin\ Cotoanță\ \(cobză\)\ -\ Folklore\ Dances\ from\ Romania,\ ca.\ 1950s\ -\ 06\ Floricica\ Olteneasca.mp3 Ion\ Constantinescu\ \(drîmba\)\ \&\ Marin\ Cotoanță\ \(cobză\)\ -\ Folklore\ Dances\ from\ Romania,\ ca.\ 1950s\ -\ 06\ Floricica\ Olteneasca.mp3 No difference. This is driving me nuts. Let's rsync --dry-run just the offending file. ❮ rsync -ai --dry-run bombias:/Volumes/Extern/Audio/Canary\ Records/Canary\ Records\ -\ Folklore\ Dances\ from\ Romania-\ ca.\ 1950s/Ion\ Constantinescu\ \(drîmba\)\ \&\ Marin\ Cotoanță\ \(cobză\)\ -\ Folklore\ Dances\ from\ Romania,\ ca.\ 1950s\ -\ 06\ Floricica\ Olteneasca.mp3 Music/Audio/Canary\ Records/Canary\ Records\ -\ Folklore\ Dances\ from\ Romania-\ ca.\ 1950s/Ion\ Constantinescu\ \(drîmba\)\ \&\ Marin\ Cotoanță\ \(cobză\)\ -\ Folklore\ Dances\ from\ Romania,\ ca.\ 1950s\ -\ 06\ Floricica\ Olteneasca.mp3 No action taken! Let's try the whole directory: ❮ rsync -ai --dry-run --delete bombias:/Volumes/Extern/Audio/Canary\ Records/Canary\ Records\ -\ Folklore\ Dances\ from\ Romania-\ ca.\ 1950s/ Music/Audio/Canary\ Records/Canary\ Records\ -\ Folklore\ Dances\ from\ Romania-\ ca.\ 1950s/ *deleting Ion Constantinescu (drîmba) & Marin Cotoanță (cobză) - Folklore Dances from Romania, ca. 1950s - 06 Floricica Olteneasca.mp3 >f+++++++++ Ion Constantinescu (drîmba) & Marin Cotoanță (cobză) - Folklore Dances from Romania, ca. 1950s - 06 Floricica Olteneasca.mp3 Indeed, now it wants to delete and copy the damn file. The directory listing must be encoded in some way that gets lost when I use ssh to copy the file. And with that, and anger in my heart, I installed Homebrew again and got a new rsync. And as it turns out, that doesn't help. I cannot use the --iconv option. ❮ rsync -ai --dry-run --delete --rsync-path=/opt/homebrew/bin/rsync --secluded-args --iconv=UTF8-MAC,UTF8 bombias:/Volumes/Extern/Audio/Canary\ Records/Canary\ Records\ -\ Folklore\ Dances\ from\ Romania-\ ca.\ 1950s/ Music/Audio/Canary\ Records/Canary\ Records\ -\ Folklore\ Dances\ from\ Romania-\ ca.\ 1950s/ iconv_open("UTF-8", "UTF8-MAC") failed rsync error: requested action not supported (code 4) at rsync.c(134) [Receiver=3.5.0] Back to the start! Here's something weird: When I use just ls the filenames are borked! Scroll over to the right: claudia@ae278fa2-d762-46b4-b521-e6c0398e3281 ~ % ls /Volumes/Extern/Audio/Canary\ Records/*Folklore*/Ion* /Volumes/Extern/Audio/Canary Records/Canary Records - Folklore Dances from Romania- ca. 1950s/Ion Constantinescu (dr??mba) & Marin Cotoan???? (cobz??) - Folklore Dances from Romania, ca. 1950s - 06 Floricica Olteneasca.mp3 And when I pass the filenames through sed, they are correct: claudia@ae278fa2-d762-46b4-b521-e6c0398e3281 ~ % ls /Volumes/Extern/Audio/Canary\ Records/*Folklore*/Ion* | sed 's/.*\///' Ion Constantinescu (drîmba) & Marin Cotoanță (cobză) - Folklore Dances from Romania, ca. 1950s - 06 Floricica Olteneasca.mp3 WTF is going on.