Post B6AlCKrfwSieol2fui by gumnos@bsd.cafe
(DIR) More posts by gumnos@bsd.cafe
(DIR) Post #B6AlCKbMv5fw0BPeUq by rl_dane@polymaths.social
0 likes, 0 repeats
Maaaan, #Insync seriously pooped the bed.This is such a mess. 🤦‍♂️8000+ duplicate "foo (1)" files.Ugh.
(DIR) Post #B6AlCKrfwSieol2fui by gumnos@bsd.cafe
0 likes, 0 repeats
@rl_dane does sha1sum (or other checksum utility) return the same underlying data for all of them?$ sha1sum foo*035032494306a84824b5a03beed075d8851c5a8e foo (1)035032494306a84824b5a03beed075d8851c5a8e foo (2)f572d396fae9206628714fb2ce00f72e94f2258f foo(3)â‹®If so, you can list file-content-duplicates with$ sha1sum * | awk 'a[$1]++{sub(/^[^ \t]*[ \t][ \t]*/, ""); print}' | tr '\012' '\000' | xargs -0 echoand then if that list looks reasonable, change the `echo` to `rm` to delete them. It should keep a single copy of each file based on the content.
(DIR) Post #B6AlCL1bLYevJXgbNw by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@gumnos @rl_dane or just use jdupes…
(DIR) Post #B6CMoa1Nvg0aI7dw8m by gumnos@bsd.cafe
0 likes, 0 repeats
@mirabilos @rl_dane sounds similar to a utility I created a while back to create hard-links/symlinks for duplicates:https://github.com/Gumnos/dedupeit was a lot faster than the other dedupe programs I'd used because it did a first-pass based on file-size, and only checksummed the files if it encountered two files of the same size (the other one I tried checksummed everything).I usually use it for managing our photo-collection, where copies get scattered to various locations (album shared with family, album shared with friends, public Christmas pictures album, etc), to ensure a single copy exists with multiple links pointing to the same underlying data.
(DIR) Post #B6CMoaDnBXvuubRqTo by rl_dane@polymaths.social
0 likes, 0 repeats
@gumnos @mirabilosThat's a clever shortcut.I don't trust insync/google drive to sync symlinks. XDProper deduping is such a pain. XD
(DIR) Post #B6CMoaLaiYAhIn64dU by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@rl_dane @gumnos I use hardlinks and no Google…
(DIR) Post #B6COWVscmtak8w3gLw by rl_dane@polymaths.social
0 likes, 0 repeats
@mirabilos @gumnosHard to sync hard links. ;)
(DIR) Post #B6COWW4g45EUkJhJ8i by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@rl_dane @gumnos huh, no. rsync does it just fine.
(DIR) Post #B6CTerc6MmLhdU8Bv6 by gumnos@bsd.cafe
0 likes, 0 repeats
@mirabilos A quick test:$ mkdir src dst$ cd src$ echo hello > a$ ln -s a symlinked$ ln a hardlinked$ cd ..$ tar cvf src.tar src/$ cd dst/$ tar xvf ../src.tar $ cd src$ ls -lsFaishows that tar(1) keeps the hard-links for the inodes and maintains the symlink too.@rl_dane
(DIR) Post #B6CTernnfHhsDlbX9c by rl_dane@polymaths.social
0 likes, 0 repeats
@gumnos @mirabilosI was discussing this with a fedifriend this morning (while having my infamously huge breakfast)......is there a better option for archiving than tar? I've never used cpio (except invisibly, when I was using rpm, of course). Pax seems to just be a variation on tar, which has now been folded into most recent versions of tar, if my wikipedia reading is correct?
(DIR) Post #B6CTerz8z6mSmwuaps by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@rl_dane @gumnos don’t confuse frontends with implementations.You have several implementations of things:GNU cpio does cpio and has a cpio frontendGNU tar does tar and has a tar frontendpaxtar (OpenBSD/MirBSD) does cpio, pax, tar (and ar, in MirBSD) and has cpio, pax and tar frontendslibarchive (FreeBSD)’s bsdtar probably all does them all as well (and PKZIP and things)also Schily’s, etc.There’s no single “for archiving”, it depends on what you need.I tend to pack either sv4crc or ustar archives with paxtar, or just back things up with (currently) restic and make copies of things with rsync.
(DIR) Post #B6CWhs1na7XDeWHAq8 by mirabilos@toot.mirbsd.org
1 likes, 0 repeats
@rl_dane @gumnos ah actually GNU cpio can at least extract ustar (standard tar) and write an old GNU tar variant