What is the best duplicate file finder, that preserves my source of truth? reddit / selfhosted ---------------------------------------------------------------------- parkercp · 2023-11-19 11:38 Having been so meticulous about taking back ups, I’ve perhaps not as been as careful about where I stored them, so I now have a loads of duplicate files in various places. I;ve tried various tools fdupes, czawka etc. , but none seems to do what I want.. I need a tool that I can tell which folder (and subfolders) is the source of truth, and to look for anything else, anywhere else that’s a duplicate, and give me an option to move or delete. Seems simple enough, but I have found nothing that allows me to do that.. Does anyone know of anything ? ---------------------------------------------------------------------- ElevenNotes · 2023-11-19 11:47 Seems like it would be easier you cleanup your backup strategy and start backups from scratch. ---------------------------------------------------------------------- speculatrix · 2023-11-19 12:07 Write a simple script which iterates over the files and generates a hash list, with the hash in the first column. find . -type f -exec md5sum {} \; >> /tmp/foo Repeat for the backup files. Then make a third file by concatenating the two, sort that file, and run "uniq -d". The output will tell you the duplicated files. You can take the output of uniq and de-duplicate. ---------------------------------------------------------------------- CrappyTan69 · 2023-11-19 12:14 Only runs on windows but I've been using double killer for years. Simple and does the trick ---------------------------------------------------------------------- parkercp · 2023-11-19 12:23 @ElevenNotes - I knew I could count on someone to state the obvious :-) - as that’s all sorted, I just want to ensure before I delete anything I can see nothing has been missed.. ---------------------------------------------------------------------- parkercp · 2023-11-19 12:28 Thanks @speculatrix - I wish I had your confidence in scripting - hence I’m hoping to find something that does all that clever stuff for me.. The key thing for me is to say something like multimedia/photos/ is the source of truth anything found elsewhere is a duplicate .. ---------------------------------------------------------------------- parkercp · 2023-11-19 12:31 Thanks @CrappyTan69 - I ideally need this to run on my NAS, and if possible be opensource/free - looks like for what I’d need Double Killer for, it’s £15/$20 - maybe an option as a last resort.. ---------------------------------------------------------------------- Mildly_Excited · 2023-11-19 12:36 I've used dupeGuru on windows for cleaning up my photos, worked great for that. Has a GUI and also works on linux! https://dupeguru.voltaicideas.net/ ---------------------------------------------------------------------- lilolalu · 2023-11-19 12:46 How should a duplicate finder know which is the source of the duplicate? ---------------------------------------------------------------------- ElevenNotes · 2023-11-19 12:58 Since you are the only one who maybe knows where what is stored: No chance. You could take one final backup of all your backup mess and archive that in case you later need something. ---------------------------------------------------------------------- parkercp · 2023-11-19 13:06 I’d like to find something that has that capability- so I can say multimedia/photos/ is the source of truth - anything identical found elsewhere is a duplicate. I hoped this would be an easy thing to as the ask is simply to ignore any duplicates in a particular folder hierarchy.. ---------------------------------------------------------------------- parkercp · 2023-11-19 13:08 That’s the thing - I know where all my backs ups are / it’s just the simplicity of the approach I’m looking for in the tool / because if there is only one source of truth then everything elsewhere is a duplicate? ---------------------------------------------------------------------- parkercp · 2023-11-19 13:10 Thanks - I think I tried that - but at the time it had no concept of a source (location) of truth to preserve / find duplicates against - has that changed ? They don’t seem to reference that specific capability on that link ? ---------------------------------------------------------------------- xewgramodius · 2023-11-19 13:18 I don't think there is a good way to tell which two duplicate files was "first" other than checking Creation Date but if this is Linux that attribute may not be enabled in your fs type. The closest thing I've seen is a python dedup scripts but after it identifies all the dups it deletes all but one of them and then puts hard links, to that real file, where all the deleted dups were. ---------------------------------------------------------------------- ElevenNotes · 2023-11-19 13:21 What is a duplicate for you? Same path structure? Same file name? Same content? Same crc32 hash? Depending on what is what this is not easily done. [Fdupes](https://github.com/adrianlopezroche/fdupes) comes to mind to find duplicate files for example. ---------------------------------------------------------------------- Mildly_Excited · 2023-11-19 13:24 Ah true, they don't have that capability. That's something I was missing as well when I was using it but only just now realized what you meant. ---------------------------------------------------------------------- parkercp · 2023-11-19 13:36 Hi @xewgranodius - I’m not actually worried about which came first, the key thing for me is which one is located in the directory (source) of truth. If it’s not in there then it’s fair game and can be moved/deleted.. ---------------------------------------------------------------------- parkercp · 2023-11-19 13:39 Same hash is a pretty good criteria for me and I use Fdupes and/or Jdupes - which are both good but they don’t quite have the preservation option - I’ve tried changing things to read only or protect them etc - but they are just work arounds - I ideally want something that a source of truth (file location wise) as its main driver for finding duplicates.. ---------------------------------------------------------------------- lilolalu · 2023-11-19 13:41 Well that's possible with a lot of deduplicators. But I'd take a look at duff: https://manpages.ubuntu.com/manpages/xenial/man1/duff.1.html https://github.com/elmindreda/duff The duff utility reports clusters of duplicates in the specified files and/or directories. In the default mode, duff prints a customizable header, followed by the names of all the files in the cluster. In excess mode, duff does not print a header, but instead for each cluster prints the names of all but the first of the files it includes. If no files are specified as arguments, duff reads file names from stdin. ---------------------------------------------------------------------- Digital-Chupacabra · 2023-11-19 13:49 > I wish I had your confidence in scripting You know how you get it? by fucking around and finding out! I'd say give it a go! Do a dry run of the de-dup to make sure you don't delete anything you care about. ---------------------------------------------------------------------- Lorric71 · 2023-11-19 15:49 Can't you edit the OP and add the requirements? You haven't even told us what NAS you have. ---------------------------------------------------------------------- UnrealisticOcelot · 2023-11-19 16:53 I use double killer on Windows and rmlint on Linux. With rmlint you can use tagged directories and tell it to keep the tagged and only match against the tagged. It has a lot of options, but no GUI. ---------------------------------------------------------------------- parkercp · 2023-11-19 16:56 Hi @Lorric71, updated my OP, however I’m happy to use anything on any platform (as I could map drives/shares etc.) the key thing is that it does what I need.. ---------------------------------------------------------------------- parkercp · 2023-11-19 17:01 Give me a few years and maybe :P - but for now I’d rather not risk important data with my own limited skills especially if there is a product out there that it’s tried and tested and hopefully recommended by someone in this sub.. I didn’t expect my ask to be quite so unique.. ---------------------------------------------------------------------- jerwong · 2023-11-19 17:10 I think you need a \\ in front of the ; i.e.: find . -type f -exec md5sum {} \\; >> /tmp/foo ---------------------------------------------------------------------- speculatrix · 2023-11-19 17:24 Thanks. I did have one but reddit saw it as an escape char and hid it. I added a second in the editor and now I see one in my comment. Cheers ---------------------------------------------------------------------- tech2but1 · 2023-11-19 19:01 Well it won't. You either tell it to assume that say oldest is always source and if there are identical files then you get asked to choose. ---------------------------------------------------------------------- Haliphone · 2023-11-19 19:25 You can set what folders you consider to be truth ---------------------------------------------------------------------- FantasticRole8610 · 2023-11-19 20:27 Directories can be marked as reference directories to which other files would be considered duplicates. ---------------------------------------------------------------------- parkercp · 2023-11-19 20:47 Hi, looking at the Help page I can’t see where that is done, could you direct me ? ---------------------------------------------------------------------- FantasticRole8610 · 2023-11-19 21:15 Here you go: https://dupeguru.voltaicideas.net/help/en/folders.html#folder-states ---------------------------------------------------------------------- nemec · 2023-11-19 22:02 If you're 100% sure that the dupes are only between your source of truth and "everything else", you can run fdupes then `grep -v /path/to/source/of/truth/root` the output - all the file paths that remain are duplicate files outside your source of truth, which can be deleted. ---------------------------------------------------------------------- speculatrix · 2023-11-19 22:06 A long time ago when I had to do stuff like this on Windows, I used ADCS https://download.cnet.com/advanced-directory-comparison-and- synchronization/3000-2248_4-10050020.html it made it very easy to compare directory trees, and find missing items or dupes. Maybe there's something like that for linux. ---------------------------------------------------------------------- parkercp · 2023-11-19 22:09 Thanks - You’ve tweaked my interest with this, as I was thinking along similar lines with however fdupes, my confusion is how do I remove everything it finds in my source of truth subfolder - and then what do I need to do with that list, is that a txt file or something ? Sorry for all the questions .. ---------------------------------------------------------------------- ZaxLofful · 2023-11-19 22:11 Normally people don’t care where it’s at…Are you sure none of the programs you tried have an option to show you locations? It seems silly to me that NONE of these types of program would have the feature to show you the locations of the files. ---------------------------------------------------------------------- frnkcg · 2023-11-19 22:25 I use jdupes. It's similar to fdupes but better. ---------------------------------------------------------------------- kbtombul · 2023-11-19 22:26 I use dupeguru as well, installed as a docker container so that it runs locally, rather than over the network with SMB. ---------------------------------------------------------------------- nemec · 2023-11-19 22:32 Something like fdupes -r ./backups ./source/of/truth > all-dupes.txt grep -v ./source/of/truth all-dupes.txt | tr -s '\n' > files-to- delete.txt Then check `files-to-delete.txt` to be very sure there is nothing in there you need to keep. while read line; do rm -v "$line" done