Finding and Killing Duplicate Files at Scale: A Data Hoarder's Guide to Deduplication
Every hoard past a certain size has duplicates in it, and not the kind you know about. It’s not usually the obvious “Copy of file (1).jpg” pattern. It’s the same movie sitting in three different folders because a download client, a manual import, and an old backup all landed a copy in a slightly different place. It’s a photo library synced from two phones that both uploaded the same shot. It’s a decade of “just copy it over to be safe” that never got cleaned up. None of this shows up until you actually go looking for it, and by then it can be a meaningful fraction of your total storage. ...