Deduplication

If you’ve spent any time reading ZFS documentation, you’ve run across zfs set dedup=on and the promise behind it: never store the same block twice, anywhere in the pool. For a data hoarder sitting on terabytes of media, backups, and downloads, that sounds like free space. It mostly isn’t. ZFS dedup is one of the most misunderstood features in the whole stack, and the honest advice from almost everyone who has actually run it at scale is the same: leave it off unless you have a specific, measured reason not to. ...

Every hoard past a certain size has duplicates in it, and not the kind you know about. It’s not usually the obvious “Copy of file (1).jpg” pattern. It’s the same movie sitting in three different folders because a download client, a manual import, and an old backup all landed a copy in a slightly different place. It’s a photo library synced from two phones that both uploaded the same shot. It’s a decade of “just copy it over to be safe” that never got cleaned up. None of this shows up until you actually go looking for it, and by then it can be a meaningful fraction of your total storage. ...