This isn’t about the block-level dedup that Proxmox Backup Server or ZFS snapshots do under the hood (covered from that angle in the PBS backup guide). This is about whole files, sitting in your actual working storage, that are genuinely redundant and safe to consolidate, if you find them correctly and handle the consolidation correctly.

Why “same size, same name” isn’t good enough

The naive approach to finding duplicates is matching on filename and size. It’s fast, and it will absolutely miss real duplicates while flagging false ones. A renamed file with identical content won’t match by name. Two unrelated files that happen to be the same size will match by size and nothing else, and if a tool stops there, you get false positives.

The correct approach, and what every dedup tool worth using actually does, is a content hash. The file’s bytes get run through a hash function (commonly xxHash, BLAKE3, or SHA-256, depending on the tool and whether it prioritizes speed or cryptographic certainty), and two files with matching hashes are byte-identical, full stop, regardless of name, location, or timestamp.

Good tools do this in two passes to avoid hashing everything on disk, which would be brutally slow on a large hoard:

1. Group by file size first. Two files with different sizes can’t possibly be duplicates, so this pass is nearly free and eliminates most of the search space instantly.
2. Hash only the files that share a size with at least one other file. This is the expensive pass, but it’s now running against a small fraction of your total file count.

Some tools add a third shortcut: hashing just the first few KB of same-sized files before committing to a full hash, so two large files that happen to share a size but differ near the start get ruled out before a full read. If a tool you’re evaluating only does step 1, or claims to detect duplicates from filename patterns, don’t trust it with anything you’d miss.

The tools that actually do this well

rmlint is a long-standing, fast C implementation that deliberately does not delete anything on its own. Its default behavior is to scan and then emit a shell script (and a JSON report) listing exactly what it would do, that you review and run yourself. That review step is not a limitation, it’s the entire safety model, and it’s why it’s a reasonable first tool to reach for.

fclones is a newer Rust-based tool with the same safety-first philosophy: it separates “find” from “act,” supports multiple actions when you do act (delete, move, hardlink, or reflink, covered below), and handles very large file counts efficiently with a smaller memory footprint than older tools tend to need.

jdupes is a modernized, actively maintained fork of the older fdupes, with first-class hardlinking support built directly into the tool rather than as a follow-up script.

czkawka (and its GUI counterpart) goes further than exact-duplicate detection, it can also find similar-but-not-identical images and videos using perceptual hashing, useful if your duplicates problem is “the same photo re-exported at a different resolution” rather than a byte-identical copy. That’s a fundamentally different and fuzzier kind of matching than the others here, worth knowing about but worth treating with more caution since “similar” is a judgment call a hash comparison isn’t.

Any of these will get you a correct, hash-verified list of true duplicates on a modern homelab box in a reasonable amount of time. The tool matters less than what you do with the list it gives you.

The part everyone gets wrong: what “deduplicate” actually does to the files

Finding duplicates is the easy, safe part. Deciding what happens to them is where people get burned, because “deduplicate” can mean three genuinely different things with very different consequences.

Straight deletion. You keep one copy, delete the rest. Simplest option, recovers the most space relative to effort, but it’s irreversible and it means you now have exactly one copy where you used to have several. If those extra copies were accidentally serving as your only redundancy for something, you’ve just reduced your safety margin, not just your clutter. Never run a delete pass without a current backup of the data you’re touching, per the 3-2-1 backup guide, because the entire point of a dedup pass is bulk, fast, hard-to-fully-review changes across a lot of files at once.

Hardlinking. Instead of deleting a copy, you replace it with a hardlink to the one you’re keeping. Both filenames now point at the exact same inode on disk, so you get the space savings of deletion (the data is stored once) while every path that used to have its own file still resolves to something. The catch, and it’s a real one: a hardlink is not a copy. If anything opens one of those linked paths and edits the file in place rather than replacing it, every other hardlinked path sees that edit too, because there is only one file, wearing multiple names. This is exactly fine for media files you never edit in place. It is a landmine for anything that might get modified later, like documents, configs, or database files that happen to have duplicate copies lying around.

Reflinking. On filesystems that support it (Btrfs, XFS with reflink support, and recent enough setups of both), a reflink shares the underlying data blocks between two files copy-on-write style, the same space-saving benefit as a hardlink, but the moment either file is modified, the filesystem transparently splits the shared blocks so the edit only affects that one file. This is the safest of the three space-saving options if your filesystem supports it, since it behaves like an independent copy for every purpose except the storage cost, until an edit actually happens. It doesn’t apply on ZFS or ext4, so check what you’re actually running before assuming it’s available.

If you’re not certain which of these you want, straight deletion after a verified backup is the least surprising choice. Hardlinking is worth it specifically when you know the files in question are static (media libraries, archives, anything you treat as read-only), and reflinking is the best of both worlds exactly when your filesystem supports it.

One landmine specific to a media library setup: hardlinks you already have on purpose

If you’re running the *arr stack (see the Sonarr/Radarr/Prowlarr guide), you already have intentional hardlinks in your setup, between the download client’s completed-download path and the media library’s organized path, specifically so the same file exists in two locations without doubling your disk usage during the import step. That’s the whole point of getting the path mapping right in that setup.

A dedup tool running across your entire library will correctly identify those as duplicates, because at the content-hash level, they are. Most decent tools (rmlint, fclones, jdupes all do this) will detect that two paths already share an inode and skip them automatically, since there’s no space to reclaim there, they’re already one file wearing two names. But it’s worth confirming before you run a full pass, and it’s a good reason not to point a dedup tool’s delete or hardlink action at a media library directory without reviewing the report first. You want to catch the true accidental duplicate, the same episode imported twice into two different show folders, not undo the good kind of duplication your import pipeline set up deliberately.

A practical workflow

1. Back up first. Not optional. Dedup passes touch a lot of files at once, exactly the kind of change you want a recent backup underneath before you start.
2. Scan and review before acting. Every tool mentioned here supports a dry-run or a “find only” mode that produces a report without touching anything. Read it. Sanity-check a sample of the matches by hand, especially anything that surprises you.
3. Start with the obvious offenders, not your whole hoard at once: Downloads folders, old manual-backup directories, duplicate photo-sync sources. These tend to have the highest ratio of true duplicates to total files, and mistakes there are lower-stakes than touching an organized media library.
4. Pick delete, hardlink, or reflink deliberately, based on whether the files in question are static and whether your filesystem supports reflinks, not by default.
5. Re-run the scan periodically, not once. Duplicates accumulate continuously from the same sources that created them the first time, so this is maintenance, not a one-time cleanup.

None of this needs exotic tooling or a lot of manual effort once it’s set up. The actual risk isn’t the dedup tools themselves, all the ones covered here are careful by design. It’s skipping the backup step, or pointing a delete action at a directory you didn’t actually review first.