If you’ve spent any time reading ZFS documentation, you’ve run across zfs set dedup=on and the promise behind it: never store the same block twice, anywhere in the pool. For a data hoarder sitting on terabytes of media, backups, and downloads, that sounds like free space. It mostly isn’t. ZFS dedup is one of the most misunderstood features in the whole stack, and the honest advice from almost everyone who has actually run it at scale is the same: leave it off unless you have a specific, measured reason not to.

This is a different topic from the file-level duplicate finders already covered on this site. Tools like rmlint and jdupes hash whole files and let you delete or hardlink the duplicates yourself, after the fact, with no filesystem feature involved. ZFS dedup works at the block level, continuously, and is baked into how the pool writes data going forward. The two solve overlapping problems in completely different ways, and for most people one of them is the wrong tool.

What ZFS dedup actually does

Every block ZFS writes already gets checksummed for the filesystem’s own integrity checking. Dedup reuses that checksum as a lookup key: before writing a new block, ZFS checks whether an identical block already exists in the pool. If it does, instead of writing the data again, ZFS just increments a reference count and points the new file at the existing block.

That bookkeeping lives in the Dedup Table (DDT), a mapping of every unique block’s checksum to its location and reference count. The DDT isn’t optional overhead you can skip, it’s the whole mechanism. Every single write and every single free operation on a deduplicated dataset has to consult and update it.

The real cost is RAM, not CPU

The commonly repeated rule of thumb is roughly 5GB of RAM for every 1TB of unique data in a deduplicated dataset, and it’s a rule of thumb for a reason: the real number depends on your average block size and record size, not a fixed ratio. What doesn’t vary is the shape of the problem. The DDT wants to live in ARC (ZFS’s read cache, covered in more depth in the RAM and ECC guide on this site) so every lookup is fast. The moment the DDT grows past what fits in RAM, every write and every delete starts needing a random-access read from disk just to check or update the table, on spinning drives that means a seek per block, and pool performance falls off a cliff that’s very hard to claw back from.

This is the part that catches people who turn dedup on, watch it work fine for a while, and then grow the dataset past the point where the DDT still fits in memory. Performance doesn’t degrade gracefully, it falls off a cliff once the table stops fitting, and by the time you notice, you’re already sitting on a pool full of deduplicated data.

Turning it off doesn’t undo it

This is the detail that bites the hardest: zfs set dedup=off only stops new writes from being deduplicated. Every block that was already deduplicated stays that way, and the DDT entries for that existing data stick around exactly as before, still consuming RAM, still needing to be checked whenever those blocks are freed or modified. The only way to actually get rid of an inflated DDT is to copy the data to a new, non-deduplicated dataset and destroy the old one, which on a large data hoard means a full rewrite of everything, not a flag flip. Test dedup on a dataset you’re prepared to migrate off of, never on your primary pool directly.

Checking what you’re already running

Before assuming either way, check the actual state of a pool:

zpool list -o name,size,alloc,free,dedupratio

A dedupratio of 1.00x means dedup is effectively doing nothing, either because it’s off, or because the data genuinely has no duplicate blocks to collapse. Anything meaningfully above 1.00x tells you dedup is active and actually saving space, worth knowing on a pool you inherited or don’t fully remember configuring. Per-dataset state is in the dedup property itself (zfs get dedup tank/dataset), since it’s set per dataset, not pool-wide, letting you run it selectively rather than as an all-or-nothing pool setting.

Newer OpenZFS releases help, but don’t change the math for most people

More recent OpenZFS versions have shipped a reworked dedup implementation, generally referred to as “fast dedup,” that restructures how the DDT is stored and pruned so stale entries get cleaned up far more efficiently than the original implementation ever managed. It’s a genuine improvement and worth having if your distribution ships it. What it doesn’t do is remove the fundamental RAM-scales-with-unique-data relationship. A better-maintained DDT is still a DDT that needs to fit comfortably in memory to perform well. Check your specific OpenZFS version’s release notes before counting on any particular behavior, rather than assuming every pool has the newer implementation.

When dedup is actually worth it

The cases where dedup earns its cost share one trait: a genuinely high rate of identical blocks, known in advance, not hoped for.

  • VM and container template storage. A pool holding many VMs cloned from the same base template, or many LXC containers built from the same image, has enormous block-level overlap by construction. Dedup can meaningfully shrink that footprint, though ZFS clones (covered in the ZFS snapshots guide on this site) already give you most of this benefit for free, with none of the DDT cost, as long as the VMs stay as clones of a common snapshot rather than independent full copies.
  • Backup targets receiving multiple near-identical full backups. If a backup tool writes full images rather than incrementals, and several of those images share most of their content (multiple full VM backups taken days apart, for instance), dedup can collapse that redundancy. Compare this against just using incremental backups or zfs send -i (covered in the ZFS send and receive guide) in the first place, which avoids writing the duplicate data at all instead of writing it and then deduplicating it after the fact.
  • A specific, measured dataset, not the whole pool. Because dedup is a per-dataset property, the right pattern is isolating the workload that actually benefits (a VM template store, say) onto its own dataset with dedup on, while the bulk of a media-focused data hoard, which has almost no duplicate blocks to begin with, stays off entirely.

What to do instead, almost always

For the overwhelming majority of data hoarding use cases, media libraries, document archives, general file storage, the blocks simply aren’t duplicated at anything like the rate that makes dedup’s RAM tax worthwhile. The better defaults:

  • Turn on compression (lz4 or zstd, covered in passing in the pool layout guide on this site) on every dataset. It’s cheap on modern CPUs, has no meaningful downside, and for compressible content saves real space with none of dedup’s RAM requirement.
  • Use snapshots and clones for workloads that are naturally duplicate-heavy by design, like VM templates, since cloning a snapshot shares the underlying blocks automatically and permanently, without a DDT tracking anything.
  • Run file-level dedup tools against a media library or document archive where the actual problem is duplicate whole files (the same movie ripped twice, a photo backed up from two different devices), which is what they were built for.

ZFS dedup isn’t a bad feature, it’s a narrow one, built for a specific shape of workload that most data hoarding setups don’t have. Measure your actual duplicate block rate before trusting the promise on the tin, and when in doubt, leave it off.