Every data hoarder eventually runs into the same uncomfortable fact: a hard drive can report a read as fully successful while quietly handing back the wrong bits. No error, no SMART warning, no crash. The file opens fine. It’s just wrong. That’s bit rot, and RAID redundancy alone does nothing to catch it, because RAID is built to survive a drive failing outright, not a drive lying to you about data it’s still perfectly willing to serve.

If you’ve been building out redundancy (see the RAID redundancy guide) and you’re not also running checksums and scrubs, you have a blind spot. This is the piece that closes it.

What bit rot actually is

Bit rot is silent, low-level data corruption that happens at the media level, not the filesystem level. A few of the real mechanisms behind it:

  • Magnetic decay. The magnetic domains that store a bit on a platter can weaken over years, especially on drives that sit powered off or unread for long stretches. A bit that was a 1 can drift toward reading as a 0.
  • Firmware and controller bugs. Drive firmware occasionally miscalculates or mishandles a sector remap, especially on cheaper consumer drives under unusual load patterns.
  • Cosmic ray and electrical noise bit flips. Rare per-bit, but you’re not storing a few bits, you’re storing tens of terabytes across years of uptime. Rare events add up at scale.
  • Cabling and controller issues. A marginal SATA cable or a flaky HBA can introduce corruption in transit that has nothing to do with the drive itself.

None of these trigger a read error. The drive’s own error-correction usually can’t tell the difference between “this is what was written” and “this is close enough to what was written that ECC didn’t flag it.” The data comes back, the read succeeds, and it’s wrong.

The scale problem is what makes this actually matter. Manufacturer unrecoverable-read-error (URE) specs for consumer drives are commonly quoted around 1 in 10^14 bits read. That sounds tiny until you do the math: 10^14 bits is about 12.5TB. If you’re reading and rebuilding arrays in the tens of terabytes, you are well within the range where a genuinely undetected error becomes plausible, not hypothetical.

Why RAID doesn’t save you here

Traditional RAID (hardware RAID controllers, mdadm, most NAS-appliance “RAID” implementations) has no concept of “is this specific bit correct.” It knows which drive holds which stripe, and it can rebuild a drive that fails to respond. It has no way to know that a drive responded successfully with corrupted data, because it never checksums the actual content, only the parity math across drives.

Worse, corruption can propagate. If a bit flips on disk and you don’t catch it, and then a real drive failure happens later, RAID’s rebuild process will happily reconstruct that corrupted block using parity that was calculated against already-wrong data. You don’t get a warning. You get a rebuilt array that’s internally consistent and still wrong in the same spot.

This is exactly why “I have RAID, I’m covered” is a common and expensive misunderstanding. RAID answers “did a drive die.” It does not answer “is my data actually correct.”

What actually catches it: checksums

The fix is a filesystem (or overlay tool) that checksums data at write time and verifies that checksum on every read, independent of whatever the drive itself reports.

ZFS is the most common way homelabbers get this, and it’s built in, not bolted on. Every block gets a checksum (fletcher4 by default, sha256 available for stronger guarantees at a small CPU cost) stored in the parent block pointer, not next to the data itself, so a corrupted block can’t corrupt its own checksum along with it. On every read, ZFS recomputes the checksum and compares it. If it doesn’t match and you have redundancy (mirror, RAIDZ), ZFS transparently pulls the correct copy from redundancy, repairs the bad copy in place, and you never see an error, just a counter that ticks up in zpool status.

Btrfs offers the same core idea, checksummed data with self-healing from redundancy, though its RAID5/6 modes have a long-documented write-hole issue that makes them a worse fit for a hoarding array than ZFS’s RAIDZ. Btrfs in a mirror (RAID1) configuration is solid and a reasonable ZFS alternative if you’re already committed to a Btrfs-based distro.

Without ZFS or Btrfs, you’re not out of options, just doing it manually. Tools like par2 can generate recovery/parity files alongside your data that let you detect and sometimes repair corruption after the fact, and periodic checksum manifests (a simple sha256sum pass logged to a file, compared against on a schedule) will at least tell you something changed, even if they can’t self-heal it. This is meaningfully more manual labor than ZFS just doing it for you continuously, but it’s better than nothing if you’re stuck on ext4/XFS for other reasons.

Scrubs: the part people skip

Checksums only catch corruption when a block is actually read. If a file sits untouched for a year, nothing checks it in the meantime, unless you run a scrub. A scrub walks every allocated block in the pool, verifies its checksum, and repairs anything that fails, whether or not that block would otherwise ever be read again.

# ZFS
zpool scrub tank

# check status / progress
zpool status tank

Scrub cadence matters more than people expect. A common, reasonable schedule:

  • Monthly for pools on enterprise or NAS-rated drives with a decent duty cycle.
  • Every 2-3 weeks for pools built on older or consumer-grade drives, or drives you bought used (see the buying used enterprise gear guide for what to actually vet before trusting a drive with real data).

Scrubs are I/O-intensive and can take many hours on a large pool, so schedule them for low-usage windows (cron plus zpool scrub is the standard approach) rather than running them ad hoc during active use. Watch zpool status after each one. The CKSUM column ticking above zero on a specific drive, even if it’s still ONLINE and passing SMART, is an early warning sign worth acting on before that drive fails outright.

SMART monitoring is a different layer, not a substitute

SMART attributes (reallocated sector count, pending sector count, UDMA CRC error count) tell you about the drive’s own health assessment of itself, and they’re worth watching (smartctl -a /dev/sdX, or a tool like smartd running continuously with email alerts). But SMART passing does not mean your data is correct, for the same reason RAID being healthy doesn’t mean your data is correct: SMART detects drive-level problems the drive itself recognizes, not silent corruption that reads back clean at the drive’s own read-verification layer. Run both. They catch different failure modes.

The part that actually matters: corruption spreads into backups

This is the piece that turns bit rot from an annoyance into a real data-loss risk. If corruption isn’t caught at the source, your backup process (see the 3-2-1 backup guide if you haven’t set this up yet) will faithfully copy the corrupted file to every destination in your backup chain. A year from now, when you actually need that file, you may discover every copy you have is wrong, because every copy was made after the corruption already happened.

This is the strongest practical argument for checksummed storage on your primary array specifically, not just your backups. Catching corruption at the source, before it ever gets backed up, is the only point in the chain where you can actually recover the original data instead of propagating the bad copy everywhere.

If you’re running backup tools with their own integrity verification (Restic and Borg both do this, see the Docker backup guide), that’s a second, independent layer of protection, since those tools checksum at backup time and can detect drift on their own repositories. It’s not a substitute for catching rot at the source, it’s a second net underneath it.

The practical setup, if you’re starting from nothing

  1. If you’re building or rebuilding a pool and have the choice, use ZFS (or Btrfs in mirror mode) specifically for the self-healing checksum behavior, not just the RAID-equivalent redundancy.
  2. Schedule scrubs on a real cadence (monthly baseline, more often on older/used drives) and actually check the results, not just fire-and-forget the cron job.
  3. Run SMART monitoring in parallel as a separate signal, not a replacement.
  4. If you’re stuck on a non-checksummed filesystem, add par2 or a periodic hash-manifest comparison as a manual substitute. It’s more work, but silent corruption with zero detection is worse.
  5. Treat a rising CKSUM count on a specific drive as an early retirement signal, the same way you’d treat climbing reallocated sectors in SMART.

None of this is exotic or expensive. It’s mostly a filesystem choice you make once and a cron job you set up once. The cost of skipping it isn’t visible until the day you actually need a file back and discover every copy of it has quietly been wrong for months.