Data-Hoarding

The 3-2-1 backup guide on this site makes the case for an off-site copy without picking a specific home for it. That’s deliberate, because “off-site” covers everything from a hard drive at your parents’ house to a proper cloud object storage tier, and which one makes sense depends entirely on how you actually plan to use it. This article is the follow-up: assuming you’ve decided cloud is part of your off-site story, which service, and why. ...

Every drive you own is going to fail eventually. The part that catches people off guard is when. Hard drive failure rates don’t sit flat over a drive’s life, they follow what’s usually called a bathtub curve: a spike of early failures right out of the gate, a long flat stretch of low failure rate through the drive’s useful life, and a rising tail of wear-related failures years later. Burn-in testing exists to deal with that first spike, on your terms, before the drive is holding data you care about. ...

Every hoard past a certain size has duplicates in it, and not the kind you know about. It’s not usually the obvious “Copy of file (1).jpg” pattern. It’s the same movie sitting in three different folders because a download client, a manual import, and an old backup all landed a copy in a slightly different place. It’s a photo library synced from two phones that both uploaded the same shot. It’s a decade of “just copy it over to be safe” that never got cleaned up. None of this shows up until you actually go looking for it, and by then it can be a meaningful fraction of your total storage. ...

Most data hoarders don’t start with a matched set of drives. They start with whatever was on sale, whatever got shucked (see the hard drive shucking guide if that’s new to you), and whatever got inherited from an old build, added one or two at a time over a couple of years. The RAID redundancy guide on this site, and ZFS along with it, both assume something that mismatched-drive hoarders usually don’t have: drives that are the same size, ideally bought in a batch, treated as one block device. Buy a 4TB, then an 8TB eighteen months later, then a random 6TB that came out of a NAS somebody was tossing, and a real RAID array either wastes most of the bigger drives down to the smallest one’s capacity, or just won’t build at all. ...

Once a media library and a media server are both sorted out, the next question most data hoarders hit is how everything actually gets into the library in the first place. Doing it by hand, manually searching, downloading, renaming, and moving files into the folder structure your media server expects, works fine at a few dozen items. It falls apart well before a few hundred. The *arr stack (Sonarr, Radarr, Prowlarr, and a handful of siblings) is the standard homelab answer: a set of small, single-purpose applications that watch for content you want, find it, grab it, rename it, and drop it into your library automatically. ...

Every data hoarder eventually hits the same wall: the array keeps growing, the electric bill for spinning a dozen drives 24/7 keeps growing with it, and the off-site backup story (see the 3-2-1 backup guide if you haven’t nailed that down yet) gets more expensive every time you add another few TB. At some point, usually somewhere past the 50-100TB mark, people start hearing the same suggestion from old-school sysadmins: buy a tape drive. It sounds like a joke. It isn’t. ...

Once you’ve got a properly organized library sitting on real storage, the next question is what actually serves it to your devices. Plex, Jellyfin, and Emby all do the same basic job, scan a library, pull metadata, transcode when needed, and stream to whatever screen you’re on, but the tradeoffs between them are real and they don’t get smaller the longer you put off the decision. Here’s how to actually pick. ...

Shucking, buying an external USB hard drive purely to crack it open and pull out the bare drive inside, has been a data hoarder rite of passage for over a decade. Drive manufacturers have spent that whole decade trying to make it less appealing, and yet it keeps coming back every time there’s a good external-drive sale. Here’s whether it still makes sense, and what you’re actually signing up for when you do it. ...

Every data hoarder eventually runs into the same uncomfortable fact: a hard drive can report a read as fully successful while quietly handing back the wrong bits. No error, no SMART warning, no crash. The file opens fine. It’s just wrong. That’s bit rot, and RAID redundancy alone does nothing to catch it, because RAID is built to survive a drive failing outright, not a drive lying to you about data it’s still perfectly willing to serve. ...

The 3-2-1 backup rule gets repeated so often it’s turned into a slogan: three copies of your data, on two different media types, with one copy off-site. It’s good advice. It’s also advice that was mostly written with a few hundred gigabytes of irreplaceable photos in mind, not the 50, 100, or 200TB array a serious data hoarder ends up running. At that scale, “just keep an off-site copy” stops being a checkbox and starts being a real infrastructure and cost decision. Here’s what each part of 3-2-1 actually means once your data won’t fit on a shelf of external drives anymore. ...