Ceph shows up in every “how do I get real shared storage for my Proxmox cluster” conversation, and for good reason, it’s the one option that’s fully native, fully integrated into the Proxmox web UI, and genuinely production-grade software running underneath some very large real-world deployments. It’s also the option most likely to leave a home cluster worse off than before you added it, because Ceph’s design assumptions (plenty of nodes, plenty of network, plenty of dedicated drives) don’t bend easily to a 3-node home lab built from whatever hardware was on hand. Here’s what Ceph actually is inside Proxmox, what it needs to run well, and when it’s the right call instead of the simpler alternatives.

What Ceph actually is

Ceph is a distributed storage system that pools raw disks from multiple nodes into one logical storage cluster, then presents that pool back to Proxmox as shared storage any node in the cluster can read and write. Unlike ZFS replication, which copies a finished disk image from one node to another on a schedule, Ceph writes are distributed and replicated in real time across multiple nodes before the write is acknowledged. That’s the core trade Ceph is selling: no replication lag, no window where a node failure loses in-flight writes, because the data was never only on one node to begin with.

Proxmox integrates Ceph directly, you can deploy, monitor, and manage a Ceph cluster entirely from the same web UI you already use for VMs and containers, no separate management plane to learn. That’s a real advantage over bolting on an external Ceph deployment, but it can also make it look easier to stand up than it is to run well long-term.

The moving parts

A working Ceph deployment has three kinds of daemons, spread across your nodes:

  • Monitors (MONs). Keep the cluster map, the authoritative record of what nodes and drives exist and where data lives. Ceph needs a majority of MONs up to keep the cluster healthy, the same quorum logic as corosync, which is why you want an odd number, 3 is the standard minimum for anything beyond a lab toy.
  • Object Storage Daemons (OSDs). One OSD per physical disk you contribute to the pool. This is where data actually lives. More OSDs spread across more nodes means more parallelism and more fault tolerance, but also more drives you’re dedicating entirely to Ceph rather than local VM storage.
  • Manager (MGR). Handles cluster metrics and the dashboard. Lighter weight than MON or OSD duty, usually co-located with a MON.

Data placement across OSDs is handled by CRUSH, an algorithm that calculates where each piece of data should live based on your cluster’s topology, rather than looking it up in a central index. This is part of why Ceph scales well at large sizes, but it also means the failure domain you configure (per-OSD, per-node, per-rack) actually matters, the default of spreading replicas across nodes is what protects you when a whole node goes down, not just a single disk.

What it actually costs to run well

This is where most home Ceph attempts go wrong, not because Ceph is unstable, but because it’s genuinely resource-hungry in ways that are easy to underestimate on hardware sized for a home lab rather than a datacenter.

  • A dedicated, fast network for Ceph’s own traffic. Replication and recovery traffic between OSDs is constant and can be heavy, especially when the cluster is rebalancing after a disk or node failure. Running this on the same 1GbE link as your VM and corosync traffic is a reliable way to get contention on all three at once. 10GbE or faster, on its own dedicated network segment, is the realistic minimum for a Ceph cluster you actually want to trust, not an optional nice-to-have.
  • At least 3 nodes, realistically more for headroom. Ceph can technically run on fewer, but you lose the ability to tolerate a node failure while keeping full redundancy, and small clusters make every single OSD a bigger fraction of total capacity, which makes rebalancing after a failure both more disruptive and more urgent.
  • RAM and CPU overhead per OSD, not just per node. Each OSD process wants real memory to do its job well, and this scales with the number of drives, not the number of nodes. A node running 4 or 5 OSDs needs meaningfully more RAM budgeted for Ceph than a node running one. This overhead is on top of whatever RAM and CPU your actual VMs need, it’s not something Ceph shares efficiently with a workload-dense host.
  • Whole disks, not partitions or thin slices of an existing array. OSDs want dedicated physical disks. You’re not adding Ceph on top of drives you’re already using for local VM storage, you’re setting drives aside specifically for it, which is a real capacity commitment on hardware that may not have drive bays to spare.
  • Losing an OSD triggers a rebalance, and rebalances are expensive. When a disk or node drops out, Ceph starts moving data around to restore full redundancy, and that background traffic competes with your actual VM workloads for both network and disk I/O. On an undersized network this can turn a single drive failure into a period of degraded performance across everything running on the cluster, right when you can least afford it.

None of this is a knock on Ceph, it behaves exactly as documented. The issue is that home labs often assemble a 3-node cluster from whatever spare hardware exists, on the same flat 1GbE network as everything else, and expect it to behave like a purpose-built storage appliance.

How it compares to the lighter options

If you’ve read the case for clustering in general, ZFS storage replication and an external NAS over NFS or iSCSI are the two lighter shared-storage paths that don’t ask nearly as much of your hardware:

  • ZFS replication is asynchronous, meaning a node failure between replication runs can lose whatever changed in that window, typically seconds to a couple of minutes depending on your schedule. Ceph’s writes are synchronous and replicated before acknowledgment, closing that gap entirely. If your workloads can tolerate a small window of potential data loss on failover, which most home services can, ZFS replication gets you 80% of the resilience benefit for a fraction of the network and hardware investment.
  • An external NAS over NFS or iSCSI gives you real shared storage without running distributed storage software at all, but it makes that one NAS box a single point of failure for every VM depending on it, and its performance ceiling is whatever that one box’s disks and network link can deliver, no parallelism across multiple nodes.
  • Ceph is the only one of the three that scales storage performance with cluster size (more nodes and OSDs means more aggregate throughput and IOPS, not just more capacity) and has no single point of failure once properly sized. That’s a real advantage for a cluster that’s actually growing, not just one that needs shared storage checked off a list.

When it’s actually worth running at home

  • You already have 3 or more nodes with spare drive bays and a real 10GbE (or faster) network you can dedicate to storage traffic. If any one of those three is missing, you’re better served by ZFS replication or an external NAS until it isn’t.
  • You’re running enough VMs, or VMs demanding enough, that a single NAS’s throughput is a real bottleneck. Ceph’s ability to scale performance with the cluster is the actual reason to reach for it over a NAS, not just the shared-storage requirement on its own.
  • You want hands-on experience with the same storage technology used in real production Kubernetes and OpenStack deployments. That’s a legitimate reason by itself, Ceph skills transfer directly to enterprise work, just go in knowing you’re taking on an operationally heavier system for the learning, not because it’s the easiest way to get shared storage working this weekend.
  • You’re rebuilding a cluster from scratch and can dedicate drives and network from day one, rather than trying to retrofit Ceph onto nodes and a network already committed to something else. Retrofitting Ceph onto an undersized network after the fact is where most of the bad experiences with it come from.

Bottom line

Ceph inside Proxmox is real, well-integrated shared storage that closes the replication-lag gap ZFS replication leaves open, and it’s the only option of the three that gets faster as you add more nodes instead of staying capped at one box’s throughput. What it costs is dedicated fast networking, dedicated whole disks, meaningful RAM and CPU overhead per OSD, and a minimum of 3 nodes to even start behaving the way it’s designed to. For a home cluster running a handful of self-hosted services, ZFS replication or a NAS over NFS usually delivers enough resilience for a fraction of the investment. Reach for Ceph when your cluster’s storage needs are actually outgrowing a single box’s throughput, or when running it is itself the point, not as the default answer the first time someone mentions shared storage.