Every other service in this stack either works or it doesn’t, and when it doesn’t, you get an error message. Email is different. You can run a flawless mail server, correctly configured down to the last header, and have every message you send quietly vanish into someone else’s spam folder, or get silently dropped before it even gets that far, with zero indication anything went wrong on your end. That one property, failure that’s invisible by default, is why self-hosting email has a reputation as the homelab project that eats the most time for the least reliable payoff. It’s worth understanding why before picking Mailcow or Mail-in-a-Box, because the software choice matters a lot less than most guides make it sound.
Why email isn’t like the rest of your homelab
Every other self-hosted service you run, Jellyfin, Nextcloud, Vaultwarden, only has to satisfy you and the people you explicitly give access to. Nobody else’s opinion of your Jellyfin server matters. Email is the opposite: every message you send has to satisfy spam filters run by Google, Microsoft, and every other mail provider on the receiving end, and you have zero visibility into their rules and zero say in how they’re applied. You’re not running a private service anymore, you’re trying to get a letter accepted by a postal system that treats strangers as guilty until proven innocent.
The specific thing that makes this brutal for a homelab is IP reputation. Residential IP ranges are, as a category, treated as a major spam source by every large mail provider, because that’s historically where most spam has come from. It doesn’t matter if your SPF record is perfect, your DKIM signature validates, and your DMARC policy is textbook. If your outbound IP sits in a residential ASN block, a meaningful share of your mail to Gmail and Outlook addresses gets silently dropped or spam-foldered anyway, purely on IP reputation, before the receiving server even looks at your headers. Many residential ISPs also block outbound port 25 entirely by default, specifically to stop compromised home machines from spamming, which means some home connections can’t send to another mail server directly no matter what you run.
None of this is a bug in Mailcow or Mail-in-a-Box. It’s true of literally any mail server running from a home IP, including a perfectly configured one. This is the part most “how to self-host email” tutorials skip, because the software setup is the interesting, demo-able part and the deliverability reality only shows up weeks later when you notice half your sent mail never arrived anywhere.
What you actually need to get right regardless of software
Three DNS records do almost all of the authentication work, and you need all three, not just one:
- SPF declares which servers are allowed to send mail for your domain. Receiving servers check the sending IP against this record and treat a mismatch as a strong spam signal.
- DKIM cryptographically signs outgoing mail with a private key, with the matching public key published in DNS, so the receiving server can verify the message wasn’t altered in transit and actually came from a server you authorized.
- DMARC tells receiving servers what to do when SPF or DKIM fails, reject, quarantine, or do nothing, and gives you a reporting address so you can see who’s sending mail claiming to be your domain.
On top of those, you need a matching reverse DNS (PTR) record for your outbound IP that resolves back to a hostname matching your mail server’s identity, and that PTR record has to be set by whoever controls the IP’s reverse zone, your ISP for a home connection, your hosting provider for a VPS. A lot of residential ISPs simply won’t set a custom PTR record for you at all, which caps your deliverability ceiling before you’ve written a single line of mail server config.
Get all of this right and you’ve done the table stakes. You still haven’t solved the IP reputation problem, which is the one that actually determines whether Gmail and Outlook accept your mail, and which has nothing to do with anything you configured.
Mailcow: the full stack, if you’re going to do this
Mailcow is a Docker Compose based stack that bundles Postfix (SMTP), Dovecot (IMAP), Rspamd (spam filtering), SOGo (webmail and calendaring), and ClamAV (antivirus) behind a single admin UI, with Let’s Encrypt handled automatically. It’s the more complete and more actively developed of the two mainstream self-hosted options, and the admin panel genuinely covers day-to-day mailbox and alias management without needing to touch config files by hand. The tradeoff is the usual Docker Compose stack tradeoff: a dozen-plus containers to keep updated together, a nontrivial RAM footprint (budget for multiple gigabytes at idle, more under real load, not something you’re running alongside everything else on a 2GB LXC), and updates that occasionally require reading the release notes carefully rather than blindly pulling latest.
If you want a self-hosted mail stack that feels like a product rather than a pile of configs you assembled yourself, this is the one to run.
Mail-in-a-Box: narrower scope, less to tune
Mail-in-a-Box takes the opposite philosophy: it wants a dedicated, freshly installed server (historically Ubuntu LTS) and installs everything itself as a single opinionated package, Postfix, Dovecot, spam filtering, webmail, and a built-in DNS server for your domain’s records. There’s deliberately very little to configure, which is the point: it’s aimed at someone who wants working email with sane defaults and doesn’t want to tune Rspamd rules or pick between SOGo and Roundcube. The cost of that simplicity is flexibility. It expects to own the whole box, doesn’t fit cleanly alongside other services the way a Compose stack can live next to your other containers, and gives you less room to grow into more advanced filtering or multi-domain setups later without working against its assumptions.
Pick Mail-in-a-Box if you want the smallest number of decisions to make and are willing to dedicate a box to it. Pick Mailcow if you want more capability and don’t mind the maintenance that comes with a real multi-container stack.
The pattern most homelabbers actually land on
Given the IP reputation problem, the workable middle ground isn’t “self-host everything” or “give up and use Gmail,” it’s splitting the job in two. Self-host Mailcow or Mail-in-a-Box at home for storage, IMAP, webmail, and your own spam filtering, the parts that are entirely under your control and genuinely benefit from being self-hosted. Then route outbound sending through a dedicated transactional relay service instead of sending directly from your home IP. Your mail server hands outgoing messages to the relay over an authenticated connection, and the relay’s own, already-warmed-up IP reputation handles actual delivery to Gmail and Outlook. You keep ownership of your mailbox data and your own spam filtering on the way in; you borrow someone else’s IP reputation on the way out, which is the one piece of this that a home connection structurally cannot win on its own.
This isn’t a cop-out, it’s the same architecture a lot of legitimate businesses use even when they’re not running anything at home: owning the mailbox, outsourcing the sending reputation. If outbound port 25 is blocked by your ISP, this stops being optional and becomes the only path that works at all.
Where self-hosting email is worth it anyway
If your goal is learning how SMTP, IMAP, and the whole authentication chain actually work, running Mailcow end to end, including the parts that will fight you, is genuinely one of the better hands-on networking exercises available in this hobby. It’s also reasonable for a low-stakes secondary domain, a catch-all for signups you don’t care deeply about, or an inbound-only archival mailbox that doesn’t need to send anything out. What doesn’t make sense is betting a domain you rely on for anything important, a business, a primary personal address, account recovery, on a from-scratch home mail server with no fallback plan for the deliverability problem above.
Bottom line
The software decision here is the easy part: Mailcow if you want a full, actively developed stack and don’t mind the container maintenance, Mail-in-a-Box if you want a dedicated box with minimal decisions to make. The hard part, the one that actually determines whether anyone receives what you send, is IP reputation and port 25 access, and no amount of correct SPF, DKIM, and DMARC configuration fixes that from a residential connection. Self-host the mailbox if you want to own your data. Relay the outbound through a dedicated sending service unless you enjoy debugging mail that silently disappeared, because on this one piece of infrastructure, “it’s self-hosted” and “it’s reliable” are not the same promise.