The hardware reality, before anything else

The single number that determines what you can run is VRAM, not CPU, not system RAM, not disk speed. Large language models are a pile of weights that has to sit in GPU memory for inference to run at a usable speed, and the model’s file size on disk is roughly what it needs in VRAM to load. A model quantized to 4-bit (the common default for local use, trading a small amount of accuracy for a large drop in size) that’s listed as needing “8GB” means an 8GB card is close to the minimum, not a comfortable margin, once you account for context window and any other GPU load on the box.

Realistic tiers, assuming 4-bit quantized models and a dedicated GPU:

• 8GB VRAM (e.g. an older mid-range card): small models in the 7-8B parameter range, usable for chat, summarization, and simple drafting. Fine, but you’ll notice the ceiling on anything requiring real reasoning depth.
• 12-16GB VRAM: mid-size models (roughly 13-14B), a meaningfully better default for general use without stepping up to serious hardware.
• 24GB+ VRAM: larger models (30B and up, depending on quantization) start to feel closer to a genuinely capable general-purpose assistant, still short of frontier cloud models but a real step up.

CPU-only inference works and is how most people start, it’s just slow, expect real time-to-first-token delays and low tokens-per-second, tolerable for an occasional query, painful for anything conversational or high-volume. If a GPU is available on the box (even an older consumer card you weren’t otherwise using), pass it through, the difference in usability is not subtle.

Ollama: the model runtime

Ollama is the piece that actually loads and runs models. It wraps llama.cpp under the hood, handles model downloading and quantization variants through a simple pull-based interface (ollama pull <model>), and exposes an OpenAI-compatible-ish local API on port 11434 that other tools, including Open WebUI, talk to. You don’t hand-manage GGUF files or compile anything yourself unless you want to, Ollama’s model library covers the popular open-weight families and updates as new releases land.

Running it in a homelab context, a few things matter more than the quickstart guides mention:

• Run it as its own service, in its own LXC or VM, not bolted onto whatever box happens to have a GPU. Model downloads are large (multi-gigabyte per model, and you’ll likely keep more than one around) and inference is CPU/GPU-intensive in bursts, isolating it means a heavy inference run doesn’t compete with anything else you’re running.
• GPU passthrough is the real setup cost. If Ollama is landing in a VM, that means proper PCIe passthrough with IOMMU groups and vfio-pci binding. If it’s landing in an LXC instead, that’s the simpler device-passthrough path (/dev/nvidia* or /dev/dri bound into the container), covered in more depth in this site’s article on Proxmox GPU passthrough for VMs vs LXCs, worth reading before you commit to either approach here.
• Set OLLAMA_HOST=0.0.0.0 deliberately, not by accident, if you want other machines on your network reaching it. The default binds to localhost only, which is the safer starting point until you’ve decided who else should be able to hit the API.
• Model storage adds up fast. A handful of model variants at a few gigabytes each turns into real disk usage quickly, point Ollama’s model directory at storage you’ve actually planned for, not whatever’s left on the boot volume.

Open WebUI: the interface people actually want

Ollama by itself is a command-line tool and a local API, functional but not something you’re going to hand a family member. Open WebUI sits in front of it (or in front of any OpenAI-compatible backend, including actual cloud APIs if you want one interface for both) and gives you a ChatGPT-style web interface: conversation history, multiple users with their own logins, model switching per conversation, and document upload for basic retrieval-augmented generation against your own files.

Deploy it as its own container, pointed at Ollama’s API address, and put it behind the same reverse proxy and TLS setup you’re already running for everything else self-hosted on your network, this is not a service you want to expose without at least basic auth in front of it if there’s any chance of it reaching beyond your LAN. The multi-user support is genuinely useful the moment more than one person in the house wants to use it, each account gets its own history and doesn’t see anyone else’s conversations by default.

Worth knowing going in: the document upload / RAG feature is convenient for quick lookups against a handful of files, but it’s not a substitute for a real vector database and retrieval pipeline if you’re trying to build something more serious against a large document set. Treat it as “ask questions about a PDF I just uploaded,” not as the foundation of a knowledge base.

Picking a model without overpromising

This is the part where honesty matters most. A locally-run open-weight model, even a good one, is not a drop-in replacement for a frontier cloud model like Claude or GPT for hard reasoning, long-context work, or anything where being wrong has a real cost. What it’s genuinely good for at homelab scale:

• Summarizing your own notes, emails, or logs without sending them anywhere
• Drafting text you’re going to edit anyway
• Simple classification and extraction tasks run in bulk, where per-token cloud pricing would add up
• A privacy-sensitive assistant for anything you specifically don’t want touching a third party’s servers
• Learning how these systems actually work, which is reason enough on its own for a lot of homelabbers

If you’re weighing this against paying for a cloud subscription, be clear-eyed about the actual tradeoff: you’re not getting equivalent capability for free, you’re trading capability for privacy, offline availability, and no per-request cost on the tasks where a smaller model is genuinely good enough. For anything that actually needs the strongest available reasoning, that’s still a cloud API call, and pretending your homelab box replaces that is the kind of overclaim that gets homelabbers a bad reputation with people who tried it once and got a worse answer than they expected.

Where this fits in a real homelab

If you already have a GPU-capable Proxmox host, or you’re weighing whether to reinstall an older card headless for exactly this purpose, this is one of the more satisfying uses for spare GPU hardware that would otherwise sit idle. Size your model choice to the VRAM you actually have rather than chasing the largest model that technically fits, a smaller model that responds in a couple of seconds beats a larger one that makes you wait, for most day-to-day use. Start with Ollama and a single mid-size model, add Open WebUI once you’ve confirmed the API responds the way you expect, and resist the urge to load five different models “just in case,” disk and load time both add up faster than it looks like they will.

Bottom line

Ollama gives you a model runtime that pulls, quantizes, and serves open-weight models with minimal ceremony. Open WebUI gives you the chat interface people actually want to use, with multi-user support and basic document Q&A on top. Together they’re a genuinely useful addition to a homelab, as long as you’re running them for what local models are actually good at, privacy, cost control on bulk tasks, and offline availability, rather than expecting them to match a frontier cloud model on hard problems. Size the hardware to the model tier you want, isolate the service in its own container, and treat the result as a capable assistant for the right jobs, not a full replacement for the AI you’re already paying for.