How to Run LLMs Locally (2026): Zero to Chat in 15 Minutes

Running a model on your own machine means no per-token bills, no rate limits, and no data leaving your computer. Modern small models are genuinely good, and the tooling has become install-and-go. Here is the whole path, honestly including the part where the API is sometimes the better deal.

Transparency: this guide describes tools we run ourselves; model fit numbers come from our live catalog of real GGUF sizes, not marketing pages. No affiliate links.

1 · Check what your hardware can run

Local AI is a memory game. The model's weights must fit in your GPU's VRAM (or a Mac's unified memory) together with the conversation context. As a rough map: 8GB VRAM runs 7-8B models, 12GB runs 12-14B, 16GB reaches 20-24B, 24GB runs the 27-32B class, and 70B needs 48GB or a 64GB+ Mac.

Skip the guesswork: our hardware checker matches your exact GPU or Mac against real model file sizes and shows the command for each fit — or jump straight to your tier: 8GB, 12GB, 16GB, 24GB, Mac 16GB.

2 · The fast path: Ollama

Ollama is a free, open-source runner for Windows, macOS and Linux. It downloads models, picks a sensible quantization, and exposes both a chat window and a local API other apps can use. Install it from ollama.com, then open a terminal:

# first run downloads the model, later runs start instantly
ollama run qwen3:8b

# a coding-focused model instead
ollama run qwen2.5-coder:7b

# see what you have / remove one
ollama list
ollama rm qwen3:8b

That is genuinely all there is to it. Every model in our checker shows its exact ollama run tag when one exists. The local API listens on localhost:11434 and speaks an OpenAI-compatible protocol, so IDE assistants and scripts can point at it.

3 · The visual path: LM Studio

LM Studio is a desktop app (free for personal use) with a model browser, a chat UI and a local server mode. It shines when you want to compare quantizations or models the Ollama library does not carry: search any GGUF from Hugging Face, see the file sizes per quant, and download the one that fits. Our checker links the exact GGUF repository for every model — including the ones without an Ollama tag.

Rule of thumb inside LM Studio: prefer files marked Q4_K_M unless you have memory to spare, and enable full GPU offload if the model fits in VRAM.

4 · Quantization in one minute

Quantization stores model weights with fewer bits. Fewer bits = smaller file = fits in less memory, at a small quality cost. What the labels mean in practice:

• Q4_K_M — the practical default. Around 4.7 bits per weight; the quality loss is barely noticeable for chat and most coding.
• Q5_K_M / Q6_K — a step closer to full quality; take them when the model still fits with context to spare.
• Q8_0 — near-lossless and nearly double the size of Q4; rarely worth it on tight memory.
• IQ4_XS — the squeeze option: smaller than Q4_K_M with slightly lower quality; useful to fit one class up on marginal hardware.

A useful counter-intuition: a bigger model at Q4 usually beats a smaller model at Q8. If you can choose between Qwen3 14B Q4 and Qwen3 8B Q8 at the same memory, take the 14B.

5 · Honest math: local vs the API

Local models cost nothing per token, but hosted APIs for capable small models have become very cheap — our value leaderboard tracks several strong models around $0.06-0.13 blended per million tokens. At light usage, that is pennies per month: buying a GPU for AI alone rarely pays back. Local wins when you need privacy (data never leaves the machine), volume (heavy daily use, batch pipelines), offline work, or tinkering freedom (fine-tunes, uncensored variants, agents hammering a model all day).

Run your own numbers: put your expected token volume into the API cost calculator and compare it with the hardware you would need on the local checker. The honest answer is often "both": a local 8-14B for private drafts and bulk work, an API frontier model for the hard 10% of tasks.

Updated 2026. Model availability and sizes refresh automatically from Hugging Face; tool behaviour verified on the versions current at writing.