Running a language model locally means no usage fees, no data leaving your machine, and no dependence on an internet connection. In 2026 the open-weight ecosystem is rich enough that local models are a genuinely viable alternative to API calls for many workloads. The challenge is picking the right model for your hardware and use case without drowning in benchmarks. This guide cuts to what actually matters.
What changed in 2026
- The 70B-class quality bar moved up. Llama 3.3 70B and Qwen 2.5 72B both match or exceed GPT-4-level quality on coding and reasoning benchmarks, running on a single high-end consumer GPU (RTX 4090, 24 GB VRAM).
- Small models got genuinely good. Phi-4 (14B) and Qwen 2.5 14B score well above their size class on reasoning and instruction-following, making M2/M3 MacBook Pro viable for serious work.
- Quantisation tooling matured. llama.cpp's GGUF format and the Q4_K_M/Q6_K quantisation levels hit a practical optimum — noticeable quality loss only at Q2 and below.
- Ollama and LM Studio both went v2. Both local serving tools added one-click model switching, concurrent model loading on multi-GPU setups, and REST APIs compatible with the OpenAI spec.
Model comparison by use case
| Model |
Size |
Best for |
Min VRAM / RAM |
Speed (M3 Max) |
| Llama 3.3 70B Q4 |
~40 GB |
General, coding, reasoning |
48 GB unified |
~18 t/s |
| Qwen 2.5 72B Q4 |
~42 GB |
Multilingual, long context |
48 GB unified |
~16 t/s |
| Llama 3.1 8B Q4 |
~5 GB |
Fast general use, edge |
8 GB |
~60–80 t/s |
| Qwen 2.5 14B Q4 |
~9 GB |
Reasoning on laptop |
12 GB |
~40–55 t/s |
| Phi-4 14B Q4 |
~9 GB |
STEM reasoning, low memory |
12 GB |
~45 t/s |
| Codestral 22B Q4 |
~14 GB |
Code generation |
16 GB |
~30 t/s |
| Mistral Nemo 12B |
~8 GB |
Multilingual, fast |
10 GB |
~50 t/s |
| DeepSeek-R1 8B Q4 |
~5 GB |
Chain-of-thought reasoning |
8 GB |
~55 t/s |
Speeds are approximate and vary with prompt length and system load.
How to pick
- Measure your hardware first. Unified memory (Apple Silicon) counts as both CPU and GPU RAM. Discrete VRAM is the binding constraint on Nvidia setups.
- Target 15+ tokens/second for interactive use. Below that, generation lags noticeably during conversations. Batch processing tolerates slower speeds.
- Match the model to the task. Writing tasks need strong instruction-following (Llama 3.3, Qwen). Code completion needs Codestral or a code-tuned Qwen. Multilingual needs Mistral Nemo or Qwen's multilingual weights.
- Start with Q4_K_M quantisation. This is the practical optimum across most models: ~5–10% quality loss vs. full precision, 2–3× smaller file.
- Use Ollama for simplicity, llama.cpp for control. Ollama gives you a working REST endpoint in one command. llama.cpp lets you tune every parameter but requires more CLI comfort.
Common mistakes
Loading a model at 3 tokens/second. A model that fits in memory but barely runs is not usable. Check community benchmarks for your specific hardware before downloading 40 GB.
Ignoring context length settings. Many models default to 2k–4k context in local serving tools even though they support 32k+. Set the context window explicitly or long prompts silently truncate.
Using outdated quantisations. Q4_0 (older format) is noticeably worse than Q4_K_M for the same file size. Always use K-quants when available.
Assuming benchmark == real-world quality. MMLU and HumanEval scores do not always correlate with how a model feels in your specific workflow. Run 20 of your own prompts before committing.
Mixing architectures without checking tokenisers. If you switch from Llama to Qwen, your system prompts may need adjustment — tokenisers and special tokens differ between model families.
What to skip
- The largest model you can physically load. If it runs at 2–4 t/s, use a smaller, faster model and accept the quality trade. Speed matters for real workflows.
- Uncensored fine-tunes unless you have a specific research reason. Base instruct models from Meta, Alibaba, and Mistral are well-aligned and more reliable for production use.
- Running 70B models on systems with less than 48 GB unified/VRAM. Memory bandwidth bottlenecks make the experience painful below that threshold. Stick to 14B–34B class on tighter memory budgets.
FAQ
Can I run a local model on a MacBook?
Yes. An M3 Pro with 36 GB handles 14B models well at 40–50 t/s. An M3 Max or M4 Max with 48–64 GB handles 70B at usable speeds. Older Intel Macs should use 7B–8B models only.
How often do top local models change?
Roughly every 2–3 months a new model displaces the previous leader. Following the Hugging Face Open LLM Leaderboard gives an early signal on new releases.
Do local models support tool calling?
Yes. Llama 3.3, Qwen 2.5, and Mistral all support OpenAI-compatible tool/function calling in both Ollama and llama.cpp server mode.
Is running locally cheaper than using an API?
At low to moderate volume, API costs are lower when you account for your hardware's amortised cost and electricity. Above ~100k–200k tokens/day, local hardware typically wins on cost.
Where to go next
Run DeepSeek locally in 2026 is the step-by-step guide for getting DeepSeek models running on your machine. AI coding agents ranked in 2026 covers how to pair local models with coding agents. AI agents that actually work in 2026 explains how to build reliable agent workflows on top of local or API models.