{"post":{"slug":"self-host-an-llm-2026","title":"Self-host an LLM in 2026: hardware, software, and when it makes sense","excerpt":"Self-hosting a language model in 2026 is practical for teams with privacy requirements or high token volume. Here is what it actually costs and takes.","category":"code","date":"April 28, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">S</span>elf-hosting a language model stopped being a research exercise in 2025. The tooling is mature, the open-weight models are capable, and the break-even economics are well-understood. In 2026, the decision is a straightforward engineering and cost analysis — not a feasibility question.\n\n## What changed in 2026\n\n- **Open-weight models closed the capability gap significantly.** Llama 4 Scout (3B) and Mistral Small 3.1 (22B) handle most production tasks that needed GPT-4 in 2023.\n- **vLLM reached production maturity** with continuous batching, PagedAttention, and OpenAI-compatible API — it is the standard inference server for anything beyond dev use.\n- **Ollama became the default local runtime**, with one-command model downloads and a built-in API.\n- **Cloud GPU spot pricing dropped ~40%.** A100s and H100s on spot are now accessible for smaller teams.\n\n## When to self-host\n\n| Driver | Details |\n| --- | --- |\n| Data privacy / compliance | HIPAA, GDPR, financial regs require data not leaving your infra |\n| High token volume | > 50M tokens/month — API costs exceed infra costs |\n| Latency control | Sub-50ms first-token latency requires co-located inference |\n| Custom fine-tuned models | Your adapter, your server — no vendor can run your weights |\n| Offline / air-gapped | Industrial, defence, or edge deployments |\n\n## Hardware sizing guide\n\n| Model size | Minimum VRAM | GPU option | Throughput |\n| --- | --- | --- | --- |\n| 3B (Q8) | 6 GB | RTX 3060 | ~120 tok/s |\n| 7B (Q4) | 8 GB | RTX 4060 | ~80 tok/s |\n| 7B (FP16) | 16 GB | RTX 4080 | ~100 tok/s |\n| 22B (Q4) | 16 GB | RTX 4090 | ~40 tok/s |\n| 70B (Q4) | 40 GB | 2x RTX 4090 or A100 | ~25 tok/s |\n\nVRAM is the binding constraint. CPU inference (llama.cpp) works for dev/testing but is 10–20× slower than GPU.\n\n## Ollama: local and development\n\n```bash\n# Install\ncurl -fsSL https://ollama.ai/install.sh | sh\n\n# Pull and run a model\nollama pull llama4:scout\nollama run llama4:scout\n\n# Use the OpenAI-compatible API\ncurl http://localhost:11434/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"llama4:scout\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Summarise this PR.\"}]\n  }'\n```\n\nOllama handles model storage, GPU allocation, and the API layer. It is the fastest path from zero to a running model.\n\n## vLLM: production inference\n\n```bash\npip install vllm\n\n# Start a production server (OpenAI-compatible)\nvllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct \\\n  --tensor-parallel-size 2 \\\n  --max-model-len 32768 \\\n  --port 8000\n```\n\nvLLM's continuous batching processes multiple concurrent requests against a single model instance — throughput is 3–10× higher than naive sequential inference.\n\n## Cost comparison\n\nAssume a 7B model running on a cloud A10G spot ($0.80/hr):\n\n- **Capacity:** ~300 requests/hour at 500 tokens each = 150k tokens/hr\n- **Cost per 1M tokens:** ~$5.30\n- **Claude Sonnet API:** ~$3/1M input + $15/1M output\n- **Break-even:** depends on output-heavy workloads; roughly 40–60M tokens/month\n\nAt 10M tokens/month, the API is cheaper. At 200M tokens/month, self-hosting is significantly cheaper. Model your specific token ratio.\n\n## How to start\n\n1. **Start with Ollama locally** — validate that the model quality meets your requirement before buying hardware.\n2. **Pick the smallest model that passes your evals** — a 7B model at 80 tok/s beats a 70B model at 5 tok/s for user-facing latency.\n3. **Use vLLM for production** — continuous batching makes a single GPU handle far more concurrent users.\n4. **Set up monitoring** — token/sec, queue depth, GPU utilisation, and error rate.\n5. **Run the OpenAI-compatible API** — all your existing client code works with `base_url=\"http://your-server:8000/v1\"`.\n\n## Common mistakes\n\n**Underestimating VRAM.** A model that \"fits\" in VRAM at 4-bit quantisation with no KV cache will OOM under real traffic. Leave 20–30% headroom.\n\n**Running inference on CPU.** Unless you have no GPU, llama.cpp on CPU is only suitable for exploration.\n\n**No autoscaling.** A single GPU server with no overflow route will fail under spike traffic. Either provision a buffer or have an API fallback.\n\n**Ignoring model updates.** Open-weight model quality improves quarterly. Establish a pipeline to re-evaluate and update models.\n\n**Quantising too aggressively.** Q2 and Q3 quantisation degrades quality noticeably. Q4 is the practical floor; Q8 is better for quality-sensitive tasks.\n\n## What to skip\n\n- **Self-hosting a 70B model** when your task evals show a 7B fine-tuned model is equivalent.\n- **Building your own inference server** from PyTorch — vLLM and TGI exist and are excellent.\n- **Hosting open-weight models for tasks where a frontier model provides a step-change in quality** — the gap matters for reasoning and code.\n\n## FAQ\n\n**Can I self-host Claude or GPT-4?**\nNo — these are closed-weight models. You can only self-host open-weight models like Llama, Mistral, Phi, and Gemma.\n\n**What is the minimum GPU for a useful self-hosted model?**\nAn RTX 3060 (12 GB VRAM) can run Llama 4 Scout 3B at quality sufficient for many tasks. It is a reasonable starting point.\n\n**How do I handle model updates without downtime?**\nRun two instances, warm up the new model, then shift traffic. vLLM supports hot-swapping with lora adapters.\n\n**Is llama.cpp worth it without a GPU?**\nFor personal use and experimentation: yes. For production user-facing inference: no — latency will be unacceptable.\n\n## Where to go next\n\n- [Best GPU for AI in 2026](/blog/best-gpu-for-ai-2026)\n- [Fine-tune a small model in 2026](/blog/fine-tune-a-small-model-2026)\n- [Best AI API providers in 2026](/blog/best-ai-api-providers-2026)","hue":321,"metaTitle":"Self-Host an LLM in 2026: Hardware, Stack, and Cost Guide","metaDescription":"Practical 2026 guide to self-hosting a language model. Hardware sizing, Ollama vs vLLM, cost breakeven, and when to choose API over self-hosted.","keywords":["self host llm 2026","run llm locally 2026","ollama vs vllm 2026","self hosted ai 2026","llm inference server 2026","run llama locally 2026","llm hardware requirements 2026","private llm deployment","open source llm hosting","llm gpu requirements 2026"],"tldr":["**Self-hosting makes financial sense above ~50M tokens/month** — below that, API pricing wins on total cost.","**Ollama and vLLM are the two main runtimes** — Ollama for local/dev, vLLM for production throughput.","**A single RTX 4090 runs a 7B model at 60–80 tokens/sec** — enough for dozens of concurrent users.","**Data residency and compliance are the non-financial drivers** — healthcare, legal, and finance often must self-host.","**Skip:** self-hosting a 70B model if your queries are simple enough for a 7B to answer correctly."],"readTime":"4 min read","wordCount":901},"related":[{"slug":"self-hosting-ai-models-2026","title":"Self-hosting AI models in 2026: when it beats the API","excerpt":"Open models got good enough to self-host for real work in 2026. Here is when it actually saves money or protects privacy, and when the API still wins.","category":"tech","date":"March 3, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":248},{"slug":"best-local-ai-models-2026","title":"Best local AI models in 2026: top open-weight models you can run on your hardware","excerpt":"Running AI models locally gives you privacy, offline access, and zero per-token costs. This guide ranks the best open-weight models worth running in 2026.","category":"ai","date":"April 24, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":202},{"slug":"how-to-self-host-saas-2026","title":"How to self-host a SaaS in 2026 (the full stack, the costs, the catches)","excerpt":"Self-hosting a small SaaS is cheaper than ever in 2026. Here is the full stack, real monthly costs, and the things people forget.","category":"code","date":"January 27, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":305}],"sameCategory":[{"slug":"bazel-build-explained-2026","title":"Bazel Explained: What Hermetic Builds Actually Cost","excerpt":"Bazel delivers reproducible, precisely cached builds and demands that you describe everything. Here is what you get, what it costs, and who should say no.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":145},{"slug":"graphql-federation-2026","title":"GraphQL Federation: One Graph, Many Teams","excerpt":"Federation composes independently-owned subgraphs into a single API. Here is what it solves, the operational cost it adds, and when a simpler gateway is enough.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":300},{"slug":"hermetic-builds-explained-2026","title":"Hermetic Builds Explained: Same Inputs, Same Output","excerpt":"A hermetic build depends only on declared inputs. Here is what that property gives you, the common leaks that break it, and how to move toward it incrementally.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":130},{"slug":"monorepo-tooling-compared-2026","title":"Monorepo Tooling Compared: What You Actually Need","excerpt":"Monorepo tools solve task orchestration and caching, not repository layout. Here is what each tier provides and how to tell when you have outgrown the simple option.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":150}],"newer":{"slug":"retirement-catch-up-contributions-2026","title":"Catch Up Contributions 2026: How the Age Tiers Work","excerpt":"Turning 50 unlocks extra retirement savings room. Here is how the 2026 catch-up tiers work, the new Roth rule for high earners, and what to skip.","category":"finance","date":"April 28, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":52},"older":{"slug":"sveltekit-vs-nextjs-2026","title":"SvelteKit vs Next.js in 2026: Which Should You Choose","excerpt":"A plain-language 2026 comparison of SvelteKit and Next.js, covering bundle size, hiring, ecosystem, and how to pick the right one without the hype.","category":"code","date":"April 28, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":138}}