Code · June 8, 2026

Self-host an LLM in 2026: hardware, software, and when it makes sense

Self-hosting a language model in 2026 is practical for teams with privacy requirements or high token volume. Here is what it actually costs and takes.

By ByteLedger Team

Self-hosting a language model stopped being a research exercise in 2025. The tooling is mature, the open-weight models are capable, and the break-even economics are well-understood. In 2026, the decision is a straightforward engineering and cost analysis — not a feasibility question.

What changed in 2026

Open-weight models closed the capability gap significantly. Llama 4 Scout (3B) and Mistral Small 3.1 (22B) handle most production tasks that needed GPT-4 in 2023.
vLLM reached production maturity with continuous batching, PagedAttention, and OpenAI-compatible API — it is the standard inference server for anything beyond dev use.
Ollama became the default local runtime, with one-command model downloads and a built-in API.
Cloud GPU spot pricing dropped ~40%. A100s and H100s on spot are now accessible for smaller teams.

When to self-host

Driver	Details
Data privacy / compliance	HIPAA, GDPR, financial regs require data not leaving your infra
High token volume	> 50M tokens/month — API costs exceed infra costs
Latency control	Sub-50ms first-token latency requires co-located inference
Custom fine-tuned models	Your adapter, your server — no vendor can run your weights
Offline / air-gapped	Industrial, defence, or edge deployments

Hardware sizing guide

Model size	Minimum VRAM	GPU option	Throughput
3B (Q8)	6 GB	RTX 3060	~120 tok/s
7B (Q4)	8 GB	RTX 4060	~80 tok/s
7B (FP16)	16 GB	RTX 4080	~100 tok/s
22B (Q4)	16 GB	RTX 4090	~40 tok/s
70B (Q4)	40 GB	2x RTX 4090 or A100	~25 tok/s

VRAM is the binding constraint. CPU inference (llama.cpp) works for dev/testing but is 10–20× slower than GPU.

Ollama: local and development

# Install
curl -fsSL https://ollama.ai/install.sh | sh

# Pull and run a model
ollama pull llama4:scout
ollama run llama4:scout

# Use the OpenAI-compatible API
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama4:scout",
    "messages": [{"role": "user", "content": "Summarise this PR."}]
  }'

Ollama handles model storage, GPU allocation, and the API layer. It is the fastest path from zero to a running model.

vLLM: production inference

pip install vllm

# Start a production server (OpenAI-compatible)
vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --port 8000

vLLM's continuous batching processes multiple concurrent requests against a single model instance — throughput is 3–10× higher than naive sequential inference.

Cost comparison

Assume a 7B model running on a cloud A10G spot ($0.80/hr):

Capacity: ~300 requests/hour at 500 tokens each = 150k tokens/hr
Cost per 1M tokens: ~$5.30
Claude Sonnet API: ~$3/1M input + $15/1M output
Break-even: depends on output-heavy workloads; roughly 40–60M tokens/month

At 10M tokens/month, the API is cheaper. At 200M tokens/month, self-hosting is significantly cheaper. Model your specific token ratio.

How to start

Start with Ollama locally — validate that the model quality meets your requirement before buying hardware.
Pick the smallest model that passes your evals — a 7B model at 80 tok/s beats a 70B model at 5 tok/s for user-facing latency.
Use vLLM for production — continuous batching makes a single GPU handle far more concurrent users.
Set up monitoring — token/sec, queue depth, GPU utilisation, and error rate.
Run the OpenAI-compatible API — all your existing client code works with base_url="http://your-server:8000/v1".

Common mistakes

Underestimating VRAM. A model that "fits" in VRAM at 4-bit quantisation with no KV cache will OOM under real traffic. Leave 20–30% headroom.

Running inference on CPU. Unless you have no GPU, llama.cpp on CPU is only suitable for exploration.

No autoscaling. A single GPU server with no overflow route will fail under spike traffic. Either provision a buffer or have an API fallback.

Ignoring model updates. Open-weight model quality improves quarterly. Establish a pipeline to re-evaluate and update models.

Quantising too aggressively. Q2 and Q3 quantisation degrades quality noticeably. Q4 is the practical floor; Q8 is better for quality-sensitive tasks.

What to skip

Self-hosting a 70B model when your task evals show a 7B fine-tuned model is equivalent.
Building your own inference server from PyTorch — vLLM and TGI exist and are excellent.
Hosting open-weight models for tasks where a frontier model provides a step-change in quality — the gap matters for reasoning and code.

FAQ

Can I self-host Claude or GPT-4? No — these are closed-weight models. You can only self-host open-weight models like Llama, Mistral, Phi, and Gemma.

What is the minimum GPU for a useful self-hosted model? An RTX 3060 (12 GB VRAM) can run Llama 4 Scout 3B at quality sufficient for many tasks. It is a reasonable starting point.

How do I handle model updates without downtime? Run two instances, warm up the new model, then shift traffic. vLLM supports hot-swapping with lora adapters.

Is llama.cpp worth it without a GPU? For personal use and experimentation: yes. For production user-facing inference: no — latency will be unacceptable.