Running a quantized model locally comes down to four decisions done in the right order: how much memory you actually have, which bit width fits that budget, which tool you want to run it with, and how you verify the result still does what you need. Skip the memory step and you will pick a file that swaps to disk and crawls. Skip the verification step and you may not notice a quality regression until it shows up in production. This guide walks through the process in order, rather than starting from a specific tool or file format.
The core idea
A quantized model file bundles a specific bit width (commonly 4-bit, 5-bit, or 8-bit) with a specific format (GGUF for llama.cpp-based tools, GPTQ or AWQ for GPU-focused serving stacks). The file size roughly tracks the bit width: a 7 billion parameter model that needs about 14GB at full 16-bit precision typically shrinks to somewhere around 4-5GB at 4-bit. Your available memory, not your ambition, should decide which of these you download first.
Step-by-step: getting a quantized model running
- Measure your actual available memory. For a GPU setup, check usable VRAM after your OS and other processes. For Apple Silicon or unified-memory laptops, check available system memory, since it is shared with everything else running.
- Pick a parameter count that leaves headroom. As a rough guide, leave at least 15-20 percent of memory free beyond the model file size for context, the operating system, and any other loaded process; a model that just barely fits at idle will fail once you load a long conversation.
- Choose a bit width. Start with a 4-bit variant such as Q4_K_M in GGUF naming, or an equivalent 4-bit GPTQ/AWQ file if using a GPU-focused server. This is the most common sweet spot between size and quality for general use.
- Pick a runtime that matches your workflow. Ollama and LM Studio wrap the setup in a simple interface and are the fastest way to get a model running with minimal configuration. Raw llama.cpp gives more control over build flags and hardware-specific optimizations, at the cost of more manual setup.
- Download from a source that documents the quantization method. Community-hosted quantized files vary in how carefully outlier weights were protected during conversion; prefer sources that specify the exact method and calibration data used.
- Load the model and run a quick smoke test. A handful of prompts representative of your actual use case, run through both the quantized model and, if possible, the full-precision version, will surface obvious regressions immediately.
- Adjust bit width based on what you see. If output quality looks noticeably worse on tasks that matter, such as multi-step reasoning or code, step up to a 5-bit or 8-bit variant, or reduce context length to free memory for a higher-precision model instead.
- Re-test after any change to model, prompt template, or runtime version. Small version bumps in the runtime or a changed chat template can shift output quality independently of the quantization itself.
Choosing a quant level: a quick reference
| Quant level |
Typical use case |
Tradeoff |
| Q8 / 8-bit |
Near full-precision quality needed, memory is not tight |
Largest file among quantized options, closest to reference quality |
| Q5_K_M |
General use with more headroom for quality than Q4 |
Good middle ground, moderate size savings |
| Q4_K_M |
Most general-purpose local use |
The common practical default: solid quality, real size savings |
| Q3 and below |
Severe memory constraints only |
Noticeable quality loss, worth avoiding unless memory truly forces it |
Common mistakes
- Downloading the biggest model that technically fits at idle. Leaving no memory headroom for context and background processes leads to swapping and a sharp drop in tokens per second.
- Ignoring the calibration data used for GPTQ or AWQ conversion. A quantized file calibrated on unrelated data can perform worse on your specific domain than one calibrated closer to your use case.
- Never comparing against the full-precision model. Without a baseline, it is easy to mistake ordinary model limitations for quantization-specific damage, or the reverse.
- Assuming one runtime is strictly better than another. Ollama, LM Studio, and llama.cpp differ mainly in convenience and configurability, not in the underlying quality of a given quantized file.
FAQ
What quant level should a beginner start with?
Q4_K_M in GGUF format through Ollama or LM Studio is the most common starting point: reasonable quality, manageable file sizes, and minimal setup.
Do I need a GPU to run a quantized model?
No. CPU-only inference works, especially for 4-bit models on modern hardware with decent memory bandwidth, though a GPU meaningfully improves speed for larger models.
How is this different from just reading about quantization concepts?
This guide covers the hands-on workflow; for the underlying tradeoffs between size, speed, and accuracy, see our explainer on quantized models.
Can I fine-tune a quantized model once it is running locally?
Yes, this is what QLoRA is built for. See our guide to parameter-efficient fine-tuning for how that process works.
Where to go next
For the conceptual background and the fine-tuning angle on this same workflow, see our guides to quantized models explained, open-source vs closed-source LLMs, and parameter-efficient fine-tuning.