Running a large language model on your own hardware is a six-step process, and skipping or misordering any one of the steps is the most common reason people give up on local LLMs before getting something useful running. The steps: size your hardware honestly, pick a quantization level that matches it, install a runtime, pull a model, connect that model to whatever you actually want to use it for, and benchmark before you trust it. None of the six steps is individually hard; the process just has more decision points than most quick-start guides admit.
How it works
A local LLM setup has exactly two moving parts. The first is the model weights — a file, typically quantized to fit your available memory. The second is the inference engine, the software that loads those weights and serves requests, optionally through an API. Nearly everything that goes wrong in a local setup traces back to a mismatch between these two: weights sized for more memory than you have, or an engine that does not expose the interface your actual workflow needs. The six steps below are really about making good decisions on both parts and wiring them together correctly.
Step-by-step setup
- Size your hardware honestly. Check available unified memory on Apple Silicon or VRAM on a discrete GPU. Roughly 16GB gets you small-to-mid models quantized, 32GB opens up mid-size models comfortably, and 64GB and up starts to approach larger open-weight models at usable quality.
- Pick a quantization level. For most people, a 4-bit-class quantization is the sweet spot — meaningfully smaller than full precision with a quality loss most people cannot detect in normal use. Go a step higher only if you notice quality problems specific to your task.
- Install a runtime. Ollama for the simplest command-line path, LM Studio if you want a GUI and would rather not touch a terminal at all. Both handle the underlying inference engine for you.
- Pull a model that matches your hardware tier, not the biggest one you can find. A mid-size model running comfortably beats a large one swapping to disk and crawling.
- Connect it to your actual workflow. Point your editor, script, or existing tooling at the local API endpoint the runtime exposes — most are OpenAI-compatible — instead of only chatting in a terminal window.
- Benchmark with a real workload, not a single toy prompt. Check tokens-per-second and, more importantly, whether output quality holds up on the kind of task you actually plan to use it for.
When local is the right call, and when it is not
| Signal |
Favors local |
Favors a hosted API |
| Usage volume |
High, daily, heavy |
Occasional or light |
| Data sensitivity |
High, cannot leave your machine |
Low to moderate |
| Quality bar |
Good enough for the task |
Needs frontier-level reasoning |
| Ops appetite |
Willing to maintain a runtime |
Wants zero maintenance |
Common mistakes
Picking a model before checking hardware fit, then blaming the tool when it swaps to disk and crawls to a near-unusable speed.
Over-quantizing to save memory and getting noticeably worse output for a task that needed the extra precision the higher-bit format would have kept.
Never connecting the model to a real workflow, so it sits in a chat window, gets used twice, and is abandoned within a week.
Benchmarking once with a trivial prompt instead of testing against real task types before deciding whether local is actually working for you.
FAQ
How much memory do I need to run a decent model locally?
Roughly 16GB for small-to-mid quantized models, 32GB to comfortably run mid-size models, and 64GB or more for larger open-weight models at good quality.
Which quantization level should I start with?
A 4-bit-class quantization for most use cases. Step up only if you notice specific quality problems, and step down only if memory is genuinely the binding constraint.
Do I need a GPU, or does a Mac with unified memory work?
A Mac with sufficient unified memory works well and is often the simplest entry point, since it avoids separate VRAM constraints. A discrete GPU still wins on raw throughput for heavier serving needs.
How do I connect a local model to my code editor?
Point the editor or tool at the OpenAI-compatible endpoint most local runtimes expose. Most modern editor integrations and coding tools support this without extra configuration.
Where to go next