An NPU, or neural processing unit, is a chip designed to do one thing efficiently: run the matrix multiplications that neural networks are built from. Unlike a CPU, which handles arbitrary instructions one after another, or a GPU, which was built for parallel graphics math and later repurposed for AI, an NPU is purpose-built silicon for the multiply-accumulate operations that dominate neural network inference. That narrow focus is what makes it fast and power-efficient for the one job it does.
What changed in 2026
- NPUs became a baseline spec, not a differentiator. Every current-generation laptop and phone chipset — Apple Silicon, Qualcomm Snapdragon, Intel Core Ultra, AMD Ryzen AI — now ships an NPU, largely to support on-device small language models and image processing without draining the battery.
- Software finally caught up to the hardware. Frameworks like ONNX Runtime, Core ML, and DirectML matured their NPU backends, so more workloads actually route to the NPU instead of silently falling back to CPU or GPU.
- TOPS (trillions of operations per second) became a marketed but noisy metric. Vendors compete on TOPS numbers that are not directly comparable across architectures, precision formats, or memory subsystems — treat them as a rough signal, not a benchmark.
How an NPU actually works
Neural networks are, at the arithmetic level, mostly repeated multiply-and-add operations across large matrices — the same operation, over and over, at massive scale. A CPU executes this inefficiently because it is optimized for varied, branching logic. A GPU does much better because it has thousands of simple cores built for parallel math. An NPU goes further: it strips out almost everything a general-purpose processor needs and replaces it with dedicated matrix-multiply units, low-precision arithmetic (INT8, INT4, FP8), and on-chip memory positioned to keep those units fed without constantly hitting system RAM.
The result is an order-of-magnitude improvement in operations-per-watt for neural network workloads, at the cost of being nearly useless for anything else. You cannot run a spreadsheet or a web browser on an NPU.
NPU vs GPU vs CPU
| Chip type |
Built for |
Strength |
Weakness |
| CPU |
General-purpose sequential logic |
Flexibility, branching code |
Slow and power-hungry for matrix math |
| GPU |
Parallel graphics, then repurposed for AI |
High raw throughput, mature software stack |
High power draw, needs external memory bandwidth |
| NPU |
Neural network inference specifically |
Best operations-per-watt for inference |
Narrow use case, weaker training support |
What NPUs are actually used for
On-device NPUs mostly handle inference for models that already fit comfortably in a phone or laptop's memory budget: keyboard prediction, camera scene detection, background blur in video calls, voice transcription, and increasingly, small language models running locally for tasks like summarization or basic assistants. If you have used a phone camera that recognizes a face in real time without a network round-trip, that was likely an NPU. See our explainer on small language models for why on-device NPUs and compact models arrived together — one made the other practical.
NPUs are largely absent from large-scale model training. Training a frontier model still requires GPU or TPU clusters with high-bandwidth interconnects, because training needs to move gradients across thousands of chips, not just run a fixed forward pass on one device.
Where NPUs fall short
The biggest limitation is not raw speed — it is software support. A model has to be converted, quantized, and compiled for a specific NPU's instruction set before it runs there at all, and that toolchain has historically lagged behind GPU support. A second limitation is memory: on-device NPUs typically share RAM with the rest of the system, so larger models still get bottlenecked, which is part of why understanding parameter counts matters when deciding what can realistically run on-device versus in the cloud.
FAQ
Does my phone already have an NPU?
Almost certainly, if it was released in the last few years. Apple calls it the Neural Engine, Qualcomm calls it Hexagon, and Google's Tensor chips have their own equivalent block.
Can an NPU replace a GPU for AI work?
Not for training or for running large models — it complements a GPU rather than replacing it. NPUs shine for lightweight, always-on, battery-sensitive inference.
Why do TOPS numbers vary so much between vendors?
TOPS is measured at different precisions (INT8 vs INT4) and under different assumptions, so a higher number does not always mean faster real-world performance. Check independent benchmarks for the workload you actually care about rather than trusting the spec sheet alone.
Will NPUs eventually handle model training too?
Some specialized on-device fine-tuning is emerging, but full-scale training remains a GPU and TPU job for the foreseeable future — the interconnect and memory requirements are a different problem than inference.
Where to go next