Spots

VRAM for local LLMs: why memory bandwidth sets your tokens per second

How much VRAM for an LLM is the wrong first question. The better one is how fast that VRAM is, because a local model generating text reads its entire set of weights from memory for every single token. That makes memory bandwidth, in gigabytes per second, the number that decides whether your coding agent types or crawls. This is the bandwidth-first companion to my Local AI Hardware Guide (2026) from March: fewer shopping lists, more of the arithmetic behind them.

Token generation at batch size 1 is memory-bandwidth

Token generation at batch size 1 is memory-bandwidth bound. A rough ceiling is bandwidth ÷ model size in memory: a 10 GB model on a 936 GB/s RTX 3090 tops out around 90 tokens a second.

When the model or its context does not

When the model or its context does not fit, layers spill to DDR5 (about 50 GB/s) over PCIe 4.0 (31.5 GB/s): a 20x bandwidth drop, and 42.5 tok/s becomes 3.8 in the benchmark below.

At 4-bit, weights cost about 5 GB for

At 4-bit, weights cost about 5 GB for 8B, 10 GB for 14B, 20 GB for 32B and 40 GB for 70B, before the KV cache. A 32k-token agent context adds several more gigabytes. The best VRAM per dollar is still a used RTX 3090 24GB, around $650–750, because it pairs 24 GB with a 384-bit bus. Why memory bandwidth, not VRAM size alone, sets LLM speed

A game is compute-bound: the CPU prepares draw

A game is compute-bound: the CPU prepares draw calls, the shaders fill pixels, a faster core means more frames. Autoregressive decoding, the way an LLM writes one token at a time, is the reverse. To compute the next token the GPU streams every weight of the model from VRAM through its compute units, does a little maths on each, and starts again. The cores mostly wait on the memory bus. The back-of-envelope formula:

Real runs land below the ceiling (roughly 60–75

Real runs land below the ceiling (roughly 60–75 tok/s on the 3090, 1.5–2 on DDR5), but the ratio holds: same model, same machine, twenty times slower because the weights moved. That is why a 5.8 GHz Core i9 and a liquid loop do nothing here. The CPU only feeds the GPU; a six-core Ryzen 5 is plenty. GPU memory bandwidth by tier

Bandwidth is set by the memory type and

Bandwidth is set by the memory type and the bus width; the marketing tier says nothing about it. The same 16 GB can be fast or slow: Figures are from Nvidia's spec pages for the RTX 3090 and RTX 4090, plus the PCIe 4.0 and JEDEC DDR5 standards.

The 4060 Ti row explains itself: 288 GB/s

The 4060 Ti row explains itself: 288 GB/s over a 5 GB model is a ~58 tok/s ceiling, and it measures 42.5 below. Fine for a starter card, out of its depth at 32B. The 20x cliff: what happens when a model does not fit in VRAM

Runtimes like llama.cpp, Ollama and vLLM don't refuse

Runtimes like llama.cpp, Ollama and vLLM don't refuse a model that is too big. They split it: some layers in VRAM, the rest in system RAM across the PCIe bus. The GPU finishes its layers in microseconds, then stalls. The benchmark from the episode, Llama 8B on an RTX 4060 Ti 16GB:

Remember the middle row. Moving a fifth of

Remember the middle row. Moving a fifth of the model off the card cost 91 % of the speed, because every token waits for the slowest pipe. "It almost fits" is not a thing: either the model and its context fit in VRAM, or you run at DDR5 speed with extra steps. How much VRAM do you need for a 32B model? Weights plus KV cache Weights are the entry fee. At 4-bit quantization (GGUF Q4_K_M or AWQ):

News

VRAM for local LLMs: why memory bandwidth sets your tokens per second

How much VRAM for an LLM is the wrong first question.

@spots #dev
Source: Dev.to
See more like this