How much VRAM do you need to run an LLM locally?

How do you calculate the VRAM an LLM needs?#

Add three numbers: the model’s weights, the KV cache for your context, and working space for the runtime. Our VRAM calculator uses this formula, and so does every figure here.

• Weights = parameters × bits per weight ÷ 8. Quantised files also store scales, so llama.cpp measures 4.89 bits per weight for Q4_K_M and 8.5 for Q8_0.
• KV cache = the keys and values every attention layer stores for each token in the context. It grows with context length and with parallel requests.
• Overhead = the GPU runtime, compute buffers and fragmentation. We assume 10% of the weights and cache combined, at least 1 GB. That’s an assumption; the real figure varies by runtime.

For example, Llama 3.1 8B at Q4_K_M with a 32K context needs 4.6 GB of weights, 4.0 GB of KV cache and 1.0 GB of overhead: 9.6 GB in total.

The weights are easy to check, because they’re the file you download. Our estimates land within 1.4% of Qwen’s official GGUF files:

How much VRAM does an 8B, 14B, 32B or 70B model need?#

Popular models at the three precisions people download, with a short and a long context. Each name links to that model’s full table.

Three patterns stand out. Precision matters most: Qwen3 32B needs about 3.1 times as much memory at FP16 as at Q4_K_M. Context is not free: going from 8K to 32K adds 6.6 GB to Qwen3 32B but only 2.1 GB to Gemma 3 27B, because 52 of Gemma’s 62 layers only look at the last 1,024 tokens. And mixture-of-experts models save compute, not memory: Qwen3 30B-A3B runs 8 of its 128 experts per token, but loads them all.

Which quantisation should you use?#

Use Q4_K_M unless you have memory to spare; then step up to Q5_K_M, Q6_K or Q8_0. Storing each weight in fewer bits is the biggest lever you have: Qwen3 14B with a 32K context needs 36 GB at FP16 and 15 GB at Q4_K_M.

llama.cpp’s maintainers measure the cost by comparing a quantised model’s predictions with the full-precision model’s on the same text. For Llama 3 8B:

Two caveats from the same source. The loss depends on the model: at Q4_K_M, Llama 3 8B’s perplexity rose 2.8% but Llama 2 7B’s only 1.4%. And perplexity is a proxy, not a task score, so test your own prompts. Files made with an importance matrix (imatrix) lose a little less.

• Q8_0: effectively the original model, at about half the FP16 size.
• Q6_K, Q5_K_M: a small step down, worth it when a few GB are spare.
• Q4_K_M: the usual default and best balance of size and quality.
• Q3_K_M, Q2_K: a last resort. At Q2_K, Llama 3 8B changed its top token almost three times in ten.

How much VRAM does context length add?#

Every token of context adds a fixed amount to the KV cache, so a long context can need more memory than the model itself. For standard attention, each token costs 2 (a key and a value) × layers × KV heads × head size × 2 bytes at FP16.

At its full 128K context, Llama 3.1 8B’s cache is 16 GB, 3.5 times its Q4_K_M weights. So set the context you need, not the maximum: measure a typical prompt with the token counter (see what a token is) and add room for the reply. Parallel requests multiply the cache: four 8K chats use as much as one 32K chat.

Quantise the KV cache

llama.cpp and Ollama can store the cache in 8 or 4 bits. An 8-bit (q8_0) cache takes 53% of the FP16 size and a 4-bit (q4_0) cache 28%. Ollama’s documentation says q8_0 “usually has no noticeable impact on the model’s quality”, while q4_0 has a loss “that may be more noticeable at higher context sizes”. Both need flash attention, which current versions turn on automatically where it’s supported.

# llama.cpp
llama-server -hf Qwen/Qwen3-14B-GGUF:Q4_K_M -c 32768 -ctk q8_0 -ctv q8_0

# Ollama (set when starting the server; applies to every model)
OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_CONTEXT_LENGTH=32768 ollama serve

What fits on an 8, 12, 16, 24, 48 or 80 GB GPU?#

“Fits” means the estimate is no bigger than the card’s memory, so leave some room for your desktop and other apps.

The big steps are 8 to 12 GB, which takes you from 8B to 14B models, and 16 to 24 GB, which opens up the 27B to 32B class. A 70B model at Q4_K_M (47 GB) needs a 48 GB card or two 24 GB cards. At FP16 it needs 147 GB, more than an 80 GB card.

OpenAI’s gpt-oss is the exception to these quantisation levels. It is released in MXFP4, a 4-bit format, and its GGUF files are about the same size whichever label they carry, so go by the official files: 11.3 GB for gpt-oss 20B, about 13 GB in total with a 32K context, which fits a 16 GB card; and 59 GB for gpt-oss 120B, about 66 GB, which fits an 80 GB card.

What if the model doesn’t fit on one GPU?#

You can split it across several GPUs or keep part of it in system RAM. Splitting across GPUs keeps everything in fast VRAM, so it’s the better option when you can.

Several GPUs

llama.cpp uses every visible GPU by default. Its default split mode, layer, puts whole layers and their share of the KV cache on each card and runs them in turn, so for a single request two GPUs add memory rather than speed. --tensor-split 3,1 sets the proportions when the cards differ. Its row and experimental tensor modes, like vLLM’s tensor parallelism (--tensor-parallel-size 2), instead split each layer across GPUs so they work at the same time. Memory adds up: Llama 3.3 70B at Q4_K_M needs 2 × 24 GB cards, and each card keeps its own buffers, so leave a little extra headroom.

CPU offload

llama.cpp and Ollama can keep some layers in system RAM and run them on the CPU. In llama.cpp, -ngl (--n-gpu-layers) sets how many layers go to the GPU; its default, auto, picks what fits your VRAM. Ollama also decides for you, and ollama ps shows the split, for example “48%/52% CPU/GPU”. Every token passes through every layer, and system RAM is much slower than VRAM, so offloaded layers slow generation down. Try a smaller quantisation or a shorter context first.

Mixture-of-experts models offload better, because only a few experts run for each token. llama.cpp’s --n-cpu-moe N keeps the expert weights of the first N layers in system RAM and leaves attention on the GPU. That suits models like gpt-oss 120B, which runs 4 of its 128 experts per token but whose official file alone is 59 GB.

Worked examples: RTX 4060, RTX 4090 and two RTX 3090s#

“I have an RTX 4060 with 8 GB”

Run 7B to 8B models at Q4_K_M. Llama 3.1 8B needs 6.6 GB and Qwen3 8B 6.8 GB with an 8K context. At 32K, Llama 3.1 8B needs 9.6 GB, but an 8-bit KV cache brings it to 7.7 GB, which just fits. Gemma 3 12B at Q4_K_M (8.8 GB) is already over the limit, so 12B to 14B models need partial CPU offload or a 12 GB card.

“I have an RTX 4090 with 24 GB”

24 GB suits 24B to 32B models at Q4_K_M. Qwen3 32B needs 22.7 GB at 8K, a tight fit. At 32K it needs 29 GB, still 25 GB with an 8-bit cache, so 16K with an 8-bit cache (22.9 GB) is its practical limit. For long contexts, Gemma 3 27B (20 GB at 32K) and Qwen3 30B-A3B (22 GB) fit. A 70B model (47 GB) doesn’t.

“I have two RTX 3090s (48 GB in total)”

This is a common home setup for 70B models. Llama 3.3 70B at Q4_K_M needs 46.9 GB of the 48 GB with an 8K context, so there is little room to spare. A 16K window with an 8-bit cache needs 47.1 GB; for 32K, drop to Q3_K_M (42.0 GB with an 8-bit cache) and accept some quality loss. Or run Qwen3 32B at near-lossless Q8_0 (38 GB).

Macs work differently, because the GPU shares system memory: see how to run LLMs locally on a Mac. To compare with a hosted API, use the LLM cost calculator.