LLM VRAM calculator: can your GPU run it?

For Llama 3.3 70B with an 8K context, about 47 GB at Q4_K_M, 80 GB at 8-bit and 147 GB at full 16-bit precision. At Q4 that needs 2 × 24 GB cards or one 48 GB card, or a Mac with 64 GB of memory. Longer contexts add more: at 128K the cache alone is 40 GB.

Yes, with quantisation. Llama 3.1 8B at Q4_K_M with an 8K context needs about 6.6 GB, which fits an 8 GB card. At full precision it needs about 18 GB, so a 24 GB card.

Quantisation stores each weight in fewer bits, for example about 4.9 bits instead of 16, which cuts memory by roughly two-thirds. 8-bit is close to lossless, Q5 and Q6 lose very little, and Q4_K_M is the usual sweet spot. Below 4 bits quality drops more quickly, especially for smaller models.

For every token in the context, each attention layer keeps its keys and values in the KV cache, so the cache grows in proportion to the context length and the batch size. Models that use grouped-query attention, sliding windows or compressed (MLA) attention keep much smaller caches, which this calculator reads from each model’s configuration. Quantising the cache to 8-bit roughly halves it.

Yes. llama.cpp, vLLM and similar runtimes can split the layers across several GPUs, so two 24 GB cards hold about 48 GB of model. Each GPU needs a little extra for its own buffers, and splitting can be slower than one big card, but it is the usual way to run 70B models at home.

Runtimes such as llama.cpp and Ollama can keep some layers in normal system RAM and run them on the CPU. It works, but every layer on the CPU slows generation down considerably. A smaller quantisation, a shorter context or a smaller model usually gives a better experience.

By default macOS lets the GPU use about two-thirds of unified memory on Macs with up to 32 GB and three-quarters on larger ones. You can raise the limit with the iogpu.wired_limit_mb setting; the calculator assumes you keep 8 GB for macOS when you do.

No: every expert has to be loaded, so a 30B mixture-of-experts model needs as much memory as a 30B dense model. What it saves is compute: only a few experts run for each token, so it generates text much faster than a dense model of the same size.