How Much VRAM Do You Need to Run an LLM Locally? (With the Math)

To estimate the VRAM an LLM needs, multiply the number of parameters by the bits stored per weight and divide by 8 to get bytes, then add the KV cache for your context length and some runtime overhead. An 8-billion-parameter model needs about 16 GB for its weights at FP16, about 8.5 GB at 8-bit (Q8_0) and about 4.9 GB at 4-bit (Q4_K_M), before the cache.

Every number below is arithmetic you can check. Where a value is an estimate, it is labelled as one.

The formula

weights (bytes)  = parameters × bits_per_weight ÷ 8
KV cache (bytes) = 2 × layers × kv_heads × head_dim × bytes_per_value × context_tokens
total            ≈ weights + KV cache + runtime overhead

The first line is the same rule Hugging Face's Transformers guide gives: a model with X billion parameters "requires roughly 2 * X GB of VRAM in bfloat16/float16 precision", because each parameter takes 16 bits, which is 2 bytes. Quantization lowers the bits per weight, so the same formula covers 8-bit and 4-bit files.

The second line is the KV cache, the stored attention keys and values for every token in the context. The Transformers guide describes it as two times the sequence length times the number of attention heads times the head dimension times the number of layers, in float values. Models with grouped-query attention store fewer key/value heads than query heads, so you use the number of KV heads, which is listed in the model's config.json as num_key_value_heads.

The third term, runtime overhead, covers working buffers the inference engine allocates. It depends on the runtime, backend and batch size, and there is no single published figure, so this article does not invent one. Leave a margin and check real usage (see the end of this article).

Bits per weight for common formats

The "4-bit" and "8-bit" formats store a little more than 4 or 8 bits per weight, because each block of weights also stores a scale. The exact values follow from the block layouts in llama.cpp's ggml-common.h:

The file types you download, such as Q4_K_M, are mixes. The _M variant keeps some tensors at a higher precision, so the average is higher than the base type. llama.cpp's quantize README measures this on Llama-3.1-8B: Q4_K_M averages 4.8944 bits per weight, Q5_K_M 5.7036, Q6_K 6.5633 and Q8_0 8.5008. Those averages vary a little between models. The table below uses 4.8944 for Q4_K_M as an estimate. What LLM quantization and GGUF are explains the formats and their quality trade-off.

VRAM for model weights, by size and format

Weights only, in GB (10⁹ bytes), computed as parameters × bits per weight ÷ 8. Add the KV cache and overhead on top.

Check one row yourself: 14 × 10⁹ × 8.5 ÷ 8 = 14.875 × 10⁹ bytes, or 14.9 GB.

The table agrees with llama.cpp's published sizes. Its README lists Llama 3.1 8B at Q4_K_M as 4.9 GB and Llama 3.1 70B at Q4_K_M as 43.1 GB. The 70B figure is a little above the table because that model has about 70.6 billion parameters, not exactly 70.

Real models rarely have round parameter counts. Qwen2.5-7B-Instruct, for example, has 7.61 billion parameters according to its model card, so use the exact count from the model card when you want a precise number.

Worked example 1: a 7B model on an 8 GB GPU

Model: Qwen2.5-7B-Instruct. From its model card and config.json: 7.61 billion parameters, 28 layers, 28 query heads, 4 KV heads, hidden size 3,584. Head dimension = 3,584 ÷ 28 = 128.

weights (Q4_K_M, estimate) = 7.61e9 × 4.8944 ÷ 8      = 4.66 GB
KV cache per token (FP16)   = 2 × 28 × 4 × 128 × 2 bytes = 57,344 bytes
KV cache at 8,192 tokens    = 57,344 × 8,192            = 0.47 GB
KV cache at 32,768 tokens   = 57,344 × 32,768           = 1.88 GB

At an 8k context the total is about 4.66 + 0.47 = 5.1 GB plus overhead, which leaves room on an 8 GB card. At 32k context it is about 6.5 GB plus overhead, which is tight. At FP16 the weights alone would be 7.61 × 2 = 15.2 GB, which does not fit at all.

Worked example 2: a 72B model

Model: Qwen2.5-72B-Instruct. From its model card and config: 72.7 billion parameters, 80 layers, 64 query heads, 8 KV heads, hidden size 8,192, so head dimension = 8,192 ÷ 64 = 128.

weights (Q4_K_M, estimate) = 72.7e9 × 4.8944 ÷ 8      = 44.5 GB
KV cache per token (FP16)   = 2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes
KV cache at 8,192 tokens    = 327,680 × 8,192           = 2.68 GB
KV cache at 32,768 tokens   = 327,680 × 32,768          = 10.7 GB

About 47 GB plus overhead at 8k context, and about 55 GB at 32k. That does not fit on any single 24 GB consumer card, and is borderline even across two of them. In practice you either split the model between GPU and system RAM (slower), choose a lower-bit quantization (lower quality), or pick a smaller model.

Why context length changes the answer

The KV cache grows linearly with the number of tokens in the context. The Transformers guide shows how large it can get. For its test model (bigcode/octocoder) at a 16,000-token context, computed as if every attention head kept its own keys and values, the cache comes to roughly 8 billion float values, "around 15 GB" at float16, about half the size of the weights. The same guide then notes that the model actually uses multi-query attention, which brings that figure down to under 400 MB. Grouped-query attention, used by both Qwen examples above, works the same way to a lesser degree, which is why their caches are so much smaller than the weights.

Two settings help when the cache is the problem:

• Use a shorter context. If you do not need 32k tokens, do not allocate them. Ollama sets its default from your VRAM: 4k below 24 GiB, 32k for 24 to 48 GiB, 256k at 48 GiB or more.
• Quantize the cache. Ollama's FAQ says OLLAMA_KV_CACHE_TYPE=q8_0 uses about half the memory of the default f16 and q4_0 about a quarter, with q4_0 losing more precision at large context sizes. It only applies when Flash Attention is enabled, and it is a global setting for all models.

When the model does not fit

llama.cpp and Ollama can keep some layers in VRAM and run the rest on the CPU from system RAM. This works, but the CPU part is much slower. Ollama's FAQ explains how to see the split: run ollama ps and read the PROCESSOR column. 100% GPU means the whole model is in VRAM; a value like 48%/52% CPU/GPU means part of it is in system memory.

How to check real usage

Estimates get you to the right model size. Measure to confirm:

ollama ps
nvidia-smi

ollama ps shows the loaded model's size, the CPU/GPU split and the context length. nvidia-smi (NVIDIA GPUs) shows memory used per process. Load the model with the context length you plan to use, send a long prompt, and read the numbers while it runs. If you have not set up a local runtime yet, how to run an LLM locally with Ollama covers installation and choosing a model tag.

Frequently asked questions

Can I run a 70B model on a 24 GB GPU?

Not entirely in VRAM at usual quantizations. At about 4.9 bits per weight (Q4_K_M), the weights of a 70-billion-parameter model alone are about 43 GB. Runtimes such as llama.cpp and Ollama can split the model between GPU and system RAM, but the part on the CPU runs much slower. Smaller quantizations (around 2 to 3 bits per weight) fit more but lose more quality.

Does system RAM count as VRAM?

Not on a typical PC with a separate graphics card. Layers that do not fit in VRAM can run from system RAM on the CPU, and Ollama shows the split in the PROCESSOR column of ollama ps, for example 48%/52% CPU/GPU. The model still works, just more slowly. Ollama's docs say Apple M-series Macs get CPU and GPU support, while Intel Macs run on the CPU only.

Why is my GPU memory use higher than the model file size?

Because the file only holds the weights. At runtime the KV cache grows with the context length, and the runtime allocates working buffers on top. A long context window can add several gigabytes, which is why the same model uses more memory at 32k context than at 4k.

Is GB the same as GiB in these numbers?

No. This article uses GB = 1,000,000,000 bytes. GPU tools and llama.cpp often report GiB (1,073,741,824 bytes), which makes the same amount look about 7% smaller. For example, llama.cpp lists Llama-3.1-8B at F16 as 14.96 GiB, which is about 16.06 GB.

Sources

• llama.cpp: tools/quantize README (bits per weight and sizes for Llama-3.1-8B)
• llama.cpp: ggml-common.h (quantization block layouts)
• Hugging Face Hub docs: GGUF quantization types
• Hugging Face Transformers docs: Optimizing LLMs for speed and memory
• Qwen2.5-7B-Instruct model card and config
• Qwen2.5-72B-Instruct model card and config
• Ollama docs: FAQ (GPU offload, KV cache quantization)
• Ollama docs: Context length
• Ollama docs: macOS system requirements