GPU VRAM Calculator: How Much GPU Memory Does an LLM Need?

Teams planning to run a language model on their own infrastructure all start with the same question: how much GPU memory will it need? This GPU VRAM calculator estimates the memory required for a given model, weight precision, context length and number of concurrent requests, then maps the result onto common GPU classes. The formula behind it is laid out below, so the number is a checkable estimate rather than a black box.

All calculations run in your browser; nothing you enter is sent to a server.

How is GPU VRAM calculated for an LLM?

During inference, a language model’s GPU memory has three parts: model weights, the KV cache and runtime headroom. Weights are the parameter count multiplied by bytes per parameter; the KV cache grows linearly with context length and the number of concurrent requests.

• Weights: parameters × bytes per parameter. 2 bytes in FP16/BF16, 1 byte in FP8/INT8, about 0.5 bytes with 4-bit quantization (GPTQ, AWQ).
• KV cache: 2 (keys and values) × layers × KV heads × head dimension × 2 bytes × tokens × concurrent requests.
• Headroom: roughly 10% of the total for the CUDA context, temporary buffers and fragmentation.

The layer, KV-head and head-dimension values in the model list come from each model’s published configuration. For a model that isn’t listed, choose “Custom model” and enter the values from its config.json: num_hidden_layers, num_key_value_heads and head_dim.

How much VRAM do popular models need?

The table below uses the calculator’s own formula for a single request with an 8,192-token context. A 70B model needs about 158 GB in FP16 and about 42 GB with 4-bit quantization.

Model	KV per token	FP16 / BF16	FP8 / INT8	INT4
Mistral 7B	128 KB	17.0 GB	9.1 GB	5.1 GB
Llama 3.1 8B	128 KB	18.8 GB	10.0 GB	5.6 GB
Qwen2.5 14B	192 KB	34.1 GB	17.9 GB	9.9 GB
Qwen2.5 32B	256 KB	73.9 GB	38.1 GB	20.2 GB
Llama 3.1 70B	320 KB	158.3 GB	80.6 GB	41.8 GB
Llama 3.1 405B	504 KB	895.7 GB	450.2 GB	227.4 GB

These figures are for a single user. The same 8B model serving eight concurrent requests at 8,192 tokens rises from 18.8 GB to 27 GB in FP16, and all of the increase comes from the KV cache. When sizing a live chat service, expected concurrency matters as much as model size.

Why do context length and concurrency matter so much?

The KV cache reserves a fixed amount of memory per token, and it accumulates across the context. Llama 3.1 70B stores 320 KB per token, so a single request at the full 131,072-token context needs about 43 GB for the KV cache alone. For long-document summarization or codebase analysis, this cache, not the weights, is often the real bottleneck.

Most recent models use Grouped-Query Attention (GQA), which keeps fewer KV heads than attention heads and shrinks the cache several times over compared with older architectures. Serving frameworks such as vLLM also split the cache into pages to reduce waste (vLLM documentation). The calculator does not count these savings; it shows the worst-case requirement, which is the safe side for planning.

How much VRAM do you need for fine-tuning?

Full fine-tuning needs far more memory than inference. Mixed-precision training with the AdamW optimizer takes about 18 bytes per parameter: 6 for weights, 8 for optimizer states and 4 for gradients (Hugging Face, Model training anatomy). By that measure, full fine-tuning of a 70B model needs well over 1.3 TB before activations, which is only practical when spread across many GPUs.

LoRA and QLoRA change the picture. LoRA freezes the base weights and trains small adapter matrices; QLoRA also quantizes the base weights to 4 bits. With QLoRA, a 70B model at an 8,192-token context comes down to roughly 50 GB and can run on a single 80 GB card. The calculator’s LoRA/QLoRA mode adds a rough allowance of 20% of the weights for adapters, optimizer states and activations; the real value depends on batch size and activation checkpointing.

Which GPU class fits which workload?

Choosing a card is about memory bandwidth and interconnect as well as capacity. The calculator maps the result onto four memory classes:

• 24 GB (RTX 4090, L4): FP16 inference for 7-8B models, 4-bit inference for 14B models, small LoRA jobs and rendering.
• 48 GB (L40S, RTX 6000 Ada): FP8/INT4 inference for 32B models, mid-size QLoRA, heavy rendering and image generation.
• 80 GB (A100, H100): FP8 inference for 70B models, large QLoRA jobs and multi-GPU FP16 serving for 70B.
• 141 GB (H200): single-card FP8 and long-context inference for 70B models, and serving large models on fewer cards.

In multi-GPU setups, the model’s layers are split across cards (tensor or pipeline parallelism). Here, NVLink between cards makes a clear speed difference compared with PCIe, so a quote should cover how the cards are connected inside the server, not just total memory.

Common sizing mistakes

1. Looking only at the weights: an 8B model seems to fit in 16 GB, but concurrent users and long contexts easily push the KV cache past that limit.
2. Counting the whole card as usable: the driver, CUDA context and fragmentation always occupy part of the memory; the calculator assumes 95% usable memory per card.
3. Ignoring the quality impact of quantization: 4-bit models cut memory to a quarter, but some tasks show measurable accuracy loss; compare on the same test set before going to production.
4. Planning training and serving on the same card: running a fine-tuning job alongside live inference slows both down.

GPU servers at MAV Cloud: support and assurance

A critical failure that stops a training or inference job is handled as critical priority under MAV Cloud’s service level agreement, with a first-response target of 15 minutes.

Model weights and training data are often an organization’s most sensitive assets. MAV Cloud’s information security management system is ISO/IEC 27001 certified and its privacy information management system is ISO/IEC 27701 certified (see our certifications). GPU server options and the rental process are covered on the GPU server rental page, and the key differences between GPU and CPU servers in our what is a GPU server guide.

Frequently Asked Questions

How much VRAM do I need for a 70B model?

Llama 3.1 70B with a single request and an 8,192-token context needs about 158 GB in FP16, about 81 GB in FP8 and about 42 GB with 4-bit quantization. More concurrent requests increase the requirement through the KV cache.

How accurate is the calculator?

It is an estimate that stays on the safe side for planning. The serving framework, batch size, paged KV cache and activation checkpointing can shift the real figure by a few GB.

Can I use it for rendering or image generation?

The calculator is built for language models. Rendering memory depends on scene textures and geometry; 24 GB and 48 GB class cards are a common starting point.

Is anything I enter stored?

No. The calculation runs in your browser and your selections are not sent to a server.

If the model choice is still open, the calculator is a quick way to compare scenarios. If the setup is already clear, the quote link on the result screen is the fastest route.