VRAM, Not FLOPS: How to Size a GPU for Local AI Work

The first question people ask when buying a GPU for machine learning work is how fast it is. It is almost always the wrong first question. The constraint that decides whether you can do the work at all is memory capacity, and the constraint that decides how pleasant it is to do is memory bandwidth. Raw compute is usually third.

Why capacity is the binding constraint

A model has to fit. If it does not fit, no amount of compute helps — you are not running slowly, you are not running. Fitting means holding the weights, the activations, the optimiser state if you are training, and the key-value cache if you are generating text with long contexts.

For inference, a rough estimate: weights consume roughly one byte per parameter at 8-bit quantisation, half a byte at 4-bit. A 70-billion-parameter model is therefore about 70 GB at 8-bit and 35 GB at 4-bit, before you account for context. That arithmetic, not benchmark scores, determines which cards can run it.

For training, multiply. Full fine-tuning needs memory for gradients and optimiser state that typically exceeds the weights several times over. Parameter-efficient methods such as LoRA change this equation dramatically, which is why they are the default for anyone not working at industrial scale.

Bandwidth decides how it feels

Text generation is memory-bandwidth bound rather than compute bound. Producing each token requires reading the model weights from memory, and that read is the bottleneck. This is why two cards with similar compute but different memory technologies produce noticeably different generation speeds, and why comparing on FLOPS alone misleads.

Batch processing changes the picture — with enough concurrent requests, you amortise the weight read and become compute bound. For single-user local work, you almost never reach that point.

Quantisation is the highest-leverage decision

Reducing numerical precision cuts memory use close to proportionally. Moving from 16-bit to 8-bit roughly halves the requirement; 4-bit halves it again. Modern quantisation methods lose remarkably little quality on most tasks at 8-bit and are usually acceptable at 4-bit.

The practical implication is that a card which cannot hold a model at full precision may hold it comfortably quantised, and a larger model quantised generally outperforms a smaller model at full precision. If you must choose, prefer the bigger model with fewer bits.

Two cards are not twice one card

Splitting a model across GPUs works, but the halves must exchange activations continuously, and that traffic crosses whatever interconnect you have. On consumer hardware over PCIe, this is slow enough that two cards frequently deliver well under the throughput the specifications suggest.

Multiple GPUs are excellent for running separate jobs in parallel — several experiments at once, or several model instances. They are much less effective at making one large model fast. Buy the single largest memory pool you can afford before buying a second card.

A short sizing guide

• Up to 16 GB: models around 7–13 billion parameters quantised. Fine for experimentation and most classical ML.
• 24 GB: comfortable with 13B at good quality, 30B quantised, and LoRA fine-tuning of smaller models. The practical sweet spot for individual researchers.
• 48 GB and above: 70B-class models quantised, longer contexts, and genuinely usable fine-tuning.

Before buying anything, run your actual workload on rented hardware for a few hours. The gap between what people expect to need and what they actually need is consistently large, and it is cheaper to discover that at hourly rates.