Why Two GPUs Are Not Twice One GPU

Two GPUs rarely give you twice the performance, and on consumer hardware they sometimes give you barely more than one. The reason is almost never the GPUs. It is what sits between them.

Where the traffic comes from

How much inter-GPU communication you generate depends entirely on how the work is split.

Data parallelism puts a full copy of the model on each device and feeds each a different slice of the batch. Devices only need to synchronise gradients once per step. Communication is proportional to model size, not to batch size, and it happens at a predictable moment. This scales well.

Model parallelism splits the model itself across devices, so activations must move between them at every layer boundary, many times per forward and backward pass. Communication is frequent, latency-sensitive, and sits directly in the critical path. This scales poorly unless the interconnect is very fast.

People reach for model parallelism when a model does not fit on one device, which means the situation that forces the harder pattern is also the one where the pattern hurts most.

The interconnect gap

The bandwidth difference between a proper GPU interconnect and standard PCIe is roughly an order of magnitude, and latency differs similarly. That gap does not matter for data-parallel work with infrequent synchronisation. It dominates for anything chatty.

It is also why the same two cards can look fine in one workload and disappointing in another, which makes benchmark comparisons misleading unless the workload matches yours.

There is a topology trap here too. Consumer motherboards frequently downgrade lane allocation when a second card is installed — a slot advertised at sixteen lanes may deliver eight, or four. Check the actual negotiated width rather than the specification, because the board will not tell you unless asked.

When multiple GPUs are clearly right

Running separate jobs in parallel. Two independent experiments, two model instances serving different requests, one training while another does inference. No communication, near-linear benefit, and this covers a large share of what research teams actually do day to day.

Data-parallel training where the model comfortably fits on one device and you want a larger effective batch.

When to buy one bigger card instead

If your constraint is that the model does not fit, a single device with more memory beats two smaller ones almost every time. You avoid the split, you avoid the traffic, and you avoid the debugging.

This is worth stating plainly because the instinct runs the other way: two cards look like more capability for the money. For the specific problem of a model that will not fit, they are usually less.

Measure before concluding

Before adding hardware, establish where the time is actually going. Profiling tools will show the split between compute, memory transfer and idle waiting. It is common to find a job is not GPU-bound at all — that it is waiting on data loading, or on preprocessing running single-threaded on the CPU.

Adding a second GPU to a data-loading bottleneck buys nothing and costs money. Checking takes an afternoon.

The short version

Multiple GPUs multiply throughput for independent work and struggle to accelerate a single large model without a fast interconnect. Buy the largest single memory pool you can afford first. Add cards to run more things at once, not to make one thing faster.