Sizing an On-Prem GPU from Weeks of Resident VRAM Data

When you move an ML inference workload from the cloud to a single on-prem box, the first question is whether one GPU is enough. The spec sheet answer is useless here. What decides it is the memory your container actually holds while idle, plus the peak it reaches under load. I sized an on-prem GPU by reading resident VRAM off three cloud GPUs that had been running the same image for weeks, and the numbers made the decision obvious.

Resident memory is the floor, not the spec sheet

An inference container that loads its models at startup does not release that memory between jobs. It holds the whole model set resident so the next request does not pay a cold load. So the number that sets your floor is not the card size and not the peak during a request. It is the steady resident footprint while the container sits idle.

That footprint is invisible until you measure it on a real, warmed-up process. A fresh container right after boot reads lower than one that has served jobs for a week, because allocator fragmentation and cache growth push the resident set up over time. If you size against a cold reading you will undershoot.

What the readings showed

Three GPUs in the cloud were running the identical image, each a 23 GB card. I read the resident VRAM of the inference container on all three while idle:

The spread from 13.7 to 17.4 GB across identical images is the fragmentation effect. The container loads its full model set at boot and holds 13.7 to 17.4 GB resident for the life of the process. More important than the average is that peak inference on top of that resident set stayed inside the 23 GB card across weeks of production. Resident plus active never crossed the card. That is the fact you cannot get from a spec sheet, only from a warmed-up process under real traffic.

Headless boot reclaims what the desktop stole

The on-prem box was a workstation, and it booted into a full desktop. The display manager and its compositor were holding several hundred MiB of GPU memory before the inference container even started. On a workstation nobody logs into, that is pure waste sitting on the exact resource you are trying to size.

Switching the default boot target to headless and disabling the display manager returned that memory to the GPU. After the switch, GPU processes dropped to just the inference container, with nothing held by a desktop that no one was going to use. If you are sizing tight, measure with the desktop off, because that is how the box will actually run in production.

The sizing decision

The on-prem card was a 24 GB GPU, about 1.5 GB larger than the 23 GB cloud cards that had already proven the workload fits. With the desktop disabled, the container's resident set of roughly 15 GB (17.4 GB worst case) left 7 to 11 GB of headroom for inference peaks on a card that was already larger than the proven-good one. No second GPU, no upgrade. One card was enough, and the weeks of cloud data said so with more confidence than any benchmark I could have run in an afternoon.

What to measure before you buy

The general method holds for any resident-heavy inference workload:

• Read resident VRAM on a process that has been serving real traffic for days, not a fresh boot. Take the maximum across identical instances, not the average.
• Confirm that resident plus inference peak stayed inside a known-good card in production. That single fact replaces a pile of synthetic benchmarks.
• Measure with the display manager off if the box is a workstation, so the desktop is not stealing headroom you counted on.
• Buy the card whose size clears your worst-case resident plus peak with room to spare, and stop. A GPU you do not need is the most expensive idle hardware in the rack.