“Will this model fit on this card?” has two kinds of answer. One is a lookup table — record measured values for common combinations and consult it. The other is an estimator that computes an answer for any combination.
The trouble with a table is combinatorial coverage: variant, quantisation format, context length, card count — four dimensions multiply out to hundreds of entries. Building an actually usable calculator means modelling VRAM consumption.
This records that modelling exercise: what the formula looks like, how to measure the one quantity you cannot look up, and two constraints that VRAM alone will not reveal.
Three known mechanisms, briefly
Three traps have to be cleared before modelling. I wrote up their symptoms and causes in detail in a separate piece on choosing between variants, so here are just the conclusions — each corresponds to a term in the formula:
One: parameter count does not give you size. Vendors define parameter counts inconsistently (total / activated / effective), while the byte count on disk is unambiguous. So the estimator’s input must be a looked-up file size, not a parameter count.
Two: quantisation coverage is not uniform. Common quantisation configs compress only linear layers, leaving embeddings at full precision. On architectures using per-layer embeddings, that uncompressed lookup table can exceed the trunk — so “few parameters” can mean “large file”. The formula must account for the uncompressed portion at its original precision.
Three: file size is not VRAM usage. Some inference engines keep the sparsely-accessed embedding table in system memory. The same file loads a different amount onto the card depending on the engine. So what you estimate is “what the engine actually loads”, not the file size.
Those three determine what the formula’s inputs should be. The rest is the formula itself.
Three terms
Fitted to within 0.15 GiB across four measured configurations. Three terms summed:
Weights, minus the portion resident in system memory. Deducted only for architectures with per-layer embeddings, and only for formats where the engine makes that choice:
CPU-resident = (vocab × layers × perLayerDim + vocab × hidden) × bytesPerParam
VRAM weights = file size - CPU-residentKV cache. Modern models often mix global attention with sliding-window attention: global layers grow KV linearly with context, windowed layers plateau at the window size. So this term cannot use the total layer count — only the global layer count. And the global share is a quantity you cannot look up; the next section covers measuring it.
Runtime overhead. Purely empirical, covering CUDA context, activations, and fragmentation:
overhead = max(0.7 × cards, weights × 0.10) + context/131072 × 0.35The two arms of the max correspond to two regimes: with a small model across several cards, the fixed per-card context cost dominates; with a large model on one card, weight-proportional activation does. It’s a max rather than a sum because in measurement the two do not stack — each becomes the bottleneck at a different scale.
The quantity you can’t look up: measuring the global share
The proportion of global attention layers may not be in vendor documentation, and may not be in the config file either. But it can be measured, and the method generalises.
The principle is that the two kinds of layer grow their KV cache differently: global layers linearly, windowed layers plateauing. So the derivative of total VRAM with respect to context length comes only from the global layers. Measure that derivative and you can back out the global layer count.
Hold everything else fixed, vary only context length, and record steady-state VRAM at each step:
for ctx in 8192 16384 32768 65536; do
start engine --ctx-size $ctx --parallel 1
wait for loading to finish and run one inference
nvidia-smi --query-gpu=memory.used --format=csv,noheader
stop engine
doneDivide the VRAM difference between adjacent steps by the context difference to get bytes of KV per token. Divide that by the theoretical per-layer per-token KV size (2 × kvHeads × headDim × bytesPerElement) for the equivalent global layer count; divide by total layers for the share. Here it came out at roughly 3/8 of the non-shared layers.
Three things to watch, each of which caught me:
Pin concurrency to 1. Most engines allocate a separate KV cache per concurrent slot. Leaving concurrency unfixed means varying a multiplier, and the slope becomes meaningless. This one hides well, because the default concurrency is often not 1.
Wait for loading to finish and one inference to run before reading. Reading mid-load captures a partial allocation, and how far along you are differs per step, so the noise swamps the signal.
Take at least three points and check linearity. Two points always form a line and reveal nothing. If three points aren’t collinear, something else is active — a step that triggers a different allocation strategy, say — and the slope itself is untrustworthy until you understand the inflection.
The method isn’t specific to attention layers. Any quantity that varies monotonically with a parameter you control can be measured by holding everything else fixed, varying one thing, and reading the slope — no knowledge of the internals required. If you can’t look it up, measure it; don’t guess.
The forgotten dimension: concurrency
I described the inputs as four dimensions — variant, format, context length, card count. There is a fifth, and it is a multiplier: the number of concurrent slots.
To serve multiple requests at once, an engine pre-allocates one KV cache per slot. So:
actual KV cache = single KV cache × concurrent slotsThe problem is that the default is often not 1. Some engines default to 4. Set --ctx-size 16384 expecting to budget for 16K of context, and what actually gets allocated is 16384 × 4 = 65536.
The failure mode is deceptive. It doesn’t report out-of-memory, because the engine spills model layers that no longer fit into system memory and keeps going. The symptom is that it starts, produces correct output, and is one to two orders of magnitude slower — in one case prompt processing fell from over a thousand tokens per second to 1.3. Without knowing about the multiplier you go hunting through drivers, hardware, and model formats, when the cause is a single default parameter.
So the estimator owes two things. Take concurrency as an explicit input rather than assuming 1. And when emitting a suggested command line, write the concurrency flag explicitly even when the value matches the default — a flag written out is visible to whoever reads the command next; a flag left to its default is discovered only after it bites.
This is also why concurrency has to be pinned to 1 when measuring the global-layer share above: it’s a multiplier, and leaving it free means the slope you measure mixes two variables.
Constraint VRAM won’t show you, part one: divisibility
Fitting in VRAM clears one bar. There is a second, independent one: tensor parallelism requires the KV head count to be divisible by the card count. Fail it and the engine does not start — this is not degraded performance, it is a startup failure.
KV head counts vary widely within a model family, anywhere from 1 to 16. Two consequences follow:
A variant with a single KV head can never use tensor parallelism. However many cards you have, it runs on one. Such variants are usually the smallest in the family and rarely need multiple cards anyway — but if your scheduling logic assumes “add a card when VRAM is short”, it will fail repeatedly on this one.
At some card counts, no variant in the family works. With 3, 5, or 6 cards it is easy for every variant’s KV head count to fail divisibility. Powers of two — four cards, eight cards — are far safer, which is a large part of why multi-GPU configurations are almost always powers of two.
Engines that split by layer have no such constraint, but be clear that they solve a different problem: they let you fit a model that won’t fit on one card, without making it faster — layers execute sequentially, and multiple cards just store them separately. Throughput still requires tensor parallelism, which still requires divisibility.
So the calculator must answer two questions: does it fit, and can it parallelise at this card count. Both must pass.
Constraint VRAM won’t show you, part two: ranking
The final design decision: several combinations fit, so which do you recommend?
The lazy implementation sorts by weight size descending — bigger means stronger. Measurement says this misfires: at one VRAM tier it recommended a small variant at high precision (16.8 GB) over a clearly stronger large variant in a quantised format (16.3 GB). Near-identical size, meaningfully different capability.
The correct ordering is two-level: among the candidates that fit, pick the most capable variant, then within it pick the highest-precision format. Variant strength comes from architecture and evaluations, not from size.
The general lesson: using an easily measured quantity as a proxy for the one you care about gives wrong answers wherever the correlation between them is unstable. Size and capability correlate monotonically within a variant (same model, higher precision is stronger) and not across variants (different models, size says nothing). Ranking has to happen inside the range where the correlation holds — here, group first, then sort within the group.
The pattern recurs elsewhere: file size as a proxy for content quality, line count as a proxy for complexity, response time as a proxy for system health. Each holds inside some range and breaks across ranges.
When the formula stops applying
Anything fitted has a domain, and writing the boundary down matters more than the formula — otherwise the next person applies it to a case it never covered and concludes they misconfigured something.
Changing engine requires re-measurement. Which tensors stay in system memory is entirely an engine implementation choice, not a model property. The same weights on a different engine may make that deduction vanish. The term should be gated on the engine, not applied unconditionally.
Changing architecture family requires re-measurement. The global-layer share is a constant inferred for one architecture. A family with different attention design makes that 3/8 meaningless — and worse, it will produce a plausible-looking number. No error, just wrong.
A new quantisation scheme means re-reading the config. Vendors may start quantising embeddings at any point. Once they do, mechanism two inverts entirely — the “small is bigger than large” anomaly disappears, and the old formula overestimates.
The overhead coefficients are empirical. 0.7 × cards and × 0.10 have no theoretical basis; they were fitted against specific driver and engine versions. A major driver update is worth re-measuring after — the fixed cost of a CUDA context genuinely shifts between versions.
The fix is not to make the formula cover every case, but to write each term’s applicability and provenance into the code comments and keep those measured configurations as a regression baseline. After any change, run the same inputs; an error beyond 0.15 GiB means something was touched that shouldn’t have been.
What’s worth keeping
An estimator’s error usually comes from a missing mechanism, not a wrong coefficient. None of the three corrections was a mistyped formula; each time the model lacked something that genuinely exists. So when estimates are off, tuning constants rarely helps — look for what you haven’t modelled.
Measure what you can’t look up; don’t guess it. The global-layer share isn’t documented, but it varies monotonically with context, so it can be measured. Anything varying monotonically with a parameter you control has the same path available.
“Good enough” is usually more than one condition. Fitting in VRAM is one; divisibility is another, and it fails in an entirely different way — not slower, but refusing to start. When designing any feasibility check, enumerate every hard constraint first, then worry about ranking and recommendation.