I quantised a matting model from 167.9 MB down to 42.1 MB, measured IoU 0.976–0.984 against the full-precision version, gained 35% on inference time, and shipped it. A week later someone sent me an image where the model returned one corner of the character’s crown and nothing else.
Same model, same preprocessing. IoU had gone from 0.98 to 0.30.
This is not a quantisation tutorial. It is a debugging log. I formed four hypotheses; the first three were killed by experiment and the fourth turned out to be right — but the real lesson is not about quantisation at all. It is about how I sampled when I validated it.
The scene
The setting is browser-side anime character matting: ONNX Runtime Web loads isnet-anime, segmentation runs on the visitor’s own device, and the mask is then vectorised. The model was compressed with quantize_dynamic(QuantType.QUInt8).
Before shipping I compared quantised against full precision on anime character art:
IoU (int8 vs fp32) 0.976 – 0.984
mean absolute diff 0.004 – 0.023
inference time -35%
size 167.9 MB -> 42.1 MBNothing in those numbers looks wrong. The image that broke it was a cartoon creature: pale blue body, gold crown, sitting against blue-green hills under a yellow sky. Subject and background share most of their palette.
The tool reported “subject coverage 8.2%”. The creature occupies at least forty percent of the frame.
Hypothesis 1: the output shape does not match
The code that crops the letterbox padding indexes straight into the output buffer with (y + dy) * 1024 + dx, which is only valid if the model really returns 1024×1024. If the actual output were some other size — a low-resolution side output, say — that indexing would read scrambled memory, and scrambled memory is exactly what a smeared blob looks like.
ISNet does have multiple side outputs during training, so this was a concrete suspicion. Reading the ONNX metadata:
input img [1, 3, 1024, 1024] tensor(float)
outputs 1
output mask [1, 1, 1024, 1024] tensor(float)
element count at run time = 1048576
element count code assumes = 1048576One output, exact match. Killed.
Hypothesis 2: the preprocessing is wrong
This model is fussy about preprocessing — it wants divide-by-255 with no mean subtraction, aspect-preserving letterbox to 1024, and the raw output used directly as alpha. I had previously fed it a generic ImageNet normalisation and got a peak output of 0.02, i.e. an almost empty mask. Given that history, the suspicion was reasonable.
I tested it by reimplementing the browser JavaScript line by line in Python: same letterbox arithmetic, same canvas fill, same channel layout, same crop and rescale. Then I ran an ordinary anime character image through it.
letterbox: w=934 h=1024 dx=45 dy=0
raw output range: 0.0000 .. 1.0000 mean 0.2183
final subject coverage: 23.9%Clean output, correct range, and a complete character silhouette. Killed.
Hypothesis 3: vectorisation is destroying the mask
After the mask there is a whole second pipeline: colour quantisation into layers, connected-component labelling, contour tracing, polyline simplification. I had already fixed a genuine bug in the contour tracer once — its stop condition was evaluated before the direction update, so a run travelling north never matched the start direction and it only ever walked the top edge. It had form.
I fed the real alpha and the real image into Node, running the same core.js that is deployed, with the exact settings the user had (6 layers, no simplification, no sharpening):
quantiseForeground -> 6 clusters
rgb(73,45,44) rgb(208,159,139) rgb(152,58,51) ...
labelComponents(minArea=417) -> 19 regions
region id=0 area=13155 bbox=[251,138,435,402]
traced contour bbox=[251,138,436,403]Across all 19 regions the contour bounding boxes matched the region bounding boxes (the one-pixel difference is the normal outer-edge offset), and the clusters were sensible character colours: dark brown hair, skin, red fabric, white trim. Killed.
Hypothesis 4: the WASM backend degrades quantised models
At this point I started suspecting the runtime. I had validated quantisation using Python’s CPU execution provider, but the browser runs the WebAssembly backend, and the two implement quantised operators differently — kernels like MatMulInteger and ConvInteger have historically been less complete on WASM.
Testing this cleanly means leaving the backend as the only variable. So I had Python dump the preprocessed input tensor to a raw .f32 file, then had onnxruntime-web in Node read those same bytes.
Python CPU backend: max=0.9727 mean=0.1264
Node WASM backend: max=0.9727 mean=0.1264Bit-identical. The backend was innocent. Killed.
What it actually was
With four hypotheses eliminated, only the quantisation itself was left. Running the same image at three precisions and comparing against fp32:
max abs diff mean abs diff [email protected]
int8 0.388 0.0547 0.296
fp16 0.001 0.00008 0.9996int8 scored IoU 0.296 on this image — against the 0.976–0.984 I had measured before shipping.
Sweeping the threshold makes it plainer:
threshold int8 fp16 fp32
48 32.4% 33.3% 33.3%
80 25.9% 32.0% 32.0%
112 11.8% 29.2% 29.2%
127 7.6% 25.9% 25.9%fp16 tracks fp32 at every threshold. int8 collapses as the threshold rises.
An aside: was this just the wrong model?
Before blaming quantisation I checked something simpler. isnet-anime is trained on anime characters, and this image is a non-humanoid cartoon creature. Would a general-purpose salient object model just handle it?
I tried isnet-general-use. The first run returned 1.9% coverage — worse than the anime model. But that number was worthless, because I had got the preprocessing wrong again.
rembg calls this model with mean=(0.5,0.5,0.5), std=(1.0,1.0,1.0), dividing by the maximum pixel value rather than by 255. I had reflexively applied ImageNet statistics. With the correct preprocessing:
ImageNet normalisation (wrong) coverage 1.9%
official preprocessing (right) coverage 21.4%21.4% is still below the anime model’s fp32 result of 25.9%, and the failure mode is identical: head and crown recovered, body and limbs and tail gone.
So swapping models does not solve this. Two models with different architectures and different training data make the same mistake on the same image, which says the difficulty is not in model selection — it is in the image. The subject’s flat pale blue and the background’s blue-green sit too close together in feature space.
Worth noting: I got normalisation parameters wrong twice during this investigation, once per model. A segmentation model’s preprocessing is part of how it was trained and does not transfer between models, even models that look like they belong to the same family.
Looking at the raw output
Once quantisation was the suspect, I saved the fp32 soft mask — unthresholded — as a greyscale image. It is more informative than any metric:
head + crown + ruff bright, close to white
body, limbs dark grey
tail barely visibleThe model does not fail to see the body. It sees it and is unsure. The quantised version pushes those already-weak signals down one more notch, and thresholding then erases them entirely.
That explains why the final result reads as random colour blobs rather than a character with pieces missing: what survives is a handful of high-confidence islands, the connections between them are severed, and vectorisation then traces each fragment as its own outline.
Why the two measurements disagree so violently
It comes down to how confident the model is about a given sample.
Quantisation error does not invent or destroy structure. It just jitters each pixel’s output value slightly. On a sample the model is sure about — clean character art, subject clearly distinct from background — outputs sit near 1 or near 0 and the middle ground is thin. A jitter of a few hundredths applied to 0.95 is still 0.95, applied to 0.03 is still 0.03, and the decision never flips. Hence IoU 0.98.
This cartoon creature’s body is flat pale blue against a blue-green background. The model was never sure, and large regions sit around 0.3–0.5. There, the same jitter is easily enough to push a pixel across the threshold.
Putting those two facts together produces an uncomfortable conclusion:
By validating quantisation only on samples the model was confident about, I was measuring sensitivity in the least sensitive region available. The 0.98 was not evidence that quantisation was safe. It was evidence that my test set contained no hard cases.
None of this is specific to quantisation. Any “the accuracy cost of compression is acceptable” claim — pruning, distillation, low-rank factorisation, downsampling — will come back equally optimistic if the validation set skews easy.
Switching to fp16
The fix was to drop int8 for fp16. Keeping the graph’s inputs and outputs at float32 means the front end needs no changes at all:
from onnxconverter_common import float16
m16 = float16.convert_float_to_float16(model, keep_io_types=True)keep_io_types=True is what makes this painless: the graph computes in fp16 but its inputs and outputs stay float32. The front end’s new ort.Tensor('float32', data, [1,3,1024,1024]) needs no change, and neither does reading the output.
Conversion emits a wall of warnings like this:
UserWarning: the float32 number -4.266044584255724e-08
will be truncated to -1e-07Those are weights whose magnitude falls below fp16’s smallest subnormal being clamped to ±1e-07. Alarming to read, but the impact is directly measurable: max absolute difference 0.001070, IoU 0.9996. Warnings of this kind have to be judged by measured difference, not by their own tone.
fp32 167.9 MB
fp16 84.0 MB IoU vs fp32 = 0.9996
int8 42.1 MB IoU vs fp32 = 0.296 (this image)That is twice the size of int8, but on WASM it measurably loads faster (79 ms versus 337 ms — a quantised graph costs extra parsing for its quantisation parameters) and inference time is a wash. For this application the 42 MB that int8 saved bought unusable output, which is not a good trade.
A second problem the same investigation exposed
When cutting the soft mask into a binary one, I had hard-coded the threshold at 127. Measuring the raw output on that image:
head region median 154
body region median 89Even at fp32, a threshold of 127 slices the body off. Limbs on a low-confidence subject naturally land in the 80–90 band, and a hard-coded threshold silently assumes a confidence level on the model’s behalf, for every image. The default is now 80, and adjustable.
How I will validate next time
The method is the part worth keeping, not the conclusion:
The validation set must contain samples the model is bad at. Deliberately go looking for inputs where subject and background share a palette, contrast is low, or the composition is atypical. If you cannot find a failure case, that is evidence your sampling is biased, not that the model is robust.
Look at IoU or per-sample metrics, not just means and peaks. The int8 peak output here was 0.9727, which reads as perfectly healthy — while IoU was 0.296. Means and peaks are blind to spatial structure collapsing.
Validate on the backend you actually ship. I nearly blamed WASM. The check is cheap: dump the input tensor to a file and have both backends read the same bytes, so the backend is the only variable. This time the backend was innocent — but without that control I would have changed the wrong thing.
Falsify before you fix. Three of the four hypotheses were entirely plausible. Acting on any one of them directly would have burned time in the wrong place and left the bug in — and the worst outcome is not “it did not help”, it is “it seemed to help a bit”, after which you carry a wrong causal model forward.
The failure mode described here is available as an interactive demonstration: the soft mask threshold lab. Drag the binarisation cut and the noise amplitude and watch the confident region hold while the uncertain one breaks into scattered pixels – with coverage barely moving, which is exactly what a mean IoU cannot see.