A segmentation model emits a soft mask — one confidence value between 0 and 1 per pixel. Almost everything downstream then cuts it into a hard black-and-white mask, and where that cut lands, together with how much quantisation noise is present, decides the result.
This experiment lets you drag both knobs and watch what the soft mask becomes. On the left is the raw model output: the model is confident about the upper half (near white) and unsure about the lower half (grey). On the right is what survives thresholding.
Watch the fragmentation figure while keeping an eye on coverage. You will see something counterintuitive: the low-confidence region shatters into scattered pixels while coverage barely moves.
What this demonstrates
It reproduces a trap we hit in practice: an int8 quantised model that was essentially lossless on the standard validation set dropped to 0.30 IoU on out-of-distribution images.
The cause was not larger quantisation error — the error was much the same everywhere. The cause was which part of the soft mask the error landed on. Where the model is confident, values sit near 0 or 1, far from the cut, and no realistic amount of noise pushes them across. Where the model is unsure, values already hover around the cut, and the smallest perturbation flips pixels back and forth. So identical noise does nothing on easy samples and perforates the entire region on hard ones.
Push the noise past 0.12 and watch only the lower half and the fragmentation number; the effect is stark.
Why area metrics cannot see it
IoU, Dice and coverage all measure area overlap. A region shattered into hundreds of isolated pixels retains roughly its original total area, so those metrics barely register. Yet for nearly every downstream use — compositing, contour tracing, connected-component analysis — a shattered mask is unusable.
Quantisation validation reported as IoU alone is therefore insufficient. At minimum, also track:
- Connected component count: a real subject is usually a handful of components; an explosion in count is fragmentation.
- Perimeter-to-area ratio: fragmentation drives perimeter up sharply relative to area.
- Error stratified by confidence band: bucket pixels by soft-mask value and report each separately. A global average dilutes the disaster in the bucket nearest the threshold.
Applying this to your own work
The mask here is generated procedurally rather than emitted by a real model, so it demonstrates a mechanism rather than specific numbers. What transfers is the validation-set construction principle: when validating a quantised model, assemble a separate batch of samples the model is genuinely unsure about — blurred boundaries, low contrast, classes rare in the training set. Validating quantisation only on clean samples systematically overstates how safe it is.
If stratified validation shows int8 is unacceptable in the hard band, falling back to fp16 is usually the right trade: half the size benefit given up for predictable behaviour.
Related reading
- Sampling bias in quantisation validation: int8 collapse out of distribution: the full measured account, including the drop from near-lossless to 0.30 IoU and the stratified validation scheme.
- A termination-condition bug in contour tracing: one concrete downstream failure caused by shattered masks.
- A client-side-only image pipeline: why tools like this can run entirely in the browser.
- Tools: other browser-side utilities on this site.