I built a tool for my site: upload a few images of a character from different angles, get automatic cutout, layer separation, conversion to SVG vector assets, and a library to keep them in for animating later.
There was exactly one constraint: nothing is uploaded; all compute and all storage happen on the visitor’s device.
That sounds like a privacy decision. The original motivation was cost — the origin is a single-core box behind a page cache, and under normal operation the CPU cost per request is close to zero. Adding a server-side image pipeline would destroy that premise entirely.
Having built it, though, the CPU turned out to be the least of what it saved.
Four things it actually saves
Storage. Where do uploaded images live, for how long, and who cleans them up? The moment there is a server-side upload those questions demand answers. Entirely client-side, they do not exist.
Moderation. Any feature that accepts and stores user-supplied images becomes a content moderation surface. A personal site has no capacity to carry that.
Privacy obligations. Receive no data and you need not state how it is held, how long it is kept, or how it is deleted.
Concurrency. The hard part of server-side image processing is not one image’s latency, it is five people arriving at once. On a single core that is an outage. Client-side processing is naturally per-person — everyone spends their own CPU.
In hindsight the CPU was the least important of the four.
How the whole pipeline runs in a browser
Input is an image file; output is a set of SVG layers with path data. Five steps, each with either a ready browser API or a pure-JS implementation.
Decode. URL.createObjectURL into an <img>, wait for onload, draw to a canvas, and getImageData for the RGBA array. This step also handles upscaling: give drawImage a target size and the browser’s own scaler is far faster and better than hand-written interpolation.
Matting. Two paths. The classic one seeds a flood fill from the four edges and treats regions close in colour to the border as background — effective and instant on artwork with a clean backdrop. Complex backgrounds need a segmentation model, run through ONNX Runtime Web on WASM.
Layering. k-means colour clustering over the foreground pixels, splitting the character into flat colour regions. The cluster count is the “layer count” the user sees.
Connected components. One colour cluster may be scattered across several disjoint areas — two sleeves, say — so components are labelled separately and tiny fragments discarded.
Vectorisation. Contour-trace each component into a point list, simplify with Douglas-Peucker, and emit SVG path data.
No network requests anywhere, except the one-time model download on the AI path.
Storage: IndexedDB and its price
The asset library has to survive across sessions, so it needs persistence. localStorage will not do — strings only, and usually capped at a few megabytes. IndexedDB stores structured objects and is sized against available disk, making it the only reasonable choice.
But it has a property that must be stated plainly to users: clearing browser data deletes it. Nobody expects “clear cache” to destroy their own work.
Which makes export not a nice-to-have but a required component of this architecture. Choosing to keep data on the user’s device obliges you to give them a way to take it out, or one mis-click is permanent loss.
IndexedDB may also be unavailable in private browsing. That case needs an explicit message rather than a save that silently fails.
ZIP export needs no compression library
Exporting multiple files invites a ZIP. Pulling in a compression library costs tens of kilobytes and buys almost nothing on already-compressed PNGs.
ZIP supports a stored mode (compression method 0): files written verbatim, no compression. That reduces the entire writer to three structures — local file header, central directory entry, end-of-central-directory record — plus a CRC32. Around a hundred lines, no dependencies.
CRC32 via a lookup table, built lazily on first use:
var TABLE = null;
function crcTable() {
if (TABLE) return TABLE;
TABLE = new Uint32Array(256);
for (var n = 0; n < 256; n++) {
var c = n;
for (var k = 0; k < 8; k++) {
c = (c & 1) ? (0xEDB88320 ^ (c >>> 1)) : (c >>> 1);
}
TABLE[n] = c >>> 0;
}
return TABLE;
}This is the archetype of using a format’s own capability instead of a dependency. ZIP supports not compressing, and we did not want compression — a compression library would have solved a problem that was never there.
The model download: the one unavoidable cost
The AI path runs a segmentation model on the visitor’s device, and the model has to arrive first. It is the only place in this architecture where the user pays a visible price.
Several things have to be right:
Fetch on demand. The model is only requested when the user actively selects AI mode. The vast majority of visitors are passing through and should not pay bandwidth for a feature they never touch.
Show progress. A download of tens of megabytes with no feedback is indistinguishable from a hang. Read the fetch body as a stream and accumulate:
var reader = res.body.getReader();
var received = 0;
(function pump() {
return reader.read().then(function (r) {
if (r.done) return assemble();
received += r.value.length;
onProgress(received, total); // total from content-length
return pump();
});
}());Let the browser cache it. The model is an immutable static file; give it long-lived cache headers and the second use costs nothing.
Choose precision by worst-case input, not by size. I first shipped an int8 quantised model — smallest file, negligible accuracy loss on my test samples — and it collapsed on images the model was unsure about. Moving to fp16 doubled the size with essentially no accuracy cost. The bandwidth saved was not worth a feature that sometimes does not work.
Image processing on the main thread freezes the whole page
Moving computation to the client runs into a constraint the server never had: the browser’s main thread is responsible for both rendering the interface and handling interaction. A loop running for a few hundred milliseconds on that thread means a few hundred milliseconds of total unresponsiveness — buttons ignore clicks, scrolling stalls, and even the “processing” spinner stops.
Image processing is inherently a long loop. Per-pixel work on a 1024×1024 image is a million iterations, and adding model inference pushes it into seconds.
The remedy is moving the heavy work into a Web Worker — a separate thread whose runtime never affects the interface. The cost is that a worker and the main thread communicate only by messages and cannot share ordinary objects.
One detail determines whether this performs well: transfer large buffers rather than letting them be structurally cloned. By default postMessage copies the data in full, so passing a 4 MB bitmap costs another 4 MB plus the copy time. Listing the underlying ArrayBuffer in the transfer list instead hands over ownership at zero copy cost:
// the second argument is the transfer list; ownership moves to the worker
worker.postMessage({ pixels: imageData.data.buffer, w, h },
[imageData.data.buffer]);
// note: after transfer the sender's buffer has length 0 and is unusableAfter transferring, the originating thread can no longer access that memory — which is not a limitation but precisely why it is free. If you still need the data, copy it first and transfer the copy, so at least the copy is one you chose to make.
OffscreenCanvas additionally hands the canvas itself to a worker, moving drawing and pixel access off the main thread entirely. Support arrived late on mobile Safari, so it needs feature detection and a fallback.
Mobile has a memory ceiling that raises no error
One boundary only reveals itself on real devices: mobile browsers cap the memory a single tab may use, and the system kills the page outright when it is exceeded.
It presents not as an exception or an error but as a blank page or a spontaneous reload. Your try/catch cannot catch it and error reporting never fires, because the execution context is gone. Everything works on desktop, and for the user it is simply “it crashes when I process anything.”
Image processing hits this line readily, because intermediate products are far larger than they look. A 4000×3000 photo may be a 3 MB JPEG, but decoded to ImageData it is 4000 × 3000 × 4 ≈ 46 MiB. Hold the original, a resized copy, a mask and the output simultaneously and four of those approach 180 MiB — already in dangerous territory.
Several constraints that actually help:
- Cap the longest side (to 2048, say) before processing rather than shrinking afterwards. Users cannot tell the difference in most scenarios, and memory drops to a quarter.
- Drop references to large objects as soon as they are done with, especially
ImageDataand temporary canvases. Setting a temporary canvas’s width and height to 0 releases its backing store sooner. - Never hold every intermediate result of a chain at once. Overwrite step by step, keeping only the current step’s input and output.
This can only be verified on a real device. Desktop browsers have gigabytes of headroom and will never surface it — unlike every other problem here, which reproduces in a console, this one requires running a large image on an actual phone.
When not to do this
The approach has clear boundaries.
If results need to be shared between users — generated assets shown to the rest of a team — the data ends up on a server regardless. Client-side processing then saves compute but not storage or moderation.
If the model runs to hundreds of megabytes, or inference takes minutes, the experience is worse than uploading. A rough test: when downloading the model takes many times longer than uploading the source image would, the client-side design stops paying.
If the same data must be reachable from multiple devices, IndexedDB is per-browser and cannot do it. Either accept manual export and import, or accept a server.
But when the shape of the problem is “process once, the result belongs to the user, nothing needs sharing” — image conversion, format work, local analysis — a client-side design saves considerably more than CPU.
A worked example of this pattern is live on the site: the image to ANSI art converter decodes, scales, searches masks and emits the file entirely in the browser, with nothing uploaded.