Adding web search to a local model meant writing a textbook egress guard. Search then returned empty results, reliably — the guard was refusing Wikipedia and the search API, and its judgement was entirely correct. The cause was synthetic addresses from a transparent proxy landing in an RFC 2544 reserved range. That check-versus-use gap turned up three times in one file.
A lookup table cannot cover variant × format × context × card count, so you model it. This records that modelling: what each of the three terms accounts for, how to measure the global-attention-layer share you cannot look up by reading a slope, how the forgotten concurrency multiplier makes inference two orders of magnitude slower, and two hard constraints beyond VRAM — tensor-parallel divisibility and why recommendations cannot be ranked by size.
Audit four ONNX deployment gates: complete assets, executable module types, configuration timing and threading conditions. Includes four HTTP 200 transfer controls, two real WASM model contract probes, complete output tensors and runnable source, with explicit limits on accuracy and timing claims.
Separate prediction agreement from labelled quality, and distinguish local debugging checks from proof of a unique cause. Includes five 4×4 mask controls, raw metric records and runnable threshold examples. Historical model numbers are retained with explicit reproduction limits.
Architecture, official benchmarks, real checkpoint sizes, and measured VRAM for E2B, E4B, 12B Unified, 26B A4B, and 31B. What PLE effective parameters, the encoder-free Unified design, and MoE active parameters each solve, and why E4B quantized is larger than 12B.
Two complete routes for going public: a Cloudflare tunnel needing no inbound ports, and direct port forwarding with your own domain and certificate. Covers the nginx layer, DNS-01 issuance, DDNS caveats, and fixing a correct port forward that still cannot complete a handshake.
Making the model useful: disable thinking mode to cut token usage by 90%, add a shim for tools that cannot customize the request body, declare multimodal capabilities correctly, plus a symptom-indexed troubleshooting table.
Build RAG on the same machine: run embeddings on CPU using no VRAM, chunk by heading then by size, combine sqlite-vec vector search with FTS5 lexical search via RRF, and wrap it as an OpenAI-compatible endpoint that returns citations.
Start from nvidia-smi, look up real checkpoint sizes instead of estimating, choose vLLM or llama.cpp by VRAM tier, install the engine, download weights, run it as a systemd service, and verify real throughput with curl.
A field report from building a local Gemma 4 inference gateway on 8GB of VRAM: why QAT checkpoints are 2x larger than parameter counts suggest, how one flag changes prompt throughput 40x, a port-forward that fails because replies leave by the wrong gateway, and four traps in RAG chunking and CJK full-text search.
