Self-Hosting Gemma 4 on an 8GB GPU: 11 Pitfalls You Only Hit in Production

Most self-hosting guides make local LLM deployment sound like three steps: download the weights, start an inference server, put a reverse proxy in front. Doing it for real reveals a gap between it runs and it runs correctly — a gap filled with failures that never raise an error. Flags get silently ignored. A download command exits 0 having fetched nothing. A service listens on the right port and still never receives your packets.

This is a field report from building a Gemma 4 inference gateway on a laptop GPU with 8 GB of VRAM — model selection, inference engine, public exposure, translation integration, and finally retrieval-augmented generation. It is not a step-by-step tutorial. It is the list of things that only bite once you actually run the thing, each with the measurement that exposed it.

1. Parameter count is a bad proxy for VRAM: official quantized checkpoints can be 2× larger than you expect

The easiest mistake during model selection is estimating VRAM as “parameters × bytes per parameter”. Gemma 4’s official QAT checkpoints carry a -qat-w4a16-ct suffix, so intuition says a 4B-class model should need 2–3 GB.

Query the actual safetensors byte counts instead:

gemma-4-E4B-it-qat-w4a16-ct     11.51 GB
gemma-4-12B-it-qat-w4a16-ct     10.26 GB
gemma-4-31B-it-qat-w4a16-ct     23.27 GB

E4B — nominally “4.5B effective parameters” — is larger than the 12B model. The reason is in config.json:

{
  "quantization_config": {
    "config_groups": {
      "group_0": {
        "targets": ["Linear"],
        "weights": { "num_bits": 4, "group_size": 32 }
      }
    }
  }
}

targets contains only Linear. Only linear layers are quantized to 4 bits; embeddings and per-layer embeddings (PLE) stay in bf16. The E-series uses a PLE architecture whose table is vocab_size × layers × per_layer_dim — for E4B that is 262144 × 42 × 256 ≈ 2.8B parameters, or 5.6 GB in bf16 by itself.

A more immediately useful corollary: a 24 GB card cannot run the 31B w4a16-ct checkpoint. The weights alone are 23.27 GB, leaving nothing for KV cache.

How to avoid it

Do not trust parameter counts. Ask the API for real byte counts:

curl -s "https://huggingface.co/api/models/<org>/<repo>?blobs=true" | python3 -c "
import json,sys
d=json.load(sys.stdin)
w=[(s['rfilename'], s.get('size') or 0) for s in d.get('siblings',[])
   if s['rfilename'].endswith(('.safetensors','.gguf'))]
print(f\"{sum(s for _,s in w)/1e9:.2f} GB\")"

GGUF q4_0 quantizes the embeddings too, so the same E4B model is 5.15 GB — the only form that fits in 8 GB. This is why the engine choice ends up VRAM-dependent: vLLM when memory is plentiful (better throughput and batching), llama.cpp + GGUF when it is tight.

2. The download command exits 0 and fetches nothing

My prefetch step looked like this:

hf download "$MODEL_ID" --exclude '*.pth' '*.gguf' 'original/*'

It printed ✓ Downloaded, returned exit code 0, the script moved on — and the inference server failed to start.

--exclude takes exactly one glob. The remaining two patterns were parsed as filenames to download, so the tool fetched two nonexistent files and skipped every real weight. The warning is there, but it scrolls past inside the progress output:

UserWarning: Ignoring `--exclude` since filenames have been explicitly set.
Fetching 0 files: 0it [00:00, ?it/s]

The correct form repeats the flag:

hf download "$MODEL_ID" \
  --exclude '*.pth' --exclude '*.gguf' --exclude 'original/*'

The broader lesson matters more than the flag: any command that can “succeed” while doing nothing needs an independent check afterwards. I added this to the deployment script:

# The command above can return 0 having downloaded nothing.
# Verify by measuring what actually landed on disk.
WEIGHT_MB="$(du -sm "$HF_HOME_DIR" | cut -f1)"
[ "${WEIGHT_MB:-0}" -ge 500 ] || die "only ${WEIGHT_MB} MB on disk — download is incomplete"

3. One flag changes prompt processing by 40×

Once the server was up, generation ran fine at 60+ tok/s — but prompt processing crawled. A 544-token input took over ten seconds.

Benchmarking the hardware directly with llama-bench gave 3254 tok/s on pp512, so the GPU was not the problem. Bisecting the server flags found it:

configuration                prompt processing (544 tokens)
baseline (no flash attn)          39.6 t/s
+ flash-attn on                 1726.5 t/s
+ q8_0 KV cache                 1654.4 t/s

A greater than 40× difference. Gemma 4’s E-series uses sliding window attention (SWA), and without flash attention the Vulkan backend falls back to a dramatically slower generic path.

What makes this a trap: most documentation frames --flash-attn as an optional performance tweak. For this model and this backend, it is mandatory.

4. --ctx-size is a total, and parallel slots divide it

I initially read --ctx-size as “context per request” and set it to 16384. llama.cpp’s --parallel defaults to 4, so it actually allocated KV cache for 65536 tokens. VRAM filled, model layers spilled to CPU, and prompt processing dropped to 1.3 tok/s.

The startup log says so, but quietly:

srv load_model: initializing, n_slots = 4, n_ctx_slot = 16384

n_slots × n_ctx_slot is the real KV cache size. The correct mental model:

# --ctx-size is the total; per-request context = ctx_size / parallel
--ctx-size 131072 --parallel 4    # 4 concurrent slots, 32K each

That division is also a useful tuning knob: translation clients fire dozens of concurrent short requests, so slot count matters more than depth. Coding assistants are the opposite.

5. Vulkan compiles shaders lazily — your first measurement is garbage

After a restart, the first large prompt measured 21 tok/s. The identical request measured 900+ tok/s on the second try. I spent a while changing configuration that was never wrong.

The cause: llama.cpp’s Vulkan backend compiles compute pipelines on first use. Warming up with a few tokens does not help — that never touches the large-batch matmul pipelines. A realistic warmup needs a prompt of several hundred tokens.

The danger is contamination of every comparison you run: if configuration A is measured first and B second, A silently absorbs the compilation cost.

6. An “empty answer” is really the reasoning trace eating your token budget

Gemma 4 has a thinking mode. llama.cpp parses the reasoning segment into message.reasoning_content; only the final answer lands in message.content.

Which produces responses like this:

{
  "choices": [{
    "finish_reason": "length",
    "message": {
      "content": "",
      "reasoning_content": "Thinking Process:\n\n1. **Analyze the Request:** ..."
    }
  }],
  "usage": { "completion_tokens": 150 }
}

content is an empty string — not null, not an error. Any client that reads only content reports “the model returned nothing”.

Two mitigations:

• Budget max_tokens generously. Reasoning counts against completion tokens; a small budget truncates before the answer begins. 2048+ for non-trivial questions.
• Handle reasoning_content in the client. Streaming delivers it as delta.reasoning_content. Ignore it and your UI freezes for tens of seconds while the model thinks, which looks indistinguishable from a hang.

Disabling reasoning: one method works, one does not

Translation, summarization, and classification do not benefit from reasoning. Two approaches, measured:

// No effect — still produced 1239 characters of reasoning
{ "reasoning_budget": 0 }

// Works — zero reasoning
{ "chat_template_kwargs": { "enable_thinking": false } }

The difference on a single technical paragraph translated to Chinese:

reasoning on:   1762 chars of reasoning, 560 output tokens
reasoning off:     0 chars of reasoning,  61 output tokens

A 9× token difference with no measurable quality loss.

7. A rate limit tuned by intuition reads as a network failure

After wiring the gateway into a browser translation extension, it reported: “Network connection failure (may be caused by server unresponsiveness, network instability or security policy restrictions).”

The server logs told a different story:

[error] limiting requests, excess: 40.800 by zone "api_zone",
        request: "POST /v1/chat/completions"

I had configured rate=120r/m (2 per second) with burst=40. Translating one page fires dozens of requests at once; the burst filled instantly and everything after it got a 503.

The conceptual error: I was using rate limiting as a throttle. Its job is to contain abuse if a key leaks, so the ceiling belongs far above real traffic. What legitimately bounds normal use is the backend’s slot count — excess requests queue there rather than being dropped.

8. Port forwarding is correct, the SYN-ACK goes out, and the client keeps retransmitting

This was the hardest failure to diagnose. Exposing the API directly to the internet, the forwarding rule was entirely correct — external port, internal IP, internal port, protocol — and external clients still could not complete a TCP handshake.

The packet capture was baffling:

<client> > <server>:9443    Flags [S]     ← SYN arrives
<server>:9443 > <client>    Flags [S.]    ← SYN-ACK is sent
<client> > <server>:9443    Flags [S]     ← client retransmits
<client> > <server>:9443    Flags [S]

Packets arrive, replies are sent, the client receives nothing. The cause: this machine’s default gateway is not the router performing NAT. To route its outbound traffic through a proxy, its default route pointed at a different device on the LAN. Replies left through that other path, the router had no NAT entry for the connection, and the packets were dropped.

Another server on the same LAN had working port forwarding all along — precisely because its default gateway was the router.

Fix: conntrack marking plus policy routing

Force replies for router-originated connections back out through the router:

# 1. Mark new connections that arrived via the router's DNAT
iptables -t mangle -A PREROUTING -i "$IFACE" ! -s "$LAN_NET" \
  -p tcp --dport "$PORT" -m conntrack --ctstate NEW \
  -j CONNMARK --set-mark "$MARK"

# 2. Restore the mark onto subsequent packets, including the locally
#    generated SYN-ACK
iptables -t mangle -A PREROUTING -i "$IFACE" -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark

# 3. Marked packets consult a table whose only route is via the router
ip route replace default via "$ROUTER" dev "$IFACE" table 100
ip rule add fwmark "$MARK" lookup 100

Two details are easy to get wrong.

First, --restore-mark belongs to the CONNMARK target, not the connmark match. Writing -m connmark --restore-mark fails outright with unknown option.

Second, the marking rule must exclude LAN sources (! -s $LAN_NET). Without it, replies to LAN clients connecting to the internal IP also get shoved into that default-route-only table, sent to the router, and lost — every internal direct connection times out. That “fix one thing, break another” failure only surfaces if you regression-test after the fix.

Then there is NAT hairpinning

With the LAN exclusion in place, internal devices reaching the service via its public hostname (NAT hairpin) broke instead. That traffic has a LAN source address, so the first rule misses it, yet it genuinely arrives through the router’s DNAT and needs the same return path.

The distinguishing signal is the source MAC — packets forwarded by the router carry the router’s MAC:

iptables -t mangle -A PREROUTING -i "$IFACE" \
  -m mac --mac-source "$ROUTER_MAC" \
  -p tcp --dport "$PORT" -m conntrack --ctstate NEW \
  -j CONNMARK --set-mark "$MARK"

With that rule, hairpin requests went from 4/6 succeeding at a 8.96 s median to 8/8 at a 0.369 s median.

9. Your measurement machine may be lying to you

While comparing latency I concluded: “the tunnel path takes 1.5–4.8 s, the direct path takes 0.2 s.” That data was worthless.

My machine runs proxy software that routes by domain name. Requests to api.example.com were proxied out to the internet and back — the server’s access log showed a carrier IP as the source, not the machine’s LAN address. The same machine hitting the raw IP went direct.

# The source IP the server sees is the ground truth
sudo tail -5 /var/log/nginx/<site>.access.log

# Force the physical interface, bypassing the TUN device
curl --interface en0 ...

A more general point: “which path is faster” depends entirely on where the client sits. The same system measured from my home network showed the direct path roughly twice as fast; measured from a server abroad, the tunnel was 15× faster — because that machine happened to sit next to the tunnel’s edge PoP. Do not carry a conclusion from one vantage point to another.

10. Four traps when adding RAG

The final addition was retrieval-augmented generation: documents → chunks → embeddings → sqlite-vec + FTS5 → hybrid retrieval → injected as a system message.

10.1 The embedding batch size limit

Ingestion failed immediately with a 500:

input (573 tokens) is too large to process.
increase the physical batch size (current batch size: 512)

llama.cpp’s --ubatch-size defaults to 512 tokens, and a document chunk easily exceeds that. The embedding model itself accepts 8192, so raise the batch to match:

--ctx-size 8192 --batch-size 8192 --ubatch-size 8192

10.2 Split by heading first, by size second

My first implementation split recursively by character count, then labelled each chunk with the last markdown heading it contained. Two consequences:

• Small documents collapsed into a single chunk, destroying retrieval granularity.
• When a chunk spanned several sections, content from the first section was labelled with the last section’s heading — citations pointing at the wrong place, and hard to notice.

The correct order is two-level: split into sections at headings (semantic boundaries), then subdivide oversized sections. Every chunk then belongs to exactly one section.

10.3 Heading detection must skip fenced code

A markdown heading regex happily matches shell comments inside fenced code blocks — they are textually identical:

```bash
# 3. Remember to remove this afterwards
sudo sed -i '/example/d' /etc/hosts
```

That comment became an H1, so a citation breadcrumb read README.md › 3. Remember to remove this afterwards.... The fix is to compute the fenced-block spans first and ignore heading matches that fall inside them.

10.4 CJK full-text search: both the tokenizer and the query need work

Two independent problems.

The tokenizer. SQLite FTS5’s default unicode61 does not segment Chinese; an entire sentence becomes one token, which makes CJK full-text search useless. Use trigram:

CREATE VIRTUAL TABLE chunks_fts USING fts5(
  text, content='chunks', content_rowid='id', tokenize='trigram'
);

The query. I first quoted the whole user question as a phrase. Under trigram, phrase matching is substring matching, so a natural-language question matches nothing at all — the entire full-text arm contributed zero results. The fix is to try the full phrase, then fall back to OR-ing the individual terms (3+ characters).

Why hybrid retrieval is not optional

This deserves emphasis: dense and lexical retrieval have complementary blind spots.

• “How do I stop the model from reasoning?” is a semantic query — only vector search finds it.
• reasoning_budget, or a specific .gguf filename, are identifiers — embeddings are notoriously insensitive to them. In testing, vector search failed to rank the correct passage; lexical search matched it exactly.

Fuse the two with Reciprocal Rank Fusion, which combines ranks rather than scores and therefore needs no normalization between two incomparable scales:

K = 60
score = {}
for rank, cid in enumerate(vec_hits):
    score[cid] = score.get(cid, 0.0) + 1.0 / (K + rank + 1)
for rank, cid in enumerate(fts_hits):
    score[cid] = score.get(cid, 0.0) + 1.0 / (K + rank + 1)

11. The frontend has a silent failure too

The control panel toggles between a login view and the main interface using the hidden attribute. Both rendered simultaneously, pushing the login card out of the viewport.

The cause is CSS specificity:

#app { display: flex; }           /* ID selector — high specificity */
/* User-agent default: [hidden] { display: none } — loses, silently */

The element carried the hidden attribute while its computed style stayed flex. The fix is one explicit rule:

[hidden] { display: none !important; }

What these have in common

In retrospect the failures fall into three families:

• Silent failure — flags ignored, commands exiting 0 without acting, content returning an empty string instead of an error. The countermeasure is to verify every critical step against an independent fact rather than trusting a return code.
• Counter-intuitive defaults — --ctx-size as a total, --parallel defaulting to 4, --ubatch-size at 512, FTS5’s tokenizer ignoring CJK. The countermeasure is to read the values the software actually computed in its startup log, not the ones you believe you set.
• Unreliable measurement — lazy shader compilation, proxy software routing by domain, conclusions that depend on vantage point. The countermeasure is to always include a control: a port that should fail, a known-good baseline, a second measurement location.

One last observation, possibly the most useful: packet capture solved two problems that no amount of log reading or config tweaking would have found. The port forwarding case is the archetype — every setting correct, the service listening, firewalls disabled, and only tcpdump could tell me the packets arrived, the replies were sent, and they left by the wrong door. When something “should work but doesn’t”, capture first. It saves hours of guessing.