Deploy a Local LLM, Part 3: Wiring It into Translation Tools, Editors and Your Own Code

This is part three of the Local AI Model Deployment series. After part two your inference service is reachable from the internet. This part is about actually using it — from browser translation extensions, code editors, and your own programs.

Along the way: one technique that cuts token usage on translation-style work to one ninth, and two failures that look like “the server does not support this” but are really client configuration.

Local LLM deployment series (4 parts): ① Model and engine → ② Exposing it → ③ Clients → ④ Knowledge base RAG. This is part 3.

First, understand thinking mode

Many current open-weight models (Gemma 4, Qwen3, and others) emit a reasoning trace before the answer. Inference engines separate the two:

{
  "choices": [{
    "finish_reason": "length",
    "message": {
      "content": "",
      "reasoning_content": "Thinking Process:\n\n1. Analyze the request..."
    }
  }],
  "usage": { "completion_tokens": 150 }
}

Reasoning counts against completion tokens. In this example all 150 went to thinking, leaving content as an empty string — not null, not an error. A client that reads only content reports “the model returned nothing”.

Two responses:

• Tasks that benefit from reasoning (coding, analysis): set max_tokens above 2048 and handle reasoning_content in the client.
• Tasks that do not (translation, summarization, classification): turn thinking off.

How to turn thinking off

Two approaches; only one works:

// No effect — measured 1239 characters of reasoning anyway
{ "reasoning_budget": 0 }

// Works — zero reasoning
{ "chat_template_kwargs": { "enable_thinking": false } }

The difference, translating one technical paragraph:

A 9× token difference with no quality change.

A dedicated path for tools that cannot customize the request body

Here is the problem: browser translation extensions let you set a base URL, an API key, and a model name — nothing else. There is nowhere to put chat_template_kwargs.

The fix is a thin server-side shim that injects it. Here is a complete FastAPI implementation:

from fastapi import FastAPI, Request, HTTPException
from fastapi.responses import Response, StreamingResponse
import httpx

app = FastAPI()
UPSTREAM = "http://127.0.0.1:8000"
NO_THINK = {"enable_thinking": False}


def upstream_auth(request: Request) -> dict:
    """Pass the caller's Authorization header through unchanged.

    If there is none, send none — let the inference server return 401.
    Never fall back to the server's own key: that turns this path into an
    unauthenticated public entry point.
    """
    auth = request.headers.get("authorization")
    return {"Authorization": auth} if auth else {}


@app.post("/nt/v1/chat/completions")
async def no_think_chat(request: Request):
    try:
        body = await request.json()
    except Exception:
        raise HTTPException(status_code=400, detail="body is not valid JSON")

    # Respect an explicit setting from the caller; only fill in what is missing
    kwargs = body.get("chat_template_kwargs")
    body["chat_template_kwargs"] = (
        {**NO_THINK, **kwargs} if isinstance(kwargs, dict) else dict(NO_THINK)
    )

    headers = upstream_auth(request)
    headers["Content-Type"] = "application/json"

    if not body.get("stream"):
        async with httpx.AsyncClient(timeout=600.0) as c:
            r = await c.post(f"{UPSTREAM}/v1/chat/completions", json=body, headers=headers)
        return Response(content=r.content, status_code=r.status_code,
                        media_type=r.headers.get("content-type", "application/json"))

    async def relay():
        async with httpx.AsyncClient(timeout=600.0) as c:
            async with c.stream("POST", f"{UPSTREAM}/v1/chat/completions",
                                json=body, headers=headers) as r:
                async for chunk in r.aiter_raw():
                    yield chunk

    return StreamingResponse(relay(), media_type="text/event-stream",
                             headers={"Cache-Control": "no-cache",
                                      "X-Accel-Buffering": "no"})

That comment on upstream_auth is a scar. My first version fell back to the server’s own key when no header was present, reasoning that it made local debugging easier. The result was a completely unauthenticated public endpoint. Local testing always passed; only an unauthenticated request from outside exposed it.

Give it an nginx location — and not one that inherits a low panel-style rate limit:

location /nt/ {
    limit_req  zone=llm_api burst=300 nodelay;
    proxy_pass http://127.0.0.1:8080;      # the FastAPI shim
    proxy_http_version 1.1;
    proxy_set_header Host       $host;
    proxy_set_header Connection "";
    proxy_buffering off;
    proxy_request_buffering off;
    proxy_read_timeout 900s;
}

Wiring up a browser translation extension

Using Immersive Translate as the example, choose a custom OpenAI-compatible API:

Use the no-thinking path. With plain /v1 every paragraph burns hundreds of reasoning tokens first — an order of magnitude slower, and the reasoning text can occasionally leak into the page as if it were the translation.

Keep the system prompt explicit so the model executes rather than improvises:

You are a translation engine. Translate to Chinese. Output only the translation.

If the extension reports “network connection failure”

Do not start with the network. Read the nginx error log:

sudo tail -20 /var/log/nginx/<your-site>.error.log

In my case the cause was rate limiting:

[error] limiting requests, excess: 40.800 by zone "llm_api",
        request: "POST /nt/v1/chat/completions"

Translating one page fires dozens of requests at once. I had configured rate=120r/m (2 per second) with burst=40; the burst filled instantly and everything after it got a 503 — which the extension surfaced as a network failure.

Rate limiting exists to contain a leaked key, not to throttle normal use. Set it far above real traffic. What actually bounds normal usage is your backend slot count, where excess requests queue instead of being dropped.

Wiring up a code editor

Most editors supporting custom OpenAI-compatible providers (OpenCode, Continue, Cursor) just need base URL, key, and model name. But there is a trap: many treat a custom provider as text-only unless you declare otherwise, and silently drop image attachments.

Using OpenCode as the example:

{
  "provider": {
    "myllm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "My Local LLM",
      "options": {
        "baseURL": "https://your-domain/v1",
        "apiKey": "sk-..."
      },
      "models": {
        "gemma4": {
          "name": "Gemma 4",
          "attachment": true,
          "modalities": { "input": ["text", "image"] },
          "reasoning": true,
          "tool_call": true,
          "limit": { "context": 32768, "output": 8192 }
        }
      }
    }
  }
}

How to tell who dropped the image

Without attachment and modalities, the client strips the image before sending and substitutes a sentence saying image input is unsupported. The model receives that sentence and dutifully repeats it — which looks like server-side unsupport, when the request never left the client.

One command settles it:

sudo tail -5 /var/log/nginx/<your-site>.access.log

No matching entry means the client blocked it. An entry returning 200 means the server is fine and the problem is how the client parses the response.

A related point that causes confusion

Multimodal models accept images, not PDFs. When a model says it cannot read a PDF, that is usually accurate rather than a bug — the client must rasterize the PDF into images first. So do not list "pdf" in modalities: declaring it makes the client actually send PDFs, which the backend can only reject.

Also, use the plain /v1 path for coding — the no-thinking path is for tasks that need execution, not reasoning.

Writing your own client

The official OpenAI SDK works; only base_url changes. This example incorporates every caveat above:

import os, base64
from openai import OpenAI

client = OpenAI(
    base_url=os.environ.get("LLM_BASE_URL", "https://your-domain/v1"),
    api_key=os.environ["LLM_API_KEY"],
)
MODEL = "gemma4"


def show_model():
    m = client.models.list().data[0]
    # Different engines report context length in different places
    meta = getattr(m, "meta", None) or {}
    ctx = getattr(m, "max_model_len", None) or (
        meta.get("n_ctx") if isinstance(meta, dict) else None)
    print(f"model {m.id}, context {ctx or 'unknown'}")


def chat(prompt: str):
    r = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": prompt}],
        # Reasoning counts as completion tokens; too small and content
        # comes back as an empty string
        max_tokens=2048,
    )
    msg = r.choices[0].message
    print(msg.content or "(empty)")
    if getattr(msg, "reasoning_content", None):
        print(f"  [reasoned for {len(msg.reasoning_content)} chars]")


def stream(prompt: str):
    s = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=2048, stream=True,
    )
    thinking = 0
    for chunk in s:
        if not chunk.choices:
            continue
        d = chunk.choices[0].delta
        # Without handling reasoning_content, nothing appears for tens of
        # seconds and the UI looks frozen
        if getattr(d, "reasoning_content", None):
            if thinking == 0:
                print("[thinking", end="", flush=True)
            thinking += 1
            if thinking % 20 == 0:
                print(".", end="", flush=True)
        if d.content:
            if thinking:
                print("]"); thinking = 0
            print(d.content, end="", flush=True)
    print()


def structured(prompt: str):
    """Force output matching a JSON Schema.

    Use the standard response_format — every engine understands it.
    Do not mix in engine-private parameters (some call it json_schema,
    others guided_json); mixing them makes some engines return empty content.
    """
    schema = {
        "type": "object",
        "properties": {"city": {"type": "string"}, "lat": {"type": "number"}},
        "required": ["city", "lat"],
    }
    r = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=2048,
        response_format={"type": "json_schema",
                         "json_schema": {"name": "loc", "schema": schema}},
    )
    print(r.choices[0].message.content or "(empty)")


def with_image(path: str, question: str):
    mime = "image/png" if path.lower().endswith(".png") else "image/jpeg"
    with open(path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode()
    r = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": [
            {"type": "image_url",
             "image_url": {"url": f"data:{mime};base64,{b64}"}},
            {"type": "text", "text": question},
        ]}],
        max_tokens=2048,
    )
    print(r.choices[0].message.content)

Two small traps when using curl

In streaming responses delta.content is an explicit null rather than a missing key, so d.get("content", "") returns None and prints the literal “None”. Write:

print(d.get("content") or "", end="")

And when sending base64 images, do not interpolate the data into a python3 -c "..." string — base64 contains / + = and nested quoting breaks easily. Pass it by environment variable:

IMG_URL="data:image/png;base64,$(base64 < pic.png | tr -d '\n')" \
python3 -c '
import json, os
print(json.dumps({
  "model": "gemma4", "max_tokens": 2048,
  "messages": [{"role": "user", "content": [
      {"type": "image_url", "image_url": {"url": os.environ["IMG_URL"]}},
      {"type": "text", "text": "Describe this image"}]}]
}))' > /tmp/req.json

curl -sS "$BASE/v1/chat/completions" \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  --data @/tmp/req.json

General notes for other tools

• Open WebUI / LobeChat: choose OpenAI-compatible, fill in base URL and key.
• LangChain: ChatOpenAI(base_url=..., api_key=..., model=...).
• Non-standard ports: many input fields silently strip the port. Save, reopen, and confirm it stuck.
• Model name must match the engine’s alias, not the full repository name.

Troubleshooting table

Next

The model is now part of your daily workflow. Part four adds a knowledge base so it can answer from your own documents — covering chunking strategy, vector store choice, and two problems specific to CJK retrieval.