Deploy a Local LLM, Part 2: Exposing It Safely with a Tunnel or Port Forwarding

This is part two of the Local AI Model Deployment series. At the end of part one you had an inference service on 127.0.0.1:8000 — stable and auto-starting, but reachable only from the machine itself. This part exposes it to the internet safely.

You get two complete routes: a Cloudflare tunnel (no inbound ports at all) and direct port forwarding (your own domain and certificate). They suit different situations, and you can run both — I do.

Local LLM deployment series (4 parts): ① Model and engine → ② Exposing it → ③ Clients → ④ Knowledge base RAG. This is part 2.

Decide which route first

	Cloudflare tunnel	Direct forwarding
Public IP required	No	Yes, and not CGNAT
Inbound ports required	No	Router forwarding needed
TLS certificate	Handled at the edge	You issue and renew it
Latency	Depends on tunnel landing PoP	Depends on client-to-home path
Attack surface	Cloudflare in front	Exposed; your rate limits only
Setup complexity	Low	Medium

Do the tunnel first. It depends on no network conditions, takes about ten minutes, and gives you a working fallback. Add direct forwarding later if latency turns out to be your bottleneck.

Step 1: Put nginx in front

Both routes need this layer. It handles three things the inference server does not: rate limiting, blocking paths that should not be public, and routing prefixes to different backends.

# Rate limiting exists to contain a leaked key, not to throttle normal use.
# Set it far above real traffic — a translation client fires dozens of
# requests for a single page.
limit_req_zone  $binary_remote_addr zone=llm_api:10m rate=1200r/m;
limit_conn_zone $binary_remote_addr zone=llm_conn:10m;

server {
    listen 127.0.0.1:8444;          # local only; the tunnel fronts it
    server_name your-domain.example;

    client_max_body_size 24m;        # multimodal image uploads

    location /v1/ {
        limit_req  zone=llm_api burst=300 nodelay;
        limit_conn llm_conn 64;

        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;
        proxy_set_header Host       $host;
        proxy_set_header X-Real-IP  $remote_addr;
        proxy_set_header Connection "";

        # Streaming responses are SSE. Buffering must be off or the whole
        # answer accumulates and arrives at once.
        proxy_buffering off;
        proxy_cache off;
        proxy_request_buffering off;
        proxy_read_timeout 900s;
        proxy_send_timeout 900s;
    }

    # /metrics exposes prompt length distributions — keep it private
    location = /metrics { return 404; }

    location = /healthz { return 200 "ok\n"; }

    location / { return 404; }
}

Three things people get wrong here:

• Do not omit proxy_buffering off. The symptom is that streaming turns into “long wait, then the whole answer at once” — which looks like a slow model but is nginx accumulating.
• Raise the timeouts. The default 60 seconds cuts long answers off mid-sentence.
• nginx -t passing does not mean the config is live. You must systemctl reload nginx. I have made this mistake more than once.

Route A: Cloudflare tunnel

The machine opens an outbound connection to Cloudflare, and inbound requests arrive through it. No public IP, no inbound ports, no certificate management.

A.1 Install cloudflared

The official apt repository is more reliable than pulling binaries from GitHub:

curl -fsSL https://pkg.cloudflare.com/cloudflare-main.gpg \
  | sudo tee /usr/share/keyrings/cloudflare-main.gpg >/dev/null
echo "deb [signed-by=/usr/share/keyrings/cloudflare-main.gpg] https://pkg.cloudflare.com/cloudflared any main" \
  | sudo tee /etc/apt/sources.list.d/cloudflared.list
sudo apt-get update && sudo apt-get install -y cloudflared
cloudflared --version

Check which cloudflared afterwards. If you previously downloaded a binary by hand, the copy in /usr/local/bin shadows the apt one. A truncated leftover binary cost me an hour: the service segfaulted instantly with no useful diagnostic.

A.2 Create the tunnel

Two credential models with very different blast radius:

Option 1: account credentials on the host (simple)

cloudflared tunnel login          # browser authorization
cloudflared tunnel create my-llm
cloudflared tunnel route dns my-llm llm.your-domain.example

The downside: ~/.cloudflared/cert.pem can create or delete any tunnel in your account and rewrite DNS for the whole zone. Putting that on an internet-facing box deserves a second thought.

Option 2: single-tunnel credentials only (recommended)

Create the tunnel and DNS record on a machine that already has credentials, then transfer only that tunnel’s credential file:

# On the machine holding cert.pem
cloudflared tunnel create my-llm
# note the UUID from the output

# Bind DNS — both arguments below matter
printf '' > /tmp/empty.yml
cloudflared tunnel --config /tmp/empty.yml route dns --overwrite-dns <UUID> llm.your-domain.example

Passing an empty --config and an explicit UUID are both required. Otherwise cloudflared reads the tunnel: field from whatever config.yml is in scope — on a host already running another tunnel, your DNS record gets bound to the wrong tunnel, and the command reports success.

Then move the credential without writing the key to an intermediate file:

cat ~/.cloudflared/<UUID>.json \
  | ssh target-host 'mkdir -p ~/.cloudflared && cat > ~/.cloudflared/<UUID>.json && chmod 400 ~/.cloudflared/<UUID>.json'

A.3 Ingress configuration

~/.cloudflared/config.yml:

tunnel: <UUID>
credentials-file: /home/youruser/.cloudflared/<UUID>.json

protocol: quic
retries: 5
grace-period: 30s

ingress:
  - hostname: llm.your-domain.example
    service: http://127.0.0.1:8444        # the nginx layer above
    originRequest:
      connectTimeout: 30s
      keepAliveTimeout: 900s              # do not cut long answers
      httpHostHeader: llm.your-domain.example
  - service: http_status:404              # catch-all

Validate before starting:

cloudflared tunnel --config ~/.cloudflared/config.yml ingress validate
cloudflared tunnel --config ~/.cloudflared/config.yml ingress rule https://llm.your-domain.example/v1/models

A.4 Run it as a service

[Unit]
Description=Cloudflare Tunnel - LLM gateway
After=network.target nginx.service

[Service]
Type=simple
User=youruser
# Use the resolved path — apt installs to /usr/bin, manual installs to /usr/local/bin
ExecStart=/usr/bin/cloudflared tunnel --config /home/youruser/.cloudflared/config.yml run
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload && sudo systemctl enable --now cloudflared-llm
sudo journalctl -u cloudflared-llm -n 20 --no-pager | grep "Registered tunnel connection"

Registered tunnel connection means you are live. Occasional datagram manager encountered a failure entries are normal reconnects — as long as a fresh registration follows.

A.5 DNS settings

Keep the orange cloud (proxy) enabled. Tunnel traffic must traverse the Cloudflare edge; switching to DNS-only breaks it. This is the exact opposite of the DDNS requirement below, which is easy to mix up.

A.6 Verify

curl -sS -o /dev/null -w "HTTP %{http_code}\n" https://llm.your-domain.example/healthz

One limit worth knowing: Cloudflare’s free tier caps a single request at 100 seconds. Long non-streaming generations can be cut off at the edge. Use stream: true for long output — each chunk resets the timer.

Route B: Direct port forwarding

Tunnel latency depends on where the tunnel lands. If that PoP is far from both you and your clients, every request takes a long detour. Direct forwarding can be dramatically faster.

B.1 Check three things first

# 1. What is your public IP, and is it CGNAT?
curl -4 -s https://1.1.1.1/cdn-cgi/trace | grep '^ip='

An address inside 100.64.0.0/10 means carrier-grade NAT — direct forwarding is impossible, use the tunnel.

# 2. Which ports are actually reachable (probe from outside, not locally)
ssh some-vps 'for p in 80 443 8443 9443 9999; do
  timeout 6 nc -z -w 5 YOUR_PUBLIC_IP $p 2>/dev/null && echo "  $p open" || echo "  $p closed"
done'

Always include a control port that should be closed (9999 above). If the control also reports “open”, your probe method is broken and the whole reading is worthless.

Many residential ISPs filter inbound 80/443, so plan on a non-standard port.

# 3. Is this machine's default gateway the router doing NAT?
ip route show default

This one is easy to skip and causes a genuinely hard-to-diagnose failure, covered below.

B.2 Dynamic DNS

Residential IPs change, so you need DDNS. Any tool works; the configuration details matter:

• Publish only an A record. Skip AAAA unless you have verified inbound IPv6 actually works — plenty of setups have addresses but no reachability.
• Do not detect IPv6 with an echo service. If your machine’s egress goes through a proxy or tunnel, the echo service reports the exit node’s address, and you publish a completely wrong AAAA that is very hard to notice.
• Set the record to DNS-only (grey cloud). Proxying defeats the purpose of DDNS.

B.3 Certificate

With port 80 filtered, HTTP-01 validation is unavailable — use DNS-01:

sudo certbot certonly --non-interactive --agree-tos \
  --dns-cloudflare --dns-cloudflare-credentials /root/.secrets/cloudflare.ini \
  --dns-cloudflare-propagation-seconds 30 \
  -d llm-direct.your-domain.example

If the machine issuing the certificate is not the one serving traffic (a reasonable choice if you would rather not put a DNS token on an internet-facing box), add a renewal hook that syncs it:

# In [renewalparams] of /etc/letsencrypt/renewal/<domain>.conf:
# renew_hook = /usr/local/bin/sync-cert.sh

# sync-cert.sh: root reads the cert, a normal user ships it,
# and the private key never lands in an intermediate file
SRC=/etc/letsencrypt/live/<domain>
ssh_peer() { sudo -u relayuser ssh -o BatchMode=yes target-host "$@"; }

ssh_peer "sudo tee /etc/llm/tls/fullchain.pem >/dev/null" < "$SRC/fullchain.pem"
ssh_peer "sudo tee /etc/llm/tls/privkey.pem  >/dev/null && sudo chmod 600 /etc/llm/tls/privkey.pem" < "$SRC/privkey.pem"
ssh_peer "sudo nginx -t && sudo systemctl reload nginx"

B.4 The public-facing nginx site

Different from the internal layer: this one binds all interfaces, terminates TLS, and exposes only API paths:

server {
    # http2 goes on the listen line — the standalone `http2 on;` directive
    # only exists from nginx 1.25.1; older versions fail to start
    listen 9443 ssl http2;
    server_name llm-direct.your-domain.example;

    ssl_certificate     /etc/llm/tls/fullchain.pem;
    ssl_certificate_key /etc/llm/tls/privkey.pem;
    ssl_protocols       TLSv1.2 TLSv1.3;

    # Deliberately no real_ip_header here.
    # The tunnel path needs CF-Connecting-IP to recover the visitor IP;
    # on this path $remote_addr is already the real client, and copying
    # that config would let anyone forge a header to bypass rate limits.

    location /v1/ {
        limit_req  zone=llm_api burst=300 nodelay;
        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;
        proxy_set_header Host       $host;
        proxy_set_header X-Real-IP  $remote_addr;
        proxy_set_header Connection "";
        proxy_buffering off;
        proxy_request_buffering off;
        proxy_read_timeout 900s;
    }

    location = /healthz { return 200 "ok\n"; }
    location / { return 404; }
}

B.5 Router forwarding

Add one rule: external port → this machine’s LAN IP and port, protocol TCP.

B.6 When forwarding is correct but nothing connects

This is where direct forwarding most often stalls, and it deserves its own section.

Symptom: every setting is right, the service is listening, firewalls are off, and external clients still cannot connect. A packet capture shows something bizarre:

sudo tcpdump -i any -nn "tcp port 9443 and not host 127.0.0.1" -c 20

<client> > <host>:9443    Flags [S]     ← SYN arrives
<host>:9443 > <client>    Flags [S.]    ← SYN-ACK is sent
<client> > <host>:9443    Flags [S]     ← client keeps retransmitting
<client> > <host>:9443    Flags [S]

Packets arrive, replies are sent, the client receives nothing. The cause is that this machine’s default gateway is not the router performing NAT — a common situation when the default route points at another LAN device to reach a proxy. Replies leave by that other path, the router has no NAT entry for the connection, and they are dropped.

The fix is conntrack marking plus policy routing, so replies return the way they came:

IFACE=your-nic      ROUTER=192.168.1.1     PORT=9443
MARK=0x9443         TABLE=100              LAN=192.168.1.0/24
ROUTER_MAC=xx:xx:xx:xx:xx:xx

# 1. External requests: source outside the LAN
iptables -t mangle -A PREROUTING -i "$IFACE" ! -s "$LAN" \
  -p tcp --dport "$PORT" -m conntrack --ctstate NEW \
  -j CONNMARK --set-mark "$MARK"

# 2. NAT hairpin: a LAN device using the public hostname still has a LAN
#    source IP, but it did arrive via the router's DNAT — identify it by
#    the router's source MAC
iptables -t mangle -A PREROUTING -i "$IFACE" \
  -m mac --mac-source "$ROUTER_MAC" \
  -p tcp --dport "$PORT" -m conntrack --ctstate NEW \
  -j CONNMARK --set-mark "$MARK"

# 3. Restore the mark onto subsequent packets, including the SYN-ACK
iptables -t mangle -A PREROUTING -i "$IFACE" -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark

# 4. Marked packets consult a table whose default route is the router
ip route replace "$LAN" dev "$IFACE" scope link table "$TABLE"
ip route replace default via "$ROUTER" dev "$IFACE" table "$TABLE"
ip rule add fwmark "$MARK" lookup "$TABLE"

Three details that will bite:

• --restore-mark belongs to the CONNMARK target, not the connmark match. Writing -m connmark --restore-mark fails with unknown option.
• Rule 1’s ! -s $LAN is not optional. Without it, replies to LAN clients connecting to the internal IP also get pushed into that default-route-only table and vanish — every internal direct connection times out.
• Rule 2 is not optional either, or devices at home using the public hostname will fail. With it, one client-side rule works both at home and away.

Install this as a systemd unit (Type=oneshot with RemainAfterExit=yes) rather than running it once — ip rule and iptables do not survive a reboot.

Choosing: a measured comparison

The same system and the same request, measured from two locations:

Vantage point	Tunnel	Direct
Home network (client in-country)	4.49s	2.61s
Overseas server	0.46s	6.9s (occasional 40s timeout)

The conclusions are opposite. That overseas machine happens to sit beside the tunnel’s landing PoP, so the tunnel is a local hop for it; meanwhile reaching a residential connection from abroad crosses international links that are slow and unreliable.

So the question is not “which is faster” but “where is my client“. Configuring both and switching by context is the least painful answer.

Three traps when measuring latency

These will produce confidently wrong conclusions:

1. Proxy software on the measuring machine that routes by domain. It proxies your request out and back. Check the source IP in the server’s access log — if it is not your machine’s address, discard the measurement. curl --interface <physical-nic> bypasses it.
2. Testing NAT hairpin from a machine at home. That path behaves nothing like the real external path, especially under concurrency.
3. Load-testing concurrency from a machine running a proxy. I measured 5/20 concurrent requests succeeding locally while a clean server hit the same backend 20/20 at the same moment — the bottleneck was the test client.

Verification checklist

# Work outward layer by layer so a break is obvious
curl -s 127.0.0.1:8000/health                        # 1. inference server
curl -s 127.0.0.1:8444/healthz                       # 2. inner nginx
curl -s https://llm.your-domain.example/healthz      # 3. tunnel
curl -sk https://llm-direct.your-domain.example:9443/healthz   # 4. direct

# Test all three auth states, and test them from outside
curl -o /dev/null -w "%{http_code}\n" -X POST https://.../v1/chat/completions -d '{}'   # expect 401
curl -o /dev/null -w "%{http_code}\n" -H 'Authorization: Bearer wrong' ...              # expect 401
curl -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $KEY" ...               # expect 200

Auth must be verified from outside; local tests cannot catch the failure. While adding a forwarding layer to my own gateway I wrote a hole: when no Authorization header was present, it fell back to the server’s own key. Local testing always passed. Only an unauthenticated request from the public internet revealed that the path had no authentication at all.

Next

Your service is now reachable from anywhere. Part three covers wiring it into daily tools — browser translation extensions, code editors, and your own clients — including a technique that cuts token usage by 90%.