Deploy a Local LLM, Part 1: Pick the Right Model and Engine for Your VRAM

This is part one of the Local AI Model Deployment series. By the end you will have an inference server running on your own machine: the right model and engine for your VRAM, installed, running as a system service with automatic restart, and verified with a single curl.

Examples use Gemma 4, but the sizing method and the debugging approach apply to any open-weight model. The only prerequisite is an NVIDIA GPU with working drivers — if nvidia-smi prints something, you are ready.

Local LLM deployment series (4 parts): ① Model and engine → ② Exposing it → ③ Clients → ④ Knowledge base RAG. This is part 1.

Step 1: Find out how much VRAM you actually have

nvidia-smi --query-gpu=name,memory.total,compute_cap --format=csv

name, memory.total [MiB], compute_cap
NVIDIA GeForce RTX 4070 Laptop GPU, 8188 MiB, 8.9

All three numbers matter:

• VRAM caps model size. Note that 8188 MiB is an 8 GB card — do not floor 8188/1024 = 7.99 to 7 GB, because one tier down means a different model.
• Compute capability decides FP8 availability: you need ≥ 8.9 (Ada / Hopper / Blackwell). An A100 is 8.0 and cannot use it.
• Subtract what is already in use. A machine driving a display typically has a few hundred MB gone before you start:

nvidia-smi --query-gpu=memory.used,memory.free --format=csv

Step 2: Look up real file sizes — do not estimate

This step gets skipped most often and causes the most damage. Parameter count does not predict VRAM. A quantized checkpoint may quantize only some layers, making it far larger than “parameters × bit width” suggests.

Ask the HuggingFace API for actual bytes:

check_size() {
  curl -s "https://huggingface.co/api/models/$1?blobs=true" | python3 -c "
import json,sys
d=json.load(sys.stdin)
w=[(s['rfilename'], s.get('size') or 0) for s in d.get('siblings',[])
   if s['rfilename'].endswith(('.safetensors','.gguf'))]
main=sum(s for n,s in w if 'mmproj' not in n)
mm=sum(s for n,s in w if 'mmproj' in n)
print(f'  main {main/1e9:.2f} GB' + (f'  + mmproj {mm/1e9:.2f} GB' if mm else ''))
"
}

check_size google/gemma-4-E4B-it-qat-w4a16-ct
check_size google/gemma-4-E4B-it-qat-q4_0-gguf

The result is surprising:

gemma-4-E4B-it-qat-w4a16-ct     main 11.51 GB
gemma-4-E4B-it-qat-q4_0-gguf    main  5.15 GB  + mmproj 0.99 GB

Two “4-bit quantized” formats of the same model differ by more than 2×. The w4a16-ct checkpoint quantizes only linear layers and keeps embeddings in bf16; GGUF q4_0 quantizes the embeddings too. The pitfalls article has the full breakdown.

Step 3: Pick a model and engine by VRAM

This table is based on measured file sizes, with headroom left for KV cache and compute buffers:

Available VRAM	Model	Engine
≥ 72 GB	31B bf16	vLLM
≥ 40 GB and compute ≥ 8.9	31B FP8	vLLM
≥ 28 GB	31B W4A16	vLLM
≥ 22 GB	31B q4_0 GGUF	llama.cpp
≥ 18 GB	26B-A4B q4_0 GGUF	llama.cpp
≥ 13 GB	12B W4A16	vLLM
≥ 9 GB	12B q4_0 GGUF	llama.cpp
≥ 7 GB	E4B q4_0 GGUF	llama.cpp
≥ 5 GB	E2B q4_0 GGUF	llama.cpp

Why the engine changes

This is not a preference — it is a question of what physically fits:

• Plenty of VRAM: use vLLM. Continuous batching and PagedAttention give substantially better throughput under concurrency, which matters for shared or high-frequency use.
• Tight VRAM: use llama.cpp with GGUF. GGUF quantizes embeddings as well, making it the only format that fits.

The good news is that both expose an OpenAI-compatible API, so your reverse proxy, clients, and documentation stay identical. Everything in the rest of this series works with either engine.

Step 4: Install the engine

Path A: llama.cpp (tight VRAM)

The official releases ship no CUDA build for Linux, but there is a Vulkan build — it runs fine on NVIDIA cards and saves you several GB of CUDA toolkit, since the driver already provides a Vulkan ICD:

# Confirm the driver exposes Vulkan
ls /usr/share/vulkan/icd.d/nvidia_icd.json

# Resolve the latest tag and download the Vulkan build
TAG=$(curl -fsSL https://api.github.com/repos/ggml-org/llama.cpp/releases/latest \
      | python3 -c 'import json,sys; print(json.load(sys.stdin)["tag_name"])')

curl -fL -o /tmp/llama.tar.gz \
  "https://github.com/ggml-org/llama.cpp/releases/download/${TAG}/llama-${TAG}-bin-ubuntu-vulkan-x64.tar.gz"

# Verify the archive before extracting — truncated downloads are common
tar tzf /tmp/llama.tar.gz >/dev/null || { echo "archive is corrupt"; exit 1; }

sudo mkdir -p /opt/llama.cpp
sudo tar xzf /tmp/llama.tar.gz -C /opt/llama.cpp --strip-components=1

Confirm it sees the GPU:

LD_LIBRARY_PATH=/opt/llama.cpp /opt/llama.cpp/llama-server --list-devices

Available devices:
  Vulkan0: NVIDIA GeForce RTX 4070 Laptop GPU (8188 MiB, 7756 MiB free)

Path B: vLLM (plenty of VRAM)

curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv /opt/llm/venv --python 3.12
uv pip install --python /opt/llm/venv/bin/python 'vllm>=0.19.0' huggingface_hub

Fair warning: vLLM pulls in the full CUDA runtime stack — the dependency tree is well over 10 GB. Budget time for the first install.

Step 5: Download weights

Depending on your region this is often the slowest step. Measured throughput varies enormously between mirrors:

ModelScope     16.7 MB/s
hf-mirror       2.4 MB/s
HuggingFace     2.1 MB/s

If you are in Asia, ModelScope is usually fastest — note the path uses master, not main:

mkdir -p /opt/llm/models

curl -L --retry 20 --retry-delay 5 --retry-all-errors -C - \
  -o /opt/llm/models/gemma-4-E4B_q4_0-it.gguf \
  "https://modelscope.cn/models/gpustack/gemma-4-E4B-it-qat-q4_0-gguf/resolve/master/gemma-4-E4B_q4_0-it.gguf"

-C - enables resume, so rerunning after an interruption picks up where it left off.

Always verify the size afterwards. Download tools reporting success while fetching nothing is a real and common failure:

ls -l /opt/llm/models/*.gguf | awk '{printf "%.0f MB  %s\n", $5/1048576, $9}'

Multimodal models need an extra projector file (usually named with mmproj). Without it the model is text-only.

Step 6: Wrap it in a systemd unit

Starting the process with nohup means it dies on reboot, never restarts after a crash, and scatters its logs. A systemd unit solves all three:

[Unit]
Description=Local LLM inference server
After=network-online.target
Wants=network-online.target

[Service]
Type=exec
User=youruser
Environment=LD_LIBRARY_PATH=/opt/llama.cpp
ExecStart=/opt/llama.cpp/llama-server \
  --model /opt/llm/models/gemma-4-E4B_q4_0-it.gguf \
  --mmproj /opt/llm/models/gemma-4-E4B-it-mmproj.gguf \
  --alias gemma4 \
  --host 127.0.0.1 --port 8000 \
  --ctx-size 131072 \
  --parallel 4 \
  --n-gpu-layers 99 \
  --flash-attn on \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --jinja \
  --metrics \
  --no-webui \
  --api-key sk-replace-me
Restart=on-failure
RestartSec=10
# Loading a large model can take minutes; do not let systemd declare it dead
TimeoutStartSec=1800
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/opt/llm

[Install]
WantedBy=multi-user.target

The flags that matter

Several defaults here are counter-intuitive and worth spelling out:

• --ctx-size is a total, not per-request context. It is divided by --parallel. Above, 131072 across 4 slots gives each request 32K. If you read it as “per request” and set 16384 with the default parallelism of 4, the engine allocates KV cache for 65536 tokens — VRAM fills, model layers spill to CPU, and throughput drops by an order of magnitude.
• --parallel is your concurrency ceiling. Translation clients fire dozens of short requests at once, so slots matter more than depth. Coding assistants are the opposite.
• --flash-attn on is mandatory for some models, not an optimization. Models using sliding window attention (Gemma 4’s E-series does) run prompt processing 40× slower on the Vulkan backend without it.
• --cache-type-k/v q8_0 compresses the KV cache to 8 bits, roughly doubling usable context on a small card with negligible quality impact.
• --api-key is not optional. Even bound to localhost, set one now — you will thank yourself when you expose the service later.

sudo systemctl daemon-reload
sudo systemctl enable --now llm-server

Step 7: Verify

7.1 Is it running

systemctl is-active llm-server
curl -s 127.0.0.1:8000/health

7.2 Read the values the engine actually computed

This matters more than reviewing what you wrote. The startup log shows what was really allocated:

sudo journalctl -u llm-server --no-pager | grep -iE "n_slots|n_ctx|loaded"

srv load_model: initializing, n_slots = 4, n_ctx_slot = 32768
srv load_model: loaded multimodal model, '.../gemma-4-E4B-it-mmproj.gguf'

n_slots × n_ctx_slot is the true KV cache size. For multimodal models this line also confirms the projector loaded.

7.3 Does VRAM usage match expectations

nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader

Reference point: E4B q4_0 plus mmproj plus a 128K q8_0 KV cache measures 5.2 GB. If yours is much higher than your estimate, the --ctx-size ÷ --parallel relationship is the usual culprit.

7.4 Send a real request

KEY=sk-replace-me
curl -s 127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"gemma4","max_tokens":2048,
       "messages":[{"role":"user","content":"Explain KV cache in one sentence"}]}' \
  | python3 -m json.tool

If content comes back as an empty string, do not assume the model is broken — check for a reasoning_content field. Models with a thinking mode (Gemma 4 included) emit reasoning before the answer, and a small max_tokens truncates before the answer starts. Raise it above 2048 and retry.

7.5 Measure real throughput

The first measurement does not count. The Vulkan backend compiles shaders lazily, so the first large prompt after a cold start is an order of magnitude slower. Warm up twice with a few-hundred-token input first:

python3 -c "
import json
p = 'Summarize the following. ' + 'Machine learning has evolved through several eras. ' * 20
print(json.dumps({'model':'gemma4','max_tokens':200,
                  'messages':[{'role':'user','content':p}]}))" > /tmp/warm.json

for i in 1 2; do
  curl -s -o /dev/null 127.0.0.1:8000/v1/chat/completions \
    -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' --data @/tmp/warm.json
done

curl -s 127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' --data @/tmp/warm.json \
| python3 -c "
import json,sys
t = json.load(sys.stdin).get('timings', {})
print(f\"prompt {t.get('prompt_per_second',0):.0f} t/s   generation {t.get('predicted_per_second',0):.1f} t/s\")"

Reference for E4B on an 8 GB laptop GPU: 900–1700 t/s prompt, 63–66 t/s generation. If prompt processing sits in the tens, check whether --flash-attn is enabled.

Troubleshooting

The service will not start

sudo journalctl -u llm-server -n 50 --no-pager

Three common causes:

• unrecognized arguments — the engine version changed and a flag was removed or renamed. Drop it and retry.
• failed to load model — incomplete weights. Go back to step 5 and check the file size.
• An immediate out-of-memory exit — reduce --ctx-size or pick a smaller model.

Can I run two models at once

Yes — two systemd units on different ports. But VRAM is shared: both sets of weights plus both KV caches must fit. If the second model is only doing embeddings, run it on CPU instead — that approach is used in part four of this series.

Why bind to 127.0.0.1 only

Deliberately. Right now the only protection is an API key; binding to 0.0.0.0 would put it bare on the network. Part two covers putting a reverse proxy in front and exposing it safely, including rate limiting, path routing, and TLS.

Next

You now have a stable, auto-starting local inference service reachable from the machine itself. Part two covers two ways to expose it to the internet — a zero-open-port tunnel and direct port forwarding — and which one suits which situation.

If something behaves strangely along the way, this companion article catalogues 11 real failures by symptom.