Large Language Models From a Developer’s Angle: Word Association at Scale

If you’ve been following tech news over the past couple of years, you’ve undoubtedly heard of ChatGPT, Claude, and various “Foundation Models”. They all fall under the umbrella of Large Language Models (LLMs).

As developers, we shouldn’t just treat them as “smart chatbots.” In my personal projects, whether building automated code auditing tools or daily script generators, I heavily rely on LLMs. This article will peel back the mystery and, combined with my real-world experience running open-source models locally, and works through how large language models actually work under the hood.

1. The Core of LLMs: The Brute Force of “Word Association”

Stripping away all the marketing jargon, what large language models fundamentally do is just one thing: **Next-token prediction**.

💡 Core Concept: When you input “The quick brown fox jumps over the lazy”, the model isn’t “understanding” the imagery of the sentence. Instead, through massive matrix multiplications against the terabytes of text data it digested, it calculates that the highest probability next word is “dog”.

Every conversation you have with ChatGPT, the code it writes, the translations it makes—everything is generated by predicting the next token, one by one, based on the context you provide.

2. The Architectural Foundation: The Transformer Workflow

The core architecture supporting this massive calculation is the Transformer, introduced in Google’s famous 2017 paper “Attention Is All You Need.” Here is a simplified flow of how information moves from your prompt to the final output:

graph TD
    A["User Input Text (Prompt)"] --> B["Tokenization"]
    B --> C["Embedding (Map to high-dimensional vectors)"]
    C --> D["Positional Encoding (Inject position data)"]
    D --> E["Transformer Architecture (N layers)"]
    
    subgraph Transformer Architecture
        E1["Self-Attention (Calculate relationships between words)"]
        E2["Feed Forward Network"]
        E1 --> E2
    end
    
    E --> Transformer Architecture
    Transformer Architecture --> F["Softmax (Calculate probability distribution of next token)"]
    F --> G["Sampling (Output the next token)"]
    G --> H{Is it an EOS token?}
    H -- No, add new token to input --> B
    H -- Yes --> I["Output complete response"]

The greatest innovation of the Transformer is the **Self-Attention mechanism**. When you ask, “What is Apple’s profit this year, and is it higher than last year?”, the attention mechanism can accurately calculate that the word “it” points to “Apple” and not “profit”.

3. Hands-on: Running an 8B Model Locally with Python

Too much theory can feel empty, so as programmers, let’s look at some code. The easiest way to run an LLM locally is using llama.cpp. Here is a minimalist inference script I frequently use for daily testing (using the llama-cpp-python library):

from llama_cpp import Llama

# 1. Load the quantized local model file (e.g., Llama-3-8B-Instruct-Q4_K_M.gguf)
# n_gpu_layers=-1 means offloading as many layers to the GPU as possible
llm = Llama(
    model_path="./models/Llama-3-8B-Instruct-Q4_K_M.gguf",
    n_gpu_layers=-1, 
    n_ctx=4096,      # Set the context window to 4K
    verbose=False
)

# 2. Construct the prompt matching the model's specific chat template
prompt = "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nPlease write a quicksort algorithm in Python<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"

# 3. Generate the response (Next-Token Prediction)
output = llm(
    prompt,
    max_tokens=256,
    temperature=0.3, # A lower temperature makes code output more stable
    stop=["<|eot_id|>"]
)

print(output["choices"][0]["text"])

4. My Experience & Hardware “Pitfalls” Guide

When experimenting with local large models, the most common question I get asked is: “Can my computer run model X?”

Through trial and error, I’ve summarized a rule of thumb for calculating **VRAM (Video RAM)** requirements:

• Base Parameter VRAM Footprint: After 4-bit quantization, 1 Billion (1B) parameters take up about 0.7 GB of VRAM. For example, a quantized Llama-3-8B (8 billion parameters) requires roughly 8 * 0.7 = 5.6 GB of VRAM.
• KV Cache Footprint: This is what most people overlook. When processing long contexts, the model needs to cache previous attention matrices. The longer the context, the scarier the memory footprint becomes. Running an 8B model with a full 8K context requires an additional 1~2 GB of VRAM.
• Bottom Line Recommendation: A GPU with 8GB of VRAM (like an RTX 4060) is just enough to smoothly run quantized 7B~8B models. If you want to run medium-sized models (14B~32B) or process massive documents, 16GB of VRAM (like an RTX 4080 16G or a Mac with unified memory) is the bare minimum starting point.

The model does not see characters; it sees tokens

The “word association” metaphor is useful, but it invites the assumption that the model works character by character. What it actually processes are tokens — byte sequences carved out by a tokeniser, which are neither characters nor words.

Roughly how it splits:

"hello world"      ->  ["hello", " world"]           2 tokens
"你好世界"           ->  ["你好", "世界"]               about 2-4
"antidisestablish" ->  ["anti", "dis", "establish"]   3
"3.14159"          ->  ["3", ".", "141", "59"]        4

Common words are single tokens, rare words get split into pieces, and numbers fragment badly. Chinese characters carry high information density, so one character typically costs 1 to 2 tokens — which means the same passage usually costs more tokens in Chinese than in English, directly affecting how much fits in the context window and what a per-token API charges.

Knowing this explains several otherwise puzzling behaviours:

• “8K context” means 8192 tokens, not 8192 characters. For Chinese that is roughly four to five thousand characters — noticeably less than intuition suggests.
• The model cannot count letters within a word. It never sees letters — “strawberry” may be only two or three tokens to it, and letter-level information was discarded at tokenisation. This is a problem of input representation, not reasoning ability.
• Exact arithmetic is unreliable. Numbers split into irregular chunks; “141” and “59” are separate tokens, and the model has no unified representation meaning “this is the number 3.14159.”

Practical advice: never estimate context usage from character counts — run the tokeniser and count. And when exact computation matters, hand it to an external tool rather than asking the model to do it mentally.

What temperature and top_p actually adjust

Running a model locally surfaces a pile of sampling parameters, and most tutorials say only “higher temperature is more creative.” That is too vague to act on.

At each step the model outputs a probability distribution over the entire vocabulary — how likely the next token is to be A, B, and so on. Sampling parameters modify that distribution and how one item is drawn from it.

Temperature acts before the softmax, dividing every score by T:

T = 1.0   original distribution unchanged
T < 1.0   score gaps widen, high-probability items dominate  -> more deterministic
T > 1.0   score gaps compress, distribution flattens          -> more random, more drift
T -> 0    degenerates to always taking the top item (greedy)

top_p (nucleus sampling) works on a different axis: sort tokens by descending probability, accumulate until the total exceeds p, sample only within that set and discard the rest. top_p = 0.9 means “consider only the candidates making up the top 90% of cumulative probability.”

The distinction is that temperature changes relative probabilities while top_p changes the candidate pool. High temperature with a small top_p means “pick randomly among a handful of candidates”; low temperature with a large top_p means “many candidates are eligible but the first is almost always chosen.”

A practical starting point: use T = 0 when you need deterministic output (editing code, extracting structured information), and T = 0.7, top_p = 0.9 for natural conversation. Vary temperature alone at first, and only reach for top_p once you can feel what temperature does — moving both at once leaves you unable to attribute the effect.

5. Takeaways for Developers

Understanding the underlying “token prediction” mechanics and hardware limitations gives us a clearer direction in actual development:

1. Fighting “Hallucinations”: Since the model is essentially guessing the next word based on probabilities, it will inevitably sound very confident while being completely wrong. In engineering, you should never let a bare model operate a core production database directly. It must be paired with RAG (Retrieval-Augmented Generation) technology to feed it external, accurate data as part of its prompt.
2. Prompt Engineering is not magic: The clearer and more logical the prefix information (Context) you provide, the more concentrated the model’s probability distribution for the correct answer becomes. Always provide the model with clear input and output format examples.

We don’t all need to train the next ChatGPT, but learning how to elegantly call APIs or deploy lightweight open-source models locally to solve specific business problems will be a mandatory skill for every developer in the future.