What an LLM is doing underneath the marketing: next-token prediction driven by the Transformer. Covers the architectural workflow, running an 8B model locally in Python, the hardware pitfalls encountered on the way, and what actually matters for developers building on top of these models.
The crux of multimodal models is modality alignment. Covers the end-to-end architecture, the mathematics behind Q-Former and cross-attention, an MLP projection layer implemented in PyTorch, and why a linear projection and a two-layer MLP behave differently in production.
Deploying LLaVA, Qwen-VL or InternVL runs into the same wall: KV cache memory. Covers the derivation of KV cache consumption, how AnyRes and dynamic resolution mitigate visual token explosion, 4-bit quantised inference with llama.cpp, and the engineering traps behind each.
The mechanics of visual instruction tuning, focusing on loss construction. Covers multimodal auto-regressive next-token prediction, the data flow from JSON to masked labels, the PyTorch label-masking implementation, and the production pitfalls that follow from getting the mask wrong.
Once you have written a small neural network from scratch, the question becomes whether you must hand-code matrix multiplication and backpropagation every time. Covers what frameworks actually provide, the two major camps and how they differ, which one suits a beginner, and the same simple layer defined in each.
Diagnose overfitting and underfitting from training and validation scores, then tune the model through complexity, regularisation, cross-validation and feature engineering.
Choose a machine learning algorithm by task type, data size, interpretability and deployment cost, covering logistic regression, decision trees, random forests, K-means and tabular baselines.
Build a feature engineering pipeline with scikit-learn Pipeline and ColumnTransformer, handling missing values, categorical variables and numeric standardisation while avoiding data leakage.
Hand-calculate early steps on a 2D quadratic, compare gradient descent, momentum, and Adam, and reproduce the loss-contour visualization.
A hand-calculated 5×5 by 3×3 convolution with output-size math, receptive fields, channel mixing, and im2col.
