Gradient Descent and Optimizer Geometry: Momentum, Adam, and Loss Surfaces
AI Learning Project
Gradient Descent and Optimizer Geometry: Momentum, Adam, and Loss Surfaces
Hand-calculate early steps on a 2D quadratic, compare gradient descent, momentum, and Adam, and reproduce the loss-contour visualization.
AI Learning Project

Gradient Descent and Optimizer Geometry: Momentum, Adam, and Loss Surfaces

Reading info

Level: Intermediate Reading time: 13 min
  • Gradient Descent
  • Momentum
  • Adam
  • Loss Surface
Open knowledge map

An optimizer decides how parameters move along gradients. To understand gradient descent, momentum, and Adam, it is better to watch their paths on a loss surface than to memorize names. By exploring the geometry of optimization, we can understand why neural networks train efficiently or fail spectacularly.

This article uses a two-dimensional quadratic function. The start point and target are the same, but different optimizers take completely different routes because they treat curvature, history, and scale differently. We will delve into the mathematical underpinnings and the real-world engineering constraints of these algorithms.

1. The Geometry of the Loss Surface and Pathological Curvature

In deep learning, the loss surface is rarely isotropic (perfectly spherical). Instead, it is highly ill-conditioned, filled with ravines and narrow valleys. Let’s look at a canonical function demonstrating this:

L(x, y) = 1/2 * (8x^2 + y^2) + 0.8xy
grad L = [8x + 0.8y, y + 0.8x]

The Hessian matrix of this function has disparate eigenvalues. The surface is steep in the x direction (high curvature) and flatter in the y direction (low curvature). This creates a “pathological curvature” problem. A plain Gradient Descent step will bounce back and forth across the steep ravine, making excruciatingly slow progress along the flat bottom towards the minimum.

Gradient descent, momentum, and Adam paths on a quadratic contour plot
Dense contour lines indicate fast loss change. Different optimizers trace wildly different paths on the same surface due to their handling of pathological curvature.

2. Hand Calculate The First Gradient Descent Step

Starting from (2.2, -2.0), the gradient is:

grad = [8*2.2 + 0.8*(-2.0), -2.0 + 0.8*2.2]
     = [16.0, -0.24]

With a learning rate of 0.08:

x_new = 2.2  - 0.08 * 16.0  = 0.92
y_new = -2.0 - 0.08 * -0.24 = -1.9808

The step taken in the x direction is massive compared to y, purely because the gradient is overwhelmingly larger in x. The lab output confirms it: step 1 for gradient descent is x=0.920000, y=-1.980800, and the loss drops from 17.840000 to 3.889516. However, if the learning rate were just slightly higher, the step in x would overshoot the valley, leading to divergence.

3. What Momentum And Adam Change

Plain gradient descent is memoryless; it only uses the current gradient. This leads to the aforementioned oscillation.

Momentum accumulates a velocity from previous gradients. Think of a heavy ball rolling down a hill. The alternating gradients in the steep x direction cancel each other out, while the consistent gradients in the flat y direction accumulate, accelerating the optimizer toward the minimum.

v_t = beta * v_{t-1} + grad_t
theta_t = theta_{t-1} - lr * v_t

Adam (Adaptive Moment Estimation) goes a step further by maintaining both first (mean) and second (uncentered variance) moments of the gradients. It dynamically scales the learning rate for each parameter individually. By dividing the update by the square root of the accumulated squared gradients, Adam normalizes the step sizes. It essentially forces the optimizer to take larger steps in flat directions and smaller steps in steep directions.

m_t = beta1 * m_{t-1} + (1-beta1) * grad_t
v_t = beta2 * v_{t-1} + (1-beta2) * grad_t^2
theta_t = theta_{t-1} - lr * m_hat / (sqrt(v_hat) + eps)

4. Optimizer Anatomy: Visualized

How do we decide which optimizer to use? The diagram below visualizes the architectural flow of these optimization algorithms.


graph TD
    A[Compute Gradient] --> B{Need history?}
    B -->|No| C[Vanilla SGD]
    B -->|Yes| D{Adaptive Scale?}
    D -->|No| E[SGD with Momentum]
    D -->|Yes| F[Compute 1st & 2nd Moments]
    F --> G[Bias Correction]
    G --> H[Adam / AdamW]
    C --> I[Apply Parameter Update]
    E --> I
    H --> I

5. Practical Python Implementation

To truly grasp these algorithms, we should build them from scratch. Here is a NumPy implementation comparing SGD, Momentum, and Adam on our quadratic surface.

import numpy as np

def grad(theta):
    x, y = theta
    return np.array([8.0 * x + 0.8 * y, y + 0.8 * x])

def run_optimizer(optimizer_name, theta_init, lr=0.08, steps=50):
    theta = np.array(theta_init)
    
    # Optimizer state
    v = np.zeros_like(theta)
    m = np.zeros_like(theta)
    beta1, beta2, eps = 0.9, 0.999, 1e-8
    
    trajectory = [theta.copy()]
    
    for t in range(1, steps + 1):
        g = grad(theta)
        
        if optimizer_name == 'SGD':
            theta -= lr * g
            
        elif optimizer_name == 'Momentum':
            v = 0.9 * v + lr * g
            theta -= v
            
        elif optimizer_name == 'Adam':
            m = beta1 * m + (1 - beta1) * g
            v = beta2 * v + (1 - beta2) * (g ** 2)
            
            # Bias correction
            m_hat = m / (1 - beta1 ** t)
            v_hat = v / (1 - beta2 ** t)
            
            theta -= lr * m_hat / (np.sqrt(v_hat) + eps)
            
        trajectory.append(theta.copy())
        
    return np.array(trajectory)

# Run test from the start point
traj_sgd = run_optimizer('SGD', [2.2, -2.0])
traj_adam = run_optimizer('Adam', [2.2, -2.0], lr=0.5)
print(f"Final Adam position: {traj_adam[-1]}")

6. What The Animation Shows

The animation compares gradient descent, momentum, and Adam on the same loss surface.

Watch whether the steep direction oscillates and whether the flatter direction progresses too slowly. Notice how Momentum swings widely like a pendulum before settling, while Adam cuts a much more direct, controlled path toward the minimum, seamlessly adjusting to the varying curvature.

7. Personal Experience / Engineer’s Perspective

In practice, the elegant math of optimizers runs into harsh hardware and systems realities. Here are a few things I’ve learned from the trenches of training large models:

  • Memory Constraints: Adam is highly effective but extremely memory-hungry. Vanilla SGD only requires memory for the parameters and gradients. Adam requires storing the moving average of gradients (first moment) and the moving average of squared gradients (second moment). This essentially triples the memory footprint of your optimizer state. When training large LLMs on GPUs with limited VRAM, this is often the bottleneck, prompting engineers to use memory-efficient variants like Adafactor or 8-bit Adam.
  • Weight Decay Pitfalls: There is a notorious difference between Adam and AdamW. Standard Adam applies L2 regularization to the gradient before the adaptive scaling. This inadvertently scales down the penalty for weights with high gradient variance, defeating the purpose of weight decay. Always use AdamW for Transformer architectures, which decouples weight decay from the gradient update.
  • Warmup is Mandatory for Adam: Because Adam uses moving averages, the variance term (second moment) is initialized to zero and can be wildly inaccurate in the first few steps, leading to massive, destabilizing updates. A learning rate warmup (starting the LR near zero and scaling up over thousands of steps) prevents the model from blowing up early in training.

8. Practical Notes

  • Plot training and validation loss before changing optimizers. Many training failures are optimizer-path problems rather than architecture problems.
  • The learning rate usually matters more than the optimizer name. A well-tuned SGD with Momentum can often match or beat Adam in generalization, especially in computer vision (ResNets).
  • Adam can still diverge when the learning rate is too high. Do not treat it as a silver bullet that requires no tuning.
  • For noisy loss curves, try lowering the learning rate or adding warmup.

9. Optimization Trace Audit Table

An optimizer experiment should not report only the final loss. To judge whether the path is trustworthy, record coordinates, gradients, state variables, and failure modes together. The audit table below turns “it seems to converge” into reproducible numerical evidence.

Audit item Values to record Question it answers
Initial condition Start point, learning rate, step count, and optimizer hyperparameters. Are the curves being compared under the same conditions?
Step trace x, y, loss, gradient vector, and update size. Is the path descending, or bouncing across the ravine?
State variables Momentum v; Adam m, v, and bias correction. Did history and adaptive scaling actually change the path?
Failure mode Divergent step, oscillation band, oversized early update, and final distance. Does failure come from learning rate, curvature, initialization, or optimizer state?

The next article moves into convolution and shows how local image operations become matrix computations.

Internal links

Related tutorials

Review foundation Backpropagation as a Computation Graph

Trace local gradients through ReLU and softmax cross-entropy in a two-layer MLP.

 Continue next Convolution and Receptive Field Math

Compute convolution output size, receptive fields, channel mixing, and im2col layout.

 Learning path AI Learning Path

Read AI foundations, machine learning, CNN work, traffic defense, and AI security in order.

 Resources Resource Library

Download companion code, datasets, diagrams, model files, and notes.

 Topic hub Student AI Projects

Connect handwritten digits, browser demos, and CNN projects into a portfolio path.

Search questions

FAQ

Who is this article for?

This article is for readers who want an intermediate-level guide to Gradient Descent and Optimizer Geometry. It takes about 13 min and focuses on Gradient Descent, Momentum, Adam, Loss Surface.

What should I read next?

The recommended next step is Convolution and Receptive Field Math, so the article connects into a longer learning route instead of ending as an isolated note.

Does this article include runnable code or companion resources?

Yes. Use the run notes, resource cards, and download links on the page to reproduce the example or inspect the companion files.

How does this article fit into the larger site?

It is connected to the article context block, learning routes, resources, and project timeline so readers can move from concept to implementation.

Article context

AI Learning Project

A practical route from AI concepts to machine learning workflow, evaluation, neural networks, Python practice, handwritten digits, a CIFAR-10 CNN, adversarial traffic-defense notes, and AI security.

Level: Intermediate Reading time: 13 min
  • Gradient Descent
  • Momentum
  • Adam
  • Loss Surface

Your next step

Continue: Convolution and Receptive Field Math
Review the foundation Open resource
Other language version 梯度下降与优化器几何：Momentum、Adam 和 loss surface 轨迹
Share summary Gradient Descent and Optimizer Geometry

Compare gradient descent, momentum, and Adam on a visible quadratic loss surface.

Download share card Open share center

Companion resources

Leave a Reply

Project timeline

Published posts

  1. AI Basics Learning Roadmap Separate AI, machine learning, and deep learning before going into implementation details.
  2. Machine Learning Workflow Follow the practical path from data and features to training, prediction, and evaluation.
  3. Model Training and Evaluation Understand loss, overfitting, train/test splits, accuracy, recall, and F1.
  4. Neural Network Basics Move from perceptrons to activation, forward propagation, backpropagation, and training loops.
  5. Matrix Calculus for Neural Networks Derive dL/dW for y = Wx + b and verify it with finite differences.
  6. Backpropagation as a Computation Graph Trace local gradients through ReLU and softmax cross-entropy in a two-layer MLP.
  7. Gradient Descent and Optimizer Geometry Compare gradient descent, momentum, and Adam on a visible quadratic loss surface.
  8. Convolution and Receptive Field Math Compute convolution output size, receptive fields, channel mixing, and im2col layout.
  9. Transformer Attention Math Hand-calculate Q/K/V scores, softmax weights, masks, multi-head structure, and KV cache.
  10. Python AI Mini Practice Run a small scikit-learn classification task and read the experiment output.
  11. Handwritten Digit Dataset Basics Read train.csv, test.csv, labels, and the flattened 28 by 28 pixel layout before training the classifier.
  12. Handwritten Digit Softmax in C Follow the C implementation from logits and softmax probabilities to confusion matrices and submission export.
  13. Handwritten Digit Playground Notes See how the offline classifier was adapted into a browser demo with drawing input and probability output.
  14. CIFAR-10 Tiny CNN Tutorial in C Build and train a small convolutional neural network for CIFAR-10 image classification, then read its loss and accuracy output.
  15. High-Entropy Traffic Defense Notes Study encrypted metadata leaks, entropy, traffic classifiers, and a defensive Python chaffing prototype.
  16. AI Security Threat Modeling Build a defense map with NIST adversarial ML, MITRE ATLAS, and OWASP LLM risks.
  17. Adversarial Examples and Robust Evaluation Evaluate clean and perturbed accuracy with an FGSM-style digits experiment.
  18. Data Poisoning and Backdoor Defense Study poison rate, trigger behavior, attack success rate, and training pipeline controls.
  19. Model Privacy and Extraction Defense Measure membership inference signal and surrogate fidelity against a local toy model.
  20. LLM, RAG, and Agent Security Separate instructions from data and enforce tool permissions against indirect prompt injection.

Published resources

  1. Python AI practice code guide The article includes a runnable scikit-learn classification script.
  2. digit_softmax_classifier.c The C source for the handwritten digit softmax classifier.
  3. train.csv.zip Compressed handwritten digit training set with 42000 labeled samples.
  4. test.csv.zip Compressed handwritten digit test set with 28000 unlabeled samples.
  5. sample_submission.csv The official submission format example for checking the final output columns.
  6. submission.csv The prediction file generated by the current C project.
  7. digit-playground-model.json The compact softmax demo model and sample set used by the browser playground.
  8. digit-sample-grid.svg A small handwritten digit preview grid extracted from the training set.
  9. Handwritten digit project bundle Contains the source file, compressed datasets, submission files, browser model, and preview grid.
  10. cifar10_tiny_cnn.c source Single-file C tiny CNN with CIFAR-10 loading, convolution, pooling, softmax, and backpropagation.
  11. model_weights.bin sample weights Model weights generated by one local small-sample run.
  12. test_predictions.csv sample predictions Sample test prediction output from the CIFAR-10 tiny CNN.
  13. CNN project explanation PDF Companion explanation material for the CNN project.
  14. Virtual Mirror redacted code skeleton A redacted mld_chaffing_v2.py control-flow skeleton with secrets, node topology, and target lists removed.
  15. Virtual Mirror stress-test template A redacted CSV template for CPU, memory, peak threads, pulse rate, latency, and error measurements.
  16. Virtual Mirror classifier-evaluation template A CSV template for TP, FN, FP, TN, accuracy, precision, recall, F1, ROC-AUC, entropy, and JS divergence.
  17. Virtual Mirror resource notes Notes explaining why the public resources include only redacted code, test templates, and architecture context.
  18. AI Security Lab README Setup, safety boundaries, and quick-run commands for the AI Security series.
  19. AI Security Lab full bundle Includes safe toy scripts, result CSVs, risk register, attack-defense matrix, and architecture diagram.
  20. AI security risk register CSV risk register template for AI threat modeling and release review.
  21. AI attack-defense matrix Maps attack surface, toy demo, metric, and defensive control into one CSV table.
  22. AI Security Lab architecture diagram Shows threat modeling, robustness, data integrity, model privacy, and RAG guardrails.
  23. FGSM digits robustness script FGSM-style perturbation and accuracy-drop experiment for a local digits classifier.
  24. Data poisoning and backdoor toy script Demonstrates poison rate, trigger behavior, and attack success rate on digits.
  25. Model privacy and extraction toy script Outputs membership AUC, target accuracy, surrogate fidelity, and surrogate accuracy.
  26. RAG prompt injection guard toy script Uses a deterministic toy agent to demonstrate external-data demotion and tool-policy blocking.
  27. Deep Learning Math Lab README Setup commands, script entry points, generated outputs, and figure notes for the math series.
  28. Deep learning math full lab bundle Bundles NumPy scripts, CSV outputs, formula diagrams, loss contours, convolution figures, and attention heatmaps.
  29. Gradient check results CSV Stores MSE analytic gradients, finite-difference gradients, and error norms.
  30. Optimizer path CSV Step-by-step coordinates and loss for gradient descent, momentum, and Adam on a 2D quadratic.
  31. Attention weights CSV Scores, softmax weights, and context vectors for a three-token scaled dot-product attention example.
  32. Deep learning math figure set Includes matrix shapes, computation graphs, loss contours, convolution scans, and attention heatmaps.
  33. Deep learning math interactive visualizer Browser modules for gradient checking, optimizer paths, convolution output size, and attention heatmaps.
  34. Deep Learning topic share card A 1200x630 SVG card for sharing the Deep Learning / CNN topic hub.
  35. Machine Learning From Scratch share card A 1200x630 SVG card for the K-means, Iris, and ML workflow topic hub.
  36. Student AI Projects share card A 1200x630 SVG card for handwritten digits, C classifiers, and browser demos.
  37. CNN convolution scan animation An 8-second Remotion animation showing how a 3x3 convolution kernel scans an input and builds a feature map.

Current route

  1. AI Basics Learning Roadmap Learning path step
  2. Machine Learning Workflow Learning path step
  3. Model Training and Evaluation Learning path step
  4. Neural Network Basics Learning path step
  5. Matrix Calculus for Neural Networks Learning path step
  6. Backpropagation as a Computation Graph Learning path step
  7. Gradient Descent and Optimizer Geometry Learning path step
  8. Convolution and Receptive Field Math Learning path step
  9. Transformer Attention Math Learning path step
  10. LLM Visualizer Learning path step
  11. Python AI Mini Practice Learning path step
  12. Handwritten Digit Dataset Basics Learning path step
  13. Handwritten Digit Softmax in C Learning path step
  14. Handwritten Digit Playground Notes Learning path step
  15. CIFAR-10 Tiny CNN Tutorial in C Learning path step
  16. High-Entropy Traffic Defense Notes Learning path step
  17. AI Security Threat Modeling Learning path step
  18. Adversarial Examples and Robust Evaluation Learning path step
  19. Data Poisoning and Backdoor Defense Learning path step
  20. Model Privacy and Extraction Defense Learning path step
  21. LLM, RAG, and Agent Security Learning path step

Next notes

  1. Add more image-classification and error-analysis cases
  2. Turn common metrics into a quick reference
  3. Add more AI security defense experiment notes
Scroll down