Deep Learning Frameworks: Why You Stop Writing Backpropagation by Hand

After learning the basics of machine learning and simple neural networks, you might wonder: do I have to write matrix multiplication and backpropagation from scratch every time I build a model? Of course not. In real-world engineering, everyone uses deep learning frameworks.

This article will introduce what deep learning frameworks do and compare the two most mainstream frameworks today: PyTorch and TensorFlow.

1. Why Do We Need Deep Learning Frameworks?

Simply put, deep learning frameworks are like the “Spring Boot” or “React” of the AI world. They package the most complex and repetitive low-level work for you.

• Autograd (Automatic Differentiation): The core step in training a neural network is computing gradients (backpropagation). Frameworks automatically handle this complex calculus for you.
• GPU Acceleration: Matrix operations are very slow on a CPU. Frameworks allow you to move computations to a graphics card (GPU) by changing just one line of code, boosting speed by dozens of times.
• Pre-built Layers and Optimizers: Whether it’s fully connected layers, convolutional layers, or the Adam optimizer, frameworks have built-in APIs ready to be called directly.

2. The Two Major Camps

Currently, the industry and academia primarily use two major frameworks: PyTorch, backed by Meta (formerly Facebook), and TensorFlow, backed by Google.

TensorFlow: The Industry Veteran

TensorFlow was released earlier and once dominated the entire deep learning landscape. Its greatest strength lies in engineering capabilities, making it exceptionally good for deploying models to mobile devices (TensorFlow Lite), web browsers (TensorFlow.js), or large-scale servers (TensorFlow Serving).

However, early versions (TF 1.x) had syntax that was hard to understand and painful to debug. Although TF 2.x adopted Keras as its high-level API and became much simpler, historical baggage still remains.

PyTorch: The Favorite of Academia and Researchers

PyTorch is the rising star, but thanks to its highly “Pythonic” design, it quickly took over academia. Writing PyTorch code feels just like writing standard Python code. It is very easy to debug, and you can simply use `print()` to see the values of tensors.

In recent years, with the explosion of the PyTorch ecosystem (like Hugging Face), the vast majority of new large models (including various open-source LLMs) are developed almost exclusively on PyTorch.

3. Which One Should Beginners Choose?

If you are a beginner, **it is highly recommended to start with PyTorch**. Here is why:

1. More Intuitive Syntax: The code logic is easier to follow, debugging is straightforward, and you won’t encounter as many bizarre error messages.
2. Rich Community Resources: Most of the latest open-source models and reproduction codes on GitHub today are written in PyTorch.
3. Hugging Face Ecosystem: Hugging Face (often considered the GitHub of AI) provides the best and most native support for PyTorch.

4. A Quick Comparison: Defining a Simple Layer

Let’s look at the difference in defining a linear layer (fully connected layer) with 10 inputs and 5 outputs using these two frameworks:

PyTorch:

import torch
import torch.nn as nn

layer = nn.Linear(in_features=10, out_features=5)

TensorFlow (Keras):

import tensorflow as tf
from tensorflow.keras import layers

layer = layers.Dense(units=5, input_shape=(10,))

As you can see, both are very concise for simple structures. However, for complex custom models, PyTorch’s object-oriented approach makes the logic much clearer.

Do not start with high-level wrappers like model.fit()

Keras’s model.fit() runs an entire training session in one line, which looks maximally beginner-friendly. But precisely because it hides the training loop, it is not what I would recommend starting with.

The PyTorch training loop above is worth writing by hand because it lays out five actions explicitly: fetch data, forward, compute loss, backpropagate, update. Those five steps are the skeleton of all deep learning training, and reading any paper’s code or debugging any training problem later means locating yourself among those five steps. Gradients not zeroed, loss computed against the wrong tensor, an update issued before backpropagation — all are visible at a glance in a hand-written loop and entirely invisible inside fit().

The sensible order is to hand-write the loop a few times, understand what each step does, and then adopt high-level wrappers for convenience. Learn to take it apart before using the assembled version — the other way round, you stall the first time you need a custom loss or gradient clipping.

The framework you train in is not the format you deploy

Everything above concerns which framework to write models in. There is something tutorials rarely make clear that you hit immediately the first time you try to actually use a model: the training framework and the deployment format are different things.

Finish training in PyTorch and you have a .pt or .pth file. That file usually contains only parameter tensors and not the network structure — loading it requires the Python code that defined the model, with matching class names, layer names and parameter shapes. A .pt file is not independently distributable; it is one half of a “code plus weights” pair.

So going to production almost always means converting to a self-contained format:

• ONNX: a cross-framework intermediate representation. Browsers, C++ services and mobile runtimes can all load it without a Python environment.
• GGUF: the format used by the llama.cpp ecosystem, designed for quantised language models and CPU-friendly inference.
• TensorRT / CoreML / TFLite: high-performance formats bound to specific hardware or platforms.

That conversion is not a save operation; it has a whole set of traps of its own. Two concrete ones: dynamic control flow in a model — an if that branches on input shape — gets frozen into whichever branch ran at export time when exported to a static graph, producing wrong results for other inputs; and quantisation can shrink a model to a quarter of its size, but post-quantisation accuracy has to be re-validated on data from the real distribution, because testing only on training-set-style samples badly overstates its reliability.

I have hit both, written up in Four ONNX-in-the-Browser Deployment Traps and How I Fooled Myself Validating Quantisation. Neither is required reading at the beginner stage, but knowing that “there is a whole road after training” saves detours when choosing tools.

The dynamic-versus-static graph distinction only matters at deployment

The technical reason behind “PyTorch is easier to debug” is the dynamic graph: the computation graph is built live on every forward pass, so you can insert print() mid-model, use Python if and for, and get errors pointing at the line you wrote.

A static graph instead defines the whole computation up front, compiles it, and then feeds data through. It is awkward to write and hard to debug — precisely what TF 1.x was criticised for.

But the trade-off has another side: a static graph can be optimised and distributed; a dynamic one cannot. Given the complete graph, a compiler can fuse operators, fold constants and reuse memory, and it can serialise the result into a file with no Python dependency. This is why deployment nearly always means freezing the dynamic graph into a static one — torch.jit.trace, torch.export, or an ONNX export.

So the accurate statement is not “dynamic graphs are better” but that dynamic graphs are better during development and static graphs are better at deployment. Modern frameworks let you develop dynamically and convert at the last step. Understanding that is worth considerably more than memorising “PyTorch is dynamic.”

How to tell the tutorial in front of you is out of date

This field moves quickly, and a beginner’s largest hidden cost is not slow learning — it is following an outdated tutorial and then spending hours debugging something that should no longer be written that way. Learning to recognise stale code is worth more than learning another API.

Several signals are visible at a glance:

• tf.Session() or sess.run(): TensorFlow 1.x style. TF 2.x executes eagerly by default, and this code simply will not run on current versions.
• Variable(x) in PyTorch code: Variable merged into Tensor back in version 0.4. Writing it today does nothing and only tells you the code is several years old.
• Reading tensor values via .data: the current form is .detach(). .data bypasses autograd tracking and can silently produce wrong gradients in some situations.
• Calls like nn.functional.sigmoid: these moved to torch.sigmoid long ago; the old spellings either warn or have been removed.
• A training loop with no optimizer.zero_grad(): not outdated, just wrong. PyTorch accumulates gradients by default, so without zeroing they pile up and training behaves incorrectly — with no error raised.

A more reliable habit is to prefer the official documentation’s tutorials over whatever a search engine ranks first. Official docs are updated with each release; blog posts are not. And search ranking rewards age and accumulated links, which biases it toward older material.

One more practical habit: before running someone else’s code, check whether it states its dependency versions. A tutorial with no requirements.txt and no stated framework version leaves you unable to tell whether a failure is your mistake or a version mismatch — and that particular uncertainty is the most corrosive thing for a beginner’s confidence.

Version hell discourages more beginners than frameworks do

One last thing tutorials skip and every beginner meets: it will not install.

Several layers sit between a framework and the GPU, each with version requirements: display driver → CUDA → cuDNN → the CUDA version the framework was compiled against. A mismatch at any layer produces the same symptom, torch.cuda.is_available() returning False, and it will not tell you which layer is at fault.

Two habits remove most of the pain:

1. Do not use the default pip install torch. Go to the official site, select your OS and CUDA version, and use the full command it generates — the default is frequently the CPU build, which imports fine and never touches the GPU.
2. One isolated environment per project. Different projects routinely have mutually incompatible framework requirements, and sharing an environment means installing the new one breaks the old.

After installing, confirm with these three lines before writing any model code:

import torch
print(torch.__version__, torch.version.cuda)
print(torch.cuda.is_available(), torch.cuda.get_device_name(0))

If the second line prints False, fix the environment before going further. Writing an entire training script in an environment that cannot reach the GPU, then discovering you have to reinstall everything, is the most common and most demoralising waste of time at this stage.

5. Next Steps

Next, you can try:

• Install PyTorch (preferably using Anaconda and a virtual environment).
• Understand the most basic data structure in the framework: the Tensor.
• Try writing a simple handwritten digit recognition (MNIST) model using PyTorch.

Remember, frameworks are just tools. Regardless of which one you use, understanding how data flows into the model and how the loss function guides parameter updates is the most essential skill.