Neural Network Basics: From Perceptrons to Multi-Layer Networks
AI Learning Project
Neural Network Basics: From Perceptrons to Multi-Layer Networks
An accessible introduction to neurons, weights, bias, activation functions, forward propagation, backpropagation, and training loops.
AI Learning Project

Neural Network Basics: From Perceptrons to Multi-Layer Networks

Reading info

Level: Intermediate Reading time: 8 min
  • Neural Networks
  • Backpropagation
  • Python
Open knowledge map

Neural networks are often presented as complicated systems, but the entry-level view can be simple: a neural network is a trainable composition of functions. Each layer transforms its input, and multiple layers together can represent more complex relationships.

This article starts with a single neuron and explains weights, bias, activation functions, forward propagation, and the intuition behind backpropagation. The goal is not to derive every formula, but to make neural network training code easier to read.

While reading, keep one main loop in mind: the network predicts with current parameters, measures loss, then updates parameters in the direction that reduces loss.

1. Start With One Neuron

A simple neuron can be written as:

z = w1 * x1 + w2 * x2 + ... + b
output = activation(z)

The parts are:

  • x: input features
  • w: weights
  • b: bias
  • activation: an activation function

Without activation functions, multiple linear layers can still be collapsed into one linear transformation. Activation functions give the network nonlinear expressive power.

2. What a Perceptron Can Do

A perceptron can be viewed as an early simple neural network. It computes a weighted sum of inputs, then applies a threshold to produce a class label.

if w1 * x1 + w2 * x2 + b > 0:
    predict 1
else:
    predict 0

This can solve linearly separable problems, where classes can be separated by a line, plane, or higher-dimensional hyperplane.

Real data often contains nonlinear relationships, so we need multi-layer networks and nonlinear activation functions.

3. What Is a Layer?

A layer sends a group of inputs through multiple neurons and returns a group of outputs. Common layer roles include:

  • Input layer: receives raw features
  • Hidden layer: performs intermediate transformations
  • Output layer: returns class probabilities or numeric predictions

A small multi-layer network can be represented as:

input features -> hidden layer 1 -> hidden layer 2 -> output layer

Each layer has its own weights and biases. Training adjusts these parameters together.

4. Forward Propagation

Forward propagation means computing from input to output, layer by layer.

x -> layer1 -> activation -> layer2 -> activation -> output

In code, this usually corresponds to a model’s forward function. It answers:

Given the current parameters and a batch of input, what does the model predict?

Both training and inference use forward propagation. During training, the prediction is also used to compute loss and update parameters.

5. The Intuition Behind Backpropagation

Backpropagation calculates how each parameter affects the loss. Intuitively, it asks:

If this weight became slightly larger or smaller, how would the final loss change?

With that information, an optimizer can update parameters in a direction that reduces loss.

prediction -> compute loss -> backpropagate gradients -> update parameters

You do not need to hand-write backpropagation at the beginning. Frameworks such as PyTorch and TensorFlow compute gradients automatically. But you should understand why training code contains steps such as loss.backward() and optimizer.step().

6. A Typical Training Loop

In pseudocode, neural network training often looks like this:

for epoch in range(num_epochs):
    for X_batch, y_batch in train_loader:
        y_pred = model(X_batch)
        loss = loss_fn(y_pred, y_batch)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

The loop can be read as five steps:

  1. Take a batch of training data
  2. Run forward propagation to get predictions
  3. Compute loss against the true labels
  4. Backpropagate gradients
  5. Let the optimizer update parameters

7. Why Deep Learning Needs More Data and Compute

Neural networks can express complex patterns, but the cost is real:

  • They have many parameters and can overfit
  • They usually need more data
  • They have a larger tuning space
  • Training speed depends more heavily on hardware

This is why it is useful to learn the traditional machine learning workflow first. Once data, features, training, and evaluation are clear, neural networks become easier to reason about.

8. Neural Networks and Large Models

Large language models, image generation systems, and speech recognition systems are deep learning systems. They use more complex architectures, larger datasets, and longer training processes.

Even when the model is large, the foundation questions remain similar:

  • How is input represented as numbers?
  • How does the model transform input into output?
  • How does the loss function measure prediction error?
  • How does training update parameters?
  • Does the evaluation method reflect real use?

Learning neural network basics is not only about training a network immediately. It gives you the shared language behind modern AI systems.

9. Common Beginner Misunderstandings

When first learning neural networks, these misunderstandings are common:

  • Assuming more layers are always better while ignoring data size, overfitting, and training cost
  • Treating activation functions as minor details instead of understanding their nonlinear role
  • Focusing only on architecture while ignoring the loss function and evaluation metrics
  • Assuming the model is reliable just because training loss goes down

Neural networks are powerful because of their expressive capacity, but reliability still depends on data splits, evaluation, and error analysis.

10. Neural Network Training Evidence Checklist

A beginner neural network experiment should leave behind enough evidence for someone else to reproduce the result and identify failure modes. The checklist below connects the concepts in this article to practical training records.

Evidence item What to record Why it matters Failure signal
Input shape Batch size, feature count, tensor layout, and normalization range Most silent neural network bugs are shape or scale mistakes Loss changes when only the batch dimension or image channel order changes
Loss curve Training loss, validation loss, and learning rate per epoch The curve shows underfitting, overfitting, or optimizer instability Training loss falls while validation loss rises for many epochs
Gradient health Gradient norm, exploding or vanishing activations, and optimizer step size Backpropagation can fail even when the code has no syntax error Weights become NaN, gradients collapse to zero, or updates oscillate wildly
Error analysis Confusion matrix, hard examples, and examples outside the training distribution Aggregate accuracy hides systematic mistakes The model is strong on common classes but unreliable on rare or shifted inputs

11. What to Read Next

The previous article is Model Training and Evaluation. To connect the whole series in one runnable exercise, continue with Python AI Mini Practice.

Internal links

Related tutorials

Review foundation Model Training and Evaluation

Understand loss, overfitting, train/test splits, accuracy, recall, and F1.

 Continue next Matrix Calculus for Neural Networks

Derive dL/dW for y = Wx + b and verify it with finite differences.

 Learning path AI Learning Path

Read AI foundations, machine learning, CNN work, traffic defense, and AI security in order.

 Topic hub Student AI Projects

Connect handwritten digits, browser demos, and CNN projects into a portfolio path.

 Knowledge map Knowledge Map

See how this article connects to projects, resources, and learning routes.

Search questions

FAQ

Who is this article for?

This article is for readers who want an intermediate-level guide to Neural Network Basics. It takes about 8 min and focuses on Neural Networks, Backpropagation, Python.

What should I read next?

The recommended next step is Matrix Calculus for Neural Networks, so the article connects into a longer learning route instead of ending as an isolated note.

Does this article include runnable code or companion resources?

This article is primarily explanatory, but the related tutorials point to runnable examples, resources, and project pages.

How does this article fit into the larger site?

It is connected to the article context block, learning routes, resources, and project timeline so readers can move from concept to implementation.

Article context

AI Learning Project

A practical route from AI concepts to machine learning workflow, evaluation, neural networks, Python practice, handwritten digits, a CIFAR-10 CNN, adversarial traffic-defense notes, and AI security.

Level: Intermediate Reading time: 8 min
  • Neural Networks
  • Backpropagation
  • Python

Your next step

Continue: Matrix Calculus for Neural Networks
Review the foundation View project
Other language version 神经网络基础：从感知机到多层网络
Share summary Neural Network Basics

Move from perceptrons to activation, forward propagation, backpropagation, and training loops.

Download share card Open share center

Leave a Reply

Project timeline

Published posts

  1. AI Basics Learning Roadmap Separate AI, machine learning, and deep learning before going into implementation details.
  2. Machine Learning Workflow Follow the practical path from data and features to training, prediction, and evaluation.
  3. Model Training and Evaluation Understand loss, overfitting, train/test splits, accuracy, recall, and F1.
  4. Neural Network Basics Move from perceptrons to activation, forward propagation, backpropagation, and training loops.
  5. Matrix Calculus for Neural Networks Derive dL/dW for y = Wx + b and verify it with finite differences.
  6. Backpropagation as a Computation Graph Trace local gradients through ReLU and softmax cross-entropy in a two-layer MLP.
  7. Gradient Descent and Optimizer Geometry Compare gradient descent, momentum, and Adam on a visible quadratic loss surface.
  8. Convolution and Receptive Field Math Compute convolution output size, receptive fields, channel mixing, and im2col layout.
  9. Transformer Attention Math Hand-calculate Q/K/V scores, softmax weights, masks, multi-head structure, and KV cache.
  10. Python AI Mini Practice Run a small scikit-learn classification task and read the experiment output.
  11. Handwritten Digit Dataset Basics Read train.csv, test.csv, labels, and the flattened 28 by 28 pixel layout before training the classifier.
  12. Handwritten Digit Softmax in C Follow the C implementation from logits and softmax probabilities to confusion matrices and submission export.
  13. Handwritten Digit Playground Notes See how the offline classifier was adapted into a browser demo with drawing input and probability output.
  14. CIFAR-10 Tiny CNN Tutorial in C Build and train a small convolutional neural network for CIFAR-10 image classification, then read its loss and accuracy output.
  15. High-Entropy Traffic Defense Notes Study encrypted metadata leaks, entropy, traffic classifiers, and a defensive Python chaffing prototype.
  16. AI Security Threat Modeling Build a defense map with NIST adversarial ML, MITRE ATLAS, and OWASP LLM risks.
  17. Adversarial Examples and Robust Evaluation Evaluate clean and perturbed accuracy with an FGSM-style digits experiment.
  18. Data Poisoning and Backdoor Defense Study poison rate, trigger behavior, attack success rate, and training pipeline controls.
  19. Model Privacy and Extraction Defense Measure membership inference signal and surrogate fidelity against a local toy model.
  20. LLM, RAG, and Agent Security Separate instructions from data and enforce tool permissions against indirect prompt injection.

Published resources

  1. Python AI practice code guide The article includes a runnable scikit-learn classification script.
  2. digit_softmax_classifier.c The C source for the handwritten digit softmax classifier.
  3. train.csv.zip Compressed handwritten digit training set with 42000 labeled samples.
  4. test.csv.zip Compressed handwritten digit test set with 28000 unlabeled samples.
  5. sample_submission.csv The official submission format example for checking the final output columns.
  6. submission.csv The prediction file generated by the current C project.
  7. digit-playground-model.json The compact softmax demo model and sample set used by the browser playground.
  8. digit-sample-grid.svg A small handwritten digit preview grid extracted from the training set.
  9. Handwritten digit project bundle Contains the source file, compressed datasets, submission files, browser model, and preview grid.
  10. cifar10_tiny_cnn.c source Single-file C tiny CNN with CIFAR-10 loading, convolution, pooling, softmax, and backpropagation.
  11. model_weights.bin sample weights Model weights generated by one local small-sample run.
  12. test_predictions.csv sample predictions Sample test prediction output from the CIFAR-10 tiny CNN.
  13. CNN project explanation PDF Companion explanation material for the CNN project.
  14. Virtual Mirror redacted code skeleton A redacted mld_chaffing_v2.py control-flow skeleton with secrets, node topology, and target lists removed.
  15. Virtual Mirror stress-test template A redacted CSV template for CPU, memory, peak threads, pulse rate, latency, and error measurements.
  16. Virtual Mirror classifier-evaluation template A CSV template for TP, FN, FP, TN, accuracy, precision, recall, F1, ROC-AUC, entropy, and JS divergence.
  17. Virtual Mirror resource notes Notes explaining why the public resources include only redacted code, test templates, and architecture context.
  18. AI Security Lab README Setup, safety boundaries, and quick-run commands for the AI Security series.
  19. AI Security Lab full bundle Includes safe toy scripts, result CSVs, risk register, attack-defense matrix, and architecture diagram.
  20. AI security risk register CSV risk register template for AI threat modeling and release review.
  21. AI attack-defense matrix Maps attack surface, toy demo, metric, and defensive control into one CSV table.
  22. AI Security Lab architecture diagram Shows threat modeling, robustness, data integrity, model privacy, and RAG guardrails.
  23. FGSM digits robustness script FGSM-style perturbation and accuracy-drop experiment for a local digits classifier.
  24. Data poisoning and backdoor toy script Demonstrates poison rate, trigger behavior, and attack success rate on digits.
  25. Model privacy and extraction toy script Outputs membership AUC, target accuracy, surrogate fidelity, and surrogate accuracy.
  26. RAG prompt injection guard toy script Uses a deterministic toy agent to demonstrate external-data demotion and tool-policy blocking.
  27. Deep Learning Math Lab README Setup commands, script entry points, generated outputs, and figure notes for the math series.
  28. Deep learning math full lab bundle Bundles NumPy scripts, CSV outputs, formula diagrams, loss contours, convolution figures, and attention heatmaps.
  29. Gradient check results CSV Stores MSE analytic gradients, finite-difference gradients, and error norms.
  30. Optimizer path CSV Step-by-step coordinates and loss for gradient descent, momentum, and Adam on a 2D quadratic.
  31. Attention weights CSV Scores, softmax weights, and context vectors for a three-token scaled dot-product attention example.
  32. Deep learning math figure set Includes matrix shapes, computation graphs, loss contours, convolution scans, and attention heatmaps.
  33. Deep learning math interactive visualizer Browser modules for gradient checking, optimizer paths, convolution output size, and attention heatmaps.
  34. Deep Learning topic share card A 1200x630 SVG card for sharing the Deep Learning / CNN topic hub.
  35. Machine Learning From Scratch share card A 1200x630 SVG card for the K-means, Iris, and ML workflow topic hub.
  36. Student AI Projects share card A 1200x630 SVG card for handwritten digits, C classifiers, and browser demos.
  37. CNN convolution scan animation An 8-second Remotion animation showing how a 3x3 convolution kernel scans an input and builds a feature map.

Current route

  1. AI Basics Learning Roadmap Learning path step
  2. Machine Learning Workflow Learning path step
  3. Model Training and Evaluation Learning path step
  4. Neural Network Basics Learning path step
  5. Matrix Calculus for Neural Networks Learning path step
  6. Backpropagation as a Computation Graph Learning path step
  7. Gradient Descent and Optimizer Geometry Learning path step
  8. Convolution and Receptive Field Math Learning path step
  9. Transformer Attention Math Learning path step
  10. LLM Visualizer Learning path step
  11. Python AI Mini Practice Learning path step
  12. Handwritten Digit Dataset Basics Learning path step
  13. Handwritten Digit Softmax in C Learning path step
  14. Handwritten Digit Playground Notes Learning path step
  15. CIFAR-10 Tiny CNN Tutorial in C Learning path step
  16. High-Entropy Traffic Defense Notes Learning path step
  17. AI Security Threat Modeling Learning path step
  18. Adversarial Examples and Robust Evaluation Learning path step
  19. Data Poisoning and Backdoor Defense Learning path step
  20. Model Privacy and Extraction Defense Learning path step
  21. LLM, RAG, and Agent Security Learning path step

Next notes

  1. Add more image-classification and error-analysis cases
  2. Turn common metrics into a quick reference
  3. Add more AI security defense experiment notes
Scroll down