Convolution and Receptive Field Math: Padding, Stride, Channels, and im2col

Replace a fully connected layer with a convolutional one and the parameter count drops by orders of magnitude while results improve. The reason is not that convolution is “more advanced” but that it encodes three assumptions about images: features are local, a feature counts wherever it appears, and the spatial relationships between pixels carry meaning. A fully connected layer uses none of that information.

This article works out the output-size formula first (what padding and stride each contribute to it), then the receptive field — how much of the original image one output pixel actually sees, and why it grows more slowly with depth than intuition suggests. It closes by implementing 2D convolution from scratch in NumPy, grounding the formulas in code.

1. The Mathematics of Output Dimensions

When you slide a kernel across an input tensor, the spatial dimensions of the output feature map are determined by four factors: Input size ($W, H$), Kernel size ($K$), Padding ($P$), and Stride ($S$).

The formula to compute the output dimension is:

Output_Size = floor((Input_Size + 2 * Padding - Kernel_Size) / Stride) + 1

Let’s break down a classic scenario: an input image of 5x5, a kernel of 3x3, a padding of 0 (Valid convolution), and a stride of 1. Plugging these into our formula:

floor((5 + 2*0 - 3) / 1) + 1 = 3

Thus, our output feature map is exactly 3x3. If we wanted the output to remain 5x5, we would need to add a padding of 1 (Same convolution), assuming a stride of 1.

2. Hand-Calculating One Output Cell

One convolution output value is the sum of element-wise products between a local patch of the input and the kernel matrix. This is essentially a dot product.

If the companion lab writes -1.000000 for row=1,col=1 in conv2d-results.csv, it means multiplying the highlighted 3×3 input patch by the 3×3 kernel matrix and summing all 9 entries yields exactly -1.0.

3. The Expanding Receptive Field

The Receptive Field (RF) is the size of the region in the original input space that affects a specific neural network feature. A single 3x3 convolution sees a 3x3 local region. However, deep neural networks stack multiple convolutional layers. How does the network ever “see” the whole picture?

When you stack a second 3x3 convolution on top of the first one, a single output neuron in the second layer connects to a 3x3 region in the first hidden layer. But each of those 9 neurons in the first hidden layer itself connects to a 3x3 region in the original input. Consequently, a single neuron in layer 2 has an effective receptive field of 5x5 on the original input.

graph TD
    sublayer_2["Layer 2 (1x1 Output)"] --> sublayer_1["Layer 1 (3x3 Feature Map)"]
    sublayer_1 --> input["Original Input (5x5 Receptive Field)"]
    
    style sublayer_2 fill:#f9f,stroke:#333,stroke-width:2px
    style sublayer_1 fill:#bbf,stroke:#333,stroke-width:2px
    style input fill:#bfb,stroke:#333,stroke-width:2px

Mathematically, the receptive field size $RF_l$ at layer $l$ can be computed using the formula:

RF_l = RF_{l-1} + (Kernel_Size_l - 1) * Stride_Product_{i=1 to l-1}

This explains how convolutional networks can start by detecting tiny edges and gradually build up to recognizing complex textures, shapes, and eventually entire objects like faces or cars.

4. Implementation: im2col and Matrix Multiplication

In practice, iterating through an image using nested loops (sliding window) is extremely slow. Modern deep learning frameworks (like PyTorch and TensorFlow) vectorize this operation by transforming the convolution into a massive matrix multiplication. This technique is known as im2col (Image to Column).

im2col extracts each local patch from the input image, flattens it into a 1D vector, and stacks them into a large matrix. The kernel is also flattened. The convolution then becomes a single, highly optimized matrix multiplication (GEMM).

import numpy as np

def conv2d_im2col(image, kernel, stride=1):
    """
    A practical Numpy implementation of 2D Convolution using im2col.
    """
    h_in, w_in = image.shape
    k_h, k_w = kernel.shape
    
    # Calculate output dimensions
    out_h = (h_in - k_h) // stride + 1
    out_w = (w_in - k_w) // stride + 1
    
    # Extract patches (im2col step)
    # Shape of cols: (out_h * out_w, k_h * k_w)
    cols = []
    for r in range(0, h_in - k_h + 1, stride):
        for c in range(0, w_in - k_w + 1, stride):
            patch = image[r:r+k_h, c:c+k_w]
            cols.append(patch.reshape(-1))
            
    im_matrix = np.vstack(cols)
    
    # Flatten kernel
    weight_matrix = kernel.reshape(-1, 1)
    
    # Perform matrix multiplication
    result = im_matrix @ weight_matrix
    
    # Reshape back to feature map dimensions
    return result.reshape(out_h, out_w)

# Test the implementation
test_img = np.arange(25).reshape(5, 5)
test_kernel = np.ones((3, 3))
output = conv2d_im2col(test_img, test_kernel)
print("Output Shape:", output.shape)
print(output)

For a 5x5 input and 3x3 kernel, there are 9 valid patches, each containing 9 values. The im_matrix will have the shape (9, 9).

5. Visualizing the Process

While watching the animation, notice two properties: the same kernel weights are reused across many spatial positions (weight sharing), and each output initially only “sees” a local region of the input (local connectivity).

6. Personal Experience / Engineer’s Perspective

Working with convolutions in real-world scenarios introduces several practical challenges that aren’t immediately obvious from the math:

The Memory vs. Compute Trade-off: The im2col approach is brilliant for fully utilizing GPU cores via matrix multiplication, but it comes with a massive cost: memory duplication. By extracting overlapping patches, im2col inflates the memory footprint of the input tensor. If you are working with large medical images (e.g., 3D CT scans), calling im2col can easily cause an Out-Of-Memory (OOM) error. In production C++/CUDA, we often use more memory-efficient implicit GEMM or Winograd algorithms.

• Debugging Dimensionality Nightmares: The number one error junior engineers encounter is the dreaded RuntimeError: size mismatch when transitioning from the final Convolutional layer to the first Fully Connected (Dense) layer. Always log or manually compute your final tensor shape using the output size formula before applying a flatten() operation.
• Checkerboard Artifacts: When using transposed convolutions (often wrongly called deconvolutions) for upsampling in Generative Adversarial Networks (GANs), you frequently encounter checkerboard artifacts. These happen when the kernel size is not evenly divisible by the stride. A practical fix I often use is to replace transposed convolutions with a nearest-neighbor upsample followed by a standard stride-1 convolution.
• Padding Effects on Edges: Zero-padding is the default, but it artificially introduces dark borders into your feature maps. If you notice your model performing poorly on objects at the edge of the image, consider switching to “Reflect” or “Replicate” padding.

7. Convolution Verification Table

A convolution implementation should be checked at three levels: shape math, numerical output, and architectural side effects. The table below provides a compact audit trail for the example in this article.

Next up, we transition from rigid, locally-connected convolutions to the flexible, global token-to-token interactions of Attention mechanisms in Transformers.