AI Glossary: Backpropagation
Backpropagation is the algorithm that makes deep learning computationally feasible. It efficiently computes gradients through millions of parameters by applying the chain rule backward through the network.
What is Backpropagation?
Backpropagation (short for "backward propagation of errors") is the core algorithm that enables neural networks to learn. It efficiently computes the gradient of the loss function with respect to every parameter in the network by applying the chain rule of calculus from the output layer backward to the input layer.
Before backpropagation was popularized in the 1980s by Rumelhart, Hinton, and Williams, training neural networks with more than a few layers was computationally intractable. Backpropagation transformed neural networks from a theoretical curiosity into a practical tool, and eventually into the foundation of modern AI.
Core Mechanism
Applies the chain rule of calculus to propagate error signals backward through the network, computing gradients for all parameters in one pass.
Computational Efficiency
Computes gradients for all parameters in O(n) time, where n is the number of parameters—vs O(n²) for naive approaches.
Historical Context
Popularized in 1986, backpropagation was the key algorithmic breakthrough that launched the first neural network revolution.
Why It's a Breakthrough
Imagine a network with 1 million parameters. Computing the gradient for each parameter individually would require evaluating the network 1 million times. Backpropagation computes all 1 million gradients in a single backward pass. This efficiency—from O(n²) to O(n)—is what makes training deep networks with billions of parameters possible.
How Backpropagation Works
The key mathematical tool behind backpropagation is the chain rule from calculus. If a function is composed of multiple functions (like a neural network is composed of layers), the chain rule tells us how to compute the derivative of the composite function:
∂L/∂w = ∂L/∂y × ∂y/∂z × ∂z/∂w
Where L is the loss, y is the output of a layer, z is the input to that layer, and w is the weight. The chain rule lets us break down the complex gradient computation into a series of simple multiplications.
Forward pass (green) computes predictions. Backward pass (red) propagates gradients using the chain rule.
Forward and Backward Passes
Forward Pass
During the forward pass, the network processes the input through each layer sequentially, computing activations and ultimately producing a prediction. The loss function then measures how far this prediction is from the true target.
Backward Pass
During the backward pass, the error signal flows from the output back to the input:
- Start with the gradient of the loss with respect to the network's output: ∂L/∂output
- For each layer (from last to first), apply the chain rule to compute gradients with respect to that layer's parameters and inputs
- Pass the gradient with respect to the layer's inputs to the previous layer
- Continue until all parameters have gradients
Computational Graph
Modern frameworks (PyTorch, TensorFlow, JAX) represent the network as a computational graph—a directed acyclic graph of mathematical operations. Backpropagation is implemented as a traversal of this graph in reverse topological order, applying the chain rule at each node. This is called automatic differentiation.
The Vanishing Gradient Problem
Backpropagation has a fundamental vulnerability: gradients can become extremely small (vanish) as they propagate backward through many layers. This happens because each layer's gradient is multiplied by the previous layer's gradient, and if these multipliers are typically less than 1, the product shrinks exponentially.
When gradients vanish, early layers learn very slowly or not at all, making it impossible to train deep networks. This was the bottleneck that prevented deep learning from working for decades.
Key Solutions
- ReLU activation — Unlike sigmoid/tanh, ReLU has a gradient of 1 for positive inputs, preventing multiplicative decay
- Residual connections (ResNet) — Skip connections provide a direct path for gradients to flow through, bypassing many layers
- Batch normalization — Normalizes layer inputs to prevent extreme values that can cause vanishing or exploding gradients
- Careful initialization — Xavier/Glorot and He initialization set initial weights to maintain gradient magnitude across layers
Exploding Gradients
The opposite of vanishing gradients—when gradients become exponentially large, causing unstable updates. This is typically solved by gradient clipping, which caps the maximum gradient norm. Both vanishing and exploding gradients are consequences of the multiplicative nature of the chain rule in deep networks.
Automatic Differentiation
Modern deep learning frameworks implement automatic differentiation (autodiff), which is a generalization of backpropagation. Autodiff automatically computes gradients for any computation expressed as a graph of differentiable operations, without requiring manual derivation.
There are two modes:
- Reverse-mode autodiff — The same as backpropagation. Efficient when there are many inputs and few outputs (typical in neural networks).
- Forward-mode autodiff — Efficient when there are few inputs and many outputs. Less common in deep learning.
PyTorch's autograd, TensorFlow's GradientTape, and JAX's grad transform all implement reverse-mode autodiff, making it trivial to compute gradients for any differentiable function.
Frequently Asked Questions
Because the error signal propagates backward through the network—from the output layer toward the input layer. This is in contrast to the forward pass, where information flows from input to output. The "propagation" refers to how the chain rule distributes the error signal across all parameters, with each layer's gradient being computed from the next layer's gradient.
Backpropagation itself computes exact gradients—it doesn't "get stuck." The gradient descent optimizer that uses those gradients can get stuck in local minima. The two are separate: backpropagation provides the gradient information, and the optimizer decides how to use it. For very large neural networks, most local minima have similar quality, so this is less of a concern than intuition might suggest.
Backpropagation requires all operations to be differentiable. For non-differentiable operations (like argmax or thresholding), common workarounds include: (1) Using a differentiable approximation (e.g., softmax instead of argmax); (2) Using the straight-through estimator, which passes the gradient through unchanged; (3) Using REINFORCE or other policy gradient methods for discrete decisions; (4) Reparameterization tricks for sampling operations.
This is a long-standing debate in neuroscience. Biological neurons don't appear to implement backpropagation in the same way artificial neural networks do—there's no evidence of symmetric backward connections or precise gradient computation. However, the brain clearly learns, and researchers have proposed biologically plausible approximations of backpropagation, such as feedback alignment and predictive coding. The question remains open.
Modern frameworks use dynamic computational graphs and automatic differentiation. In PyTorch, each tensor operation is recorded in a graph during the forward pass. When .backward() is called, the graph is traversed in reverse, applying the chain rule at each node. Tensors that require gradients store their gradients in a .grad attribute. This approach is flexible, efficient, and handles arbitrary Python control flow (if statements, loops) naturally.
Continue your AI learning journey
Next, explore Loss Functions—the mathematical measures that tell models how wrong they are.
Read Next: Loss Function →