What is the difference between gradient descent and stochastic gradient descent?

Batch gradient descent computes the gradient using the entire training dataset before each update—accurate but slow for large datasets. Stochastic gradient descent (SGD) computes the gradient using a single random training example, making it much faster per update but noisier. Mini-batch SGD (the most common variant) uses a small batch of examples, balancing speed and stability.

What is the learning rate and why is it important?

The learning rate controls the size of each step in gradient descent. Too high: the algorithm may overshoot the minimum, oscillate, or diverge. Too low: convergence is very slow and may get stuck in local minima. Choosing the right learning rate is one of the most important hyperparameter decisions in training neural networks. Modern optimizers like Adam use adaptive learning rates that adjust automatically.

What is momentum in gradient descent?

Momentum is a technique that accelerates gradient descent by accumulating a velocity vector in directions of persistent gradient. It helps the optimizer: (1) Navigate ravines where the surface curves more steeply in one direction than another; (2) Dampen oscillations in noisy gradient estimates; (3) Escape shallow local minima. It's like a ball rolling downhill—it builds up speed and carries through small bumps.

What is the Adam optimizer?

Adam (Adaptive Moment Estimation) combines momentum and adaptive learning rates. It maintains two moving averages: one for the gradient (first moment, like momentum) and one for the squared gradient (second moment, for adaptive learning rates). Adam is the most popular optimizer for deep learning because it works well out of the box with minimal hyperparameter tuning, typically using default values of learning rate=0.001, beta1=0.9, beta2=0.999.

How does gradient descent relate to backpropagation?

Backpropagation is the algorithm that efficiently computes the gradients needed for gradient descent in neural networks. Gradient descent is the optimization algorithm that uses those gradients to update the weights. Together, they form the core learning algorithm: backpropagation computes the direction of steepest descent (the gradient), and gradient descent takes a step in that direction to reduce the loss.

📐Optimization

AI Glossary: Gradient Descent

Q: What is gradient descent?

Gradient descent is an iterative optimization algorithm used to find the minimum of a function—typically a loss function in machine learning. It works by repeatedly taking steps in the direction of the steepest descent (the negative gradient), gradually moving toward the minimum. It's the fundamental algorithm that enables neural networks and most other machine learning models to learn from data.

Gradient descent is the fundamental optimization algorithm that powers virtually all modern machine learning. It's the engine that adjusts model parameters to minimize loss, turning raw data into intelligent predictions.

📅Published: Jan 2024

⏱️14 min read

📊Level: Intermediate

Table of Contents

What is Gradient Descent?
How Gradient Descent Works
Variants: Batch, Stochastic, Mini-batch
Advanced Optimizers
The Learning Rate Challenge
Optimizer Comparison
Frequently Asked Questions

What is Gradient Descent?

Gradient descent is an iterative optimization algorithm that finds the minimum of a function by repeatedly taking steps in the direction of the steepest descent. In machine learning, that function is the loss function, and we're trying to find the model parameters (weights) that minimize it.

Think of it like finding the lowest point in a mountainous landscape while blindfolded. At each step, you feel the ground around you to determine which direction is downhill, then take a small step in that direction. Repeat until you can't go any lower. That's gradient descent.

Core Equation

w := w - α × ∇L(w) — where w is the parameter, α is the learning rate, and ∇L(w) is the gradient of the loss.

Role in ML

The primary optimization method for training virtually all neural networks, from simple linear regression to trillion-parameter LLMs.

Key Insight

The negative gradient points in the direction of steepest descent. Following it repeatedly converges to a minimum.

Why It's Foundational

Without gradient descent, modern AI wouldn't exist. Every neural network you've ever heard of—GPT, Claude, DALL-E, AlphaFold—was trained using some variant of gradient descent. It's the algorithmic engine that converts data into learned patterns, turning raw computation into intelligence.

How Gradient Descent Works

The algorithm follows a simple loop:

Initialize parameters — Start with random weights (or pre-trained weights in transfer learning)
Forward pass — Compute the model's predictions on the training data
Compute loss — Measure how wrong the predictions are using the loss function
Backward pass (backpropagation) — Compute the gradient of the loss with respect to each parameter
Update parameters — Take a small step in the opposite direction of the gradient: w = w - α × ∇Loss
Repeat — Go back to step 2 until the loss stops decreasing

Gradient descent iteratively follows the downward slope (negative gradient) until reaching a minimum of the loss function.

Mathematical Intuition

The gradient ∇L(w) is a vector of partial derivatives—one for each parameter. Each partial derivative ∂L/∂wᵢ tells us how much the loss changes when we slightly adjust parameter wᵢ. If the derivative is positive, increasing wᵢ increases the loss, so we should decrease wᵢ. If negative, we should increase wᵢ. The magnitude tells us how steep the slope is—how much the loss changes per unit change in the parameter.

Variants: Batch, Stochastic, Mini-batch

Batch Gradient Descent

Computes the gradient using the entire training dataset. Each update is accurate and stable, but for large datasets, it's impossibly slow—you need to process millions of examples before taking a single step.

Stochastic Gradient Descent (SGD)

Computes the gradient using one random training example at a time. Each update is noisy (the gradient from one example may not point in the true direction of steepest descent), but it's very fast. The noise can actually help escape shallow local minima.

Mini-batch Gradient Descent

The sweet spot used in practice. Computes the gradient using a small batch of examples (typically 32, 64, 128, or 256). This balances the stability of batch gradient descent with the speed of SGD. GPU architectures are optimized for this approach.

Variant	Data per Update	Speed	Noise	Best For
Batch GD	Entire dataset	Very Slow	None	Small datasets (<10K examples)
Stochastic GD	1 example	Fast	Very High	Online learning
Mini-batch GD	32-256 examples	Fast	Moderate	Most practical applications

Advanced Optimizers

Momentum

Momentum adds a velocity term that accumulates past gradients. Instead of taking each step based only on the current gradient, momentum-based optimizers maintain a running average. This helps navigate ravines (where the loss surface is steep in one direction but flat in another) and dampens oscillations.

RMSprop

RMSprop (Root Mean Square Propagation) adapts the learning rate for each parameter individually. Parameters with large gradients get smaller learning rates; parameters with small gradients get larger ones. This is particularly useful for problems with sparse gradients.

Adam (Adaptive Moment Estimation)

Adam combines momentum and adaptive learning rates. It's the most popular optimizer for deep learning today:

Maintains an exponentially decaying average of past gradients (first moment estimate, like momentum)
Maintains an exponentially decaying average of past squared gradients (second moment estimate, like RMSprop)
Bias-corrects both estimates to account for their initialization at zero
Default hyperparameters (α=0.001, β₁=0.9, β₂=0.999) work well for most problems

AdamW

A variant of Adam that decouples weight decay from the gradient update. This small change makes AdamW more effective for large models and is the default optimizer for training modern LLMs.

Practical Recommendation

For most deep learning tasks, start with Adam (or AdamW for large models) with the default learning rate of 0.001. If you need more control or are training in a production setting, SGD with momentum (learning rate 0.01, momentum 0.9) with a learning rate schedule often achieves better final performance, though it requires more tuning.

The Learning Rate Challenge

The learning rate α is the most important hyperparameter in gradient descent. Choosing it well is both an art and a science:

Too large — The optimizer may overshoot the minimum, oscillate wildly, or diverge entirely (loss goes to infinity)
Too small — Convergence is painfully slow; the model may get stuck in local minima or plateaus
Just right — Steady, efficient convergence to a good minimum

Learning Rate Schedules

Rather than using a fixed learning rate, modern training uses schedules that adjust the learning rate over time:

Step decay — Reduce the learning rate by a factor every N epochs
Exponential decay — α(t) = α₀ × e^(-kt)
Cosine annealing — Smoothly decrease the learning rate following a cosine curve
Warmup — Start with a very small learning rate and gradually increase it (critical for training large transformers)
Reduce on plateau — Reduce the learning rate when validation loss stops improving

Optimizer Comparison

Optimizer	Adaptive LR	Momentum	Default LR	Best For
SGD	No	No	0.01	Simple problems, baselines
SGD + Momentum	No	Yes	0.01	Image classification (CNNs)
RMSprop	Yes	No	0.001	RNNs, sequence models
Adam	Yes	Yes	0.001	General purpose (most tasks)
AdamW	Yes	Yes	0.001	Transformers, large models

Frequently Asked Questions

Does gradient descent always find the global minimum?

No. Gradient descent is guaranteed to find the global minimum only for convex functions. Neural network loss surfaces are highly non-convex with many local minima, saddle points, and plateaus. In practice, gradient descent typically finds a "good enough" local minimum, and modern techniques (momentum, adaptive learning rates, good initialization) help it find better minima. Interestingly, for very large neural networks, most local minima tend to have similar quality.

What is the difference between gradient descent and backpropagation?

They are two parts of the same learning process. Backpropagation is the algorithm for computing gradients—it efficiently calculates how much each parameter contributed to the loss using the chain rule. Gradient descent is the algorithm that uses those gradients to update the parameters. You can think of backpropagation as the "measurement" step and gradient descent as the "adjustment" step.

Why do we use mini-batches instead of the full dataset?

Mini-batches offer several advantages: (1) Speed—processing 32 examples is much faster than processing millions; (2) GPU efficiency—mini-batches fit in GPU memory and leverage parallel computation; (3) Regularization effect—the noise in mini-batch gradients helps prevent overfitting; (4) Faster convergence—you get many more updates per epoch compared to batch gradient descent. The batch size is typically chosen to maximize GPU utilization.

What is gradient clipping and when is it used?

Gradient clipping limits the maximum norm of the gradient vector to prevent exploding gradients. If the gradient norm exceeds a threshold, it's scaled down to that threshold. This is essential for training RNNs, LSTMs, and large transformers, where gradients can become extremely large (explode) due to the chain rule multiplying many terms together. Typical clipping values are between 1.0 and 5.0.

How do I choose between Adam and SGD?

Use Adam for: rapid prototyping, new problem domains, when you want good results with minimal tuning, or when training transformers. Use SGD with momentum for: image classification with CNNs, when you need the absolute best performance and have time to tune hyperparameters, or when you need reproducible results (Adam's adaptivity can make exact reproduction harder). Many practitioners start with Adam and switch to SGD if they need a final performance boost.

Continue your AI learning journey

Next, explore Backpropagation—the algorithm that efficiently computes the gradients gradient descent needs.