What is a loss function in machine learning?

A loss function (also called a cost function or objective function) is a mathematical function that quantifies how wrong a model's predictions are compared to the true values. It takes the model's predictions and the actual targets as input and returns a single number representing the 'cost' or 'penalty.' The goal of training is to minimize this number—the lower the loss, the better the model's predictions.

What is the difference between loss function and cost function?

In practice, the terms are often used interchangeably. Technically, the loss function computes the error for a single training example, while the cost function is the average loss over the entire training dataset. Most modern frameworks and practitioners use 'loss function' for both. The objective function is the broader term that includes both loss/cost and any regularization terms.

Which loss function should I use for classification?

For binary classification: binary cross-entropy (log loss). For multi-class classification: categorical cross-entropy (if labels are one-hot encoded) or sparse categorical cross-entropy (if labels are integer indices). Cross-entropy loss is the standard for classification because it heavily penalizes confident wrong predictions and works well with softmax outputs.

Which loss function should I use for regression?

For standard regression: Mean Squared Error (MSE) is the most common. For regression robust to outliers: Mean Absolute Error (MAE) or Huber loss. MSE penalizes large errors more heavily (squared), making it sensitive to outliers. MAE treats all errors linearly, making it more robust. Huber loss combines both—MSE for small errors and MAE for large errors.

What is cross-entropy loss and why is it used?

Cross-entropy loss measures the difference between two probability distributions—the predicted distribution and the true distribution. It's the standard loss for classification because it: (1) Works naturally with probabilities (softmax output); (2) Heavily penalizes confident wrong predictions; (3) Has nice gradient properties that prevent the 'learning slowdown' problem of MSE with sigmoid activations; (4) Is derived from maximum likelihood estimation.

Can I create my own custom loss function?

Yes, modern deep learning frameworks make it easy to define custom loss functions. You can create one as long as it's differentiable (so gradients can be computed via backpropagation). Custom losses are useful when standard losses don't capture your problem's specific requirements—for example, you might want to penalize certain types of errors more heavily than others based on business constraints or domain knowledge.

📏Model Training

AI Glossary: Loss Function

A loss function is the mathematical compass that guides machine learning—it quantifies how wrong predictions are, and models learn by minimizing this error. Choosing the right loss function is one of the most critical decisions in ML.

📅Published: Jan 2024

⏱️13 min read

📊Level: Beginner-Intermediate

Table of Contents

What is a Loss Function?
Loss Functions for Regression
Loss Functions for Classification
Specialized Loss Functions
How to Choose a Loss Function
Frequently Asked Questions

What is a Loss Function?

A loss function (also called a cost function or objective function) is a mathematical function that measures the difference between a model's predictions and the actual target values. It takes the prediction ŷ and the true value y as input, and returns a single number representing the "penalty" or "error." The larger the number, the worse the model is performing.

The entire purpose of training a machine learning model is to find the parameters that minimize the loss function. This is the fundamental optimization problem: find w such that L(f(x; w), y) is minimized across all training examples.

Core Purpose

Quantify how wrong the model's predictions are, providing a single number that training can optimize to minimize.

Guiding Principle

The loss function defines what "good" means. Different loss functions optimize for different properties of the predictions.

Key Property

Must be differentiable (for gradient-based optimization) and should be convex or at least smooth for reliable convergence.

Loss vs Cost vs Objective

Loss function: Error for a single example. Cost function: Average loss over the entire dataset. Objective function: The overall function being optimized, which includes the cost function plus any regularization terms (e.g., L(w) = Cost(w) + λ||w||²). In practice, these terms are often used interchangeably.

Loss Functions for Regression

Mean Squared Error (MSE)

The most common regression loss. Computes the average of squared differences between predictions and targets:

MSE = (1/n) × Σ(yᵢ - ŷᵢ)²

MSE heavily penalizes large errors (because they're squared), making it sensitive to outliers. It's differentiable everywhere and has nice mathematical properties for optimization.

Mean Absolute Error (MAE)

Computes the average of absolute differences:

MAE = (1/n) × Σ|yᵢ - ŷᵢ|

MAE treats all errors equally, making it more robust to outliers than MSE. However, it's not differentiable at zero (the absolute value function has a kink), which can cause optimization issues.

Huber Loss

Combines the best of both worlds: MSE for small errors (smooth, differentiable) and MAE for large errors (robust to outliers). Controlled by a threshold δ:

If |y - ŷ| ≤ δ: use MSE (0.5 × (y - ŷ)²)
If |y - ŷ| > δ: use MAE (δ × |y - ŷ| - 0.5δ²)

Loss Functions for Classification

Binary Cross-Entropy (Log Loss)

The standard loss for binary classification. Measures the dissimilarity between predicted probabilities and true labels:

BCE = -[y × log(ŷ) + (1 - y) × log(1 - ŷ)]

Cross-entropy heavily penalizes confident wrong predictions. If the true label is 1 and the model predicts 0.01, the loss is -log(0.01) = 4.6. If it predicts 0.99, the loss is -log(0.99) ≈ 0.01.

Categorical Cross-Entropy

For multi-class classification with K classes. Generalizes binary cross-entropy to multiple classes using softmax probabilities:

CCE = -Σ yᵢ × log(ŷᵢ) for i = 1 to K

Hinge Loss

Used primarily for Support Vector Machines (SVMs). Encourages the correct class to have a score higher than incorrect classes by at least a margin:

Hinge = max(0, 1 - y × ŷ)

Unlike cross-entropy, hinge loss doesn't care about the exact probability—just that the correct class wins by a sufficient margin.

Specialized Loss Functions

Loss Function	Task	Key Property
CTC Loss	Speech recognition, OCR	Handles unaligned sequences (input and output have different lengths)
Triplet Loss	Face recognition, similarity learning	Ensures similar items are closer than dissimilar items in embedding space
Contrastive Loss	Siamese networks, representation learning	Pulls similar pairs together and pushes dissimilar pairs apart
Focal Loss	Object detection (imbalanced classes)	Down-weights easy examples, focusing training on hard examples
KL Divergence	VAEs, knowledge distillation	Measures how one probability distribution diverges from another
Dice Loss	Medical image segmentation	Optimizes overlap between predicted and ground truth regions

How to Choose a Loss Function

Choosing the right loss function depends on several factors:

Task type — Regression (MSE, MAE, Huber), binary classification (BCE), multi-class (CCE), or specialized (CTC, triplet)
Data characteristics — Are there outliers? (Use MAE or Huber). Are classes imbalanced? (Use focal loss or weighted cross-entropy).
What you care about — Do you want to penalize large errors more heavily? (MSE). Do you want robust predictions? (MAE). Do you want calibrated probabilities? (Cross-entropy).
Optimization properties — Is the loss differentiable? Smooth? Convex? These affect how easily gradient descent can optimize it.

Practical Rule of Thumb

Start with the standard loss for your task: MSE for regression, cross-entropy for classification. These are standard for good reasons—they're well-understood, well-behaved, and work well in most cases. Only switch to specialized losses when you have a specific reason to do so.

Frequently Asked Questions

Why not use MSE for classification?

MSE with sigmoid activations can cause a "learning slowdown" problem. When the model is very wrong (predicting near 0 when the answer is 1), the sigmoid gradient is nearly zero, so the model learns very slowly. Cross-entropy loss doesn't have this problem—it produces large gradients when the model is confident and wrong, leading to faster learning.

Can I use multiple loss functions together?

Yes, this is common in multi-task learning. You can combine losses with weighted sums: L_total = w₁ × L₁ + w₂ × L₂ + ... + wₙ × Lₙ. The weights control the relative importance of each objective. This is used in models that need to optimize for multiple criteria simultaneously, like a model that both classifies images and detects objects.

What happens if I choose the wrong loss function?

The model will optimize for the wrong thing. For example, using MSE for classification may produce a model that predicts probabilities well on average but makes poor classification decisions. Using MAE for a problem where large errors are catastrophic might produce a model that's too tolerant of outliers. The loss function defines what "good" means—if you define it wrong, the model will be good at the wrong thing.

How does regularization relate to the loss function?

Regularization adds a penalty term to the loss function: L_total = L_data + λR(w). The data loss L_data measures prediction error, and the regularization term R(w) penalizes model complexity. Common regularization terms include L1 (|w|) for sparsity and L2 (w²) for small weights. The λ hyperparameter controls the trade-off between fitting the data and keeping the model simple.

What is the difference between loss and accuracy?

Loss is a continuous, differentiable measure that tells the model how to improve (through gradients). Accuracy is a discrete, non-differentiable measure that tells you how well the model is performing. You train by minimizing loss, but you evaluate by measuring accuracy. A model can have decreasing loss while accuracy stays flat—this often means the model is becoming more confident in its correct predictions without changing its classification decisions.

Continue your AI learning journey

Next, explore Activation Functions—the nonlinear transformations that give neural networks their power.