AI Glossary: Activation Function
Activation functions are the secret sauce that gives neural networks their power—they introduce nonlinearity, enabling networks to learn complex patterns that linear models can't. Without them, deep learning would be impossible.
What is an Activation Function?
An activation function is a mathematical function applied to the output of each neuron in a neural network. It takes the weighted sum of inputs (z = Σ wᵢxᵢ + b) and transforms it into the neuron's output: a = f(z). This transformation determines whether and how strongly the neuron "fires."
The activation function is the source of nonlinearity in neural networks. Without it, no matter how many layers you stack, the entire network would be equivalent to a single linear transformation—and linear models can only learn linear relationships, which is insufficient for most real-world problems.
Core Purpose
Introduce nonlinearity so the network can learn complex, nonlinear relationships between inputs and outputs.
Mathematical Role
Transform the linear combination z = Σ wᵢxᵢ + b into the neuron's output a = f(z).
Key Requirement
Must be differentiable (or at least have a defined subgradient) so gradients can flow through during backpropagation.
Think of It Like This
An activation function is like a neuron's decision threshold. If the combined input signal is strong enough, the neuron fires (producing a strong output). If it's weak, the neuron stays quiet. Different activation functions implement different "firing rules"—some are all-or-nothing, some are graded, and some can even produce negative outputs.
Why Nonlinearity Matters
Consider a neural network without activation functions. Each layer computes z = Wx + b—a linear transformation. The composition of linear transformations is still linear: W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂) = W'x + b'. So a multi-layer network without activations is just a single-layer linear model, no matter how deep it is.
With nonlinear activation functions, each layer can transform the data in complex ways, allowing the network to learn hierarchical features. The first layer might learn simple patterns (edges in images), the second layer combines them into more complex patterns (shapes), and deeper layers form high-level concepts (objects).
Classic Activation Functions
Sigmoid (Logistic)
f(x) = 1 / (1 + e^(-x)). Outputs values between 0 and 1, making it natural for binary classification. Once dominant, now rarely used in hidden layers because of the vanishing gradient problem—its gradient approaches zero for very positive or negative inputs, slowing learning.
Tanh (Hyperbolic Tangent)
f(x) = (e^x - e^(-x)) / (e^x + e^(-x)). Outputs values between -1 and 1. Zero-centered (unlike sigmoid), which helps with optimization. Still suffers from vanishing gradients at extreme values.
ReLU (Rectified Linear Unit)
f(x) = max(0, x). The most widely used activation in hidden layers. Simple, fast, and effectively solves the vanishing gradient problem for positive inputs. Its sparsity (zero output for negative inputs) is a useful regularization effect. Main drawback: "dying ReLU" where neurons permanently output 0.
Leaky ReLU
f(x) = max(αx, x) where α is a small constant (e.g., 0.01). Fixes the dying ReLU problem by allowing a small gradient for negative inputs. This keeps neurons "alive" even when they receive negative inputs.
Softmax
Used exclusively in the output layer for multi-class classification. Converts a vector of raw scores into probabilities that sum to 1: softmax(zᵢ) = e^(zᵢ) / Σ e^(zⱼ).
Modern Activation Functions
GELU (Gaussian Error Linear Unit)
Used in BERT, GPT-2, and many transformer models. f(x) = x × Φ(x), where Φ is the cumulative distribution function of the standard normal distribution. It's a smooth approximation of ReLU that weights inputs by their probability of being positive, rather than using a hard threshold.
Swish / SiLU
f(x) = x × sigmoid(x). Discovered by automated search at Google. Smooth, non-monotonic (it can decrease for negative inputs), and consistently outperforms ReLU on deep networks. Self-gated, meaning it uses its own value to control the information flow.
SwiGLU
A gated variant used in Llama, PaLM, and other recent LLMs. Combines the Swish activation with a gating mechanism: SwiGLU(x) = Swish(xW₁) ⊙ (xW₂). The gating multiplies two linear projections, one of which has the activation applied. This has been shown to improve language model performance.
Activation Function Comparison
| Function | Range | Pros | Cons | Best For |
|---|---|---|---|---|
| Sigmoid | (0, 1) | Smooth, probabilistic | Vanishing gradients, not zero-centered | Binary classification output |
| Tanh | (-1, 1) | Zero-centered, smooth | Vanishing gradients | RNNs (legacy) |
| ReLU | [0, ∞) | Fast, no vanishing gradient (positive) | Dying ReLU, not zero-centered | Hidden layers (general) |
| Leaky ReLU | (-∞, ∞) | Fixes dying ReLU | Extra hyperparameter α | When dying ReLU is a problem |
| GELU | (-0.17, ∞) | Smooth, outperforms ReLU | More compute than ReLU | Transformers, modern NLP |
| SwiGLU | (-∞, ∞) | State-of-the-art for LLMs | Doubles parameters | Large language models |
Frequently Asked Questions
For most applications, start with ReLU. It's simple, fast, and works well. For modern NLP/transformers, use GELU. For state-of-the-art LLMs, consider SwiGLU. If you encounter dying ReLU (many neurons outputting zero), switch to Leaky ReLU or ELU. Modern frameworks make it easy to swap activation functions, so you can experiment.
You can use a linear activation (f(x) = x), but if you use it in every layer, the entire network becomes linear. A composition of linear functions is linear, so a multi-layer network with linear activations is equivalent to a single-layer linear model. You can use linear activation in the output layer for regression tasks, but you need nonlinear activations in hidden layers.
ReLU outputs 0 for all negative inputs and has a gradient of 0 for negative inputs. Leaky ReLU outputs a small fraction (α, typically 0.01) of the input for negative values, so it has a small non-zero gradient for negative inputs. This prevents the dying ReLU problem where neurons get stuck outputting 0 forever. The trade-off is a small additional hyperparameter (α) to tune.
GELU is smoother than ReLU (differentiable everywhere), which provides better gradient flow. It applies a probabilistic gating mechanism—instead of hard-thresholding at zero, it weights inputs by their probability of being positive. This smoothness and probabilistic interpretation have been shown to improve performance on language tasks, which is why models like BERT and GPT adopted it.
Absolutely. This is common practice. For example, a CNN might use ReLU in convolutional layers, sigmoid in the final binary classification layer, and no activation in intermediate layers that need to preserve negative values. Modern architectures often mix activation functions based on the role of each layer.
Continue your AI learning journey
Next, explore Neural Network Layers—the fundamental building blocks of deep learning architectures.
Read Next: Neural Network Layer →