AI Glossary: Recurrent Neural Network
Recurrent Neural Networks (RNNs) were the original deep learning architecture for sequential data. They maintain a hidden state that carries information across time steps, enabling them to process everything from text to time series.
What is a Recurrent Neural Network (RNN)?
A Recurrent Neural Network (RNN) is a type of neural network architecture designed specifically for processing sequential data—data where the order matters, such as text, time series, audio, and video. Unlike feedforward networks that process each input independently, RNNs maintain a hidden state that carries information from previous time steps to inform the processing of the current step.
The key innovation of RNNs is the recurrent connection—the output at time t becomes part of the input at time t+1. This creates a form of memory, allowing the network to capture temporal dependencies: the meaning of "the bank" depends on whether the previous words were "I sat on the" (river bank) or "I deposited money in the" (financial bank).
Core Innovation
Recurrent connections that feed output back as input, creating a memory of previous inputs.
Key Advantage
Can process variable-length sequences and capture temporal dependencies between inputs.
Current Status
Largely replaced by transformers for NLP, but still useful for time series and resource-constrained applications.
The "Memory" Metaphor
Think of an RNN reading a sentence word by word. At each word, it updates its "understanding" of the sentence based on the current word and its accumulated understanding so far. By the end of the sentence, the hidden state captures the meaning of the entire sequence. This is fundamentally different from feedforward networks, which would see the entire sentence at once without any sense of order.
How RNNs Work
At each time step t, an RNN computes:
h_t = tanh(W_hh × h_{t-1} + W_xh × x_t + b_h)
Where h_t is the hidden state at time t, x_t is the input at time t, and W_hh and W_xh are weight matrices. The same weights are used at every time step (weight sharing), making RNNs parameter-efficient and able to handle variable-length sequences.
The same RNN cell is applied at each time step. The hidden state flows forward, carrying information from earlier steps to later ones.
Backpropagation Through Time (BPTT)
RNNs are trained using BPTT, which "unrolls" the network across all time steps. The unrolled network is essentially a very deep feedforward network where each layer corresponds to a time step. Gradients are computed by backpropagating through this unrolled network. The deeper the unrolling (longer the sequence), the more the gradients can vanish or explode.
Limitations of RNNs
Vanishing and Exploding Gradients
The fundamental problem: during BPTT, gradients are multiplied many times. If the recurrent weight matrix has eigenvalues < 1, gradients vanish exponentially, and the network can't learn long-range dependencies. If eigenvalues > 1, gradients explode, causing unstable training.
Sequential Processing Bottleneck
RNNs process tokens one at a time—you can't compute h_t until you've computed h_{t-1}. This makes them inherently sequential and slow, unable to leverage GPU parallelism effectively. For a 1000-token sequence, the RNN must go through 1000 sequential steps.
Limited Memory
The hidden state is a fixed-size vector that must compress all information from the entire history. For long sequences, this "bottleneck" means information from early in the sequence gets overwritten by more recent information.
RNN Variants
| Variant | Description | Key Feature |
|---|---|---|
| Vanilla RNN | Basic RNN with tanh activation | Simplest; most affected by vanishing gradients |
| LSTM | Long Short-Term Memory | Gates control information flow; much better at long-range dependencies |
| GRU | Gated Recurrent Unit | Simplified LSTM with fewer gates; similar performance |
| Bidirectional RNN | Processes sequence in both directions | Captures both past and future context |
| Deep RNN | Stacks multiple RNN layers | Higher capacity for learning complex patterns |
RNN vs LSTM vs Transformer
| Feature | RNN | LSTM | Transformer |
|---|---|---|---|
| Long-range dependency | Poor | Good | Excellent |
| Parallelization | None (sequential) | None (sequential) | Full (parallel) |
| Training speed | Slow | Slower | Fast |
| Memory mechanism | Hidden state | Cell state + gates | Self-attention |
| Current dominance | Legacy | Declining | Dominant |
Frequently Asked Questions
The vanishing gradient problem makes it impossible for RNNs to learn dependencies that span many time steps. Information from hundreds of steps ago is effectively lost because the gradient signal has decayed to near zero. LSTMs and GRUs partially address this with gating mechanisms, but even they struggle with very long sequences (thousands of steps). Transformers solved this by replacing sequential processing with direct attention between all positions.
Use RNNs when: (1) You're working on resource-constrained hardware (mobile, embedded) where transformers are too large; (2) You're processing streaming data where you need to update state incrementally; (3) You're doing time series forecasting where the sequential processing is natural; (4) You need extremely low latency and can't afford the quadratic complexity of attention. For most NLP tasks, transformers are the better choice.
Teacher forcing is a training technique where, instead of feeding the RNN's own previous output as the next input, you feed the ground-truth previous output. This stabilizes training by preventing the model from compounding its own errors. During inference, the model must use its own outputs since ground truth isn't available. Scheduled sampling is a compromise that gradually switches from teacher forcing to self-generated inputs.
RNNs naturally handle variable-length sequences because the same weights are applied repeatedly—once per time step. The sequence length determines how many times the weights are applied. For batching, sequences are typically padded to the same length with a special padding token, and the RNN ignores the padding positions. Some frameworks also support dynamic unrolling for truly variable-length sequences.
State-space models (SSMs) like Mamba and S4 are a newer class of architectures that can be viewed as modern descendants of RNNs. They maintain a hidden state like RNNs but use structured state matrices that can be efficiently computed in parallel during training (like transformers) while still being efficient for sequential inference (like RNNs). SSMs are emerging as a promising alternative that combines the best of both worlds.
Continue your AI learning journey
Next, explore LSTM—the gated RNN variant that solved the vanishing gradient problem.
Read Next: LSTM →