AI Glossary: Reinforcement Learning
A comprehensive guide to reinforcement learning — the AI paradigm where agents learn through trial, error, and reward. From Markov decision processes to deep RL, discover how AlphaGo mastered Go and how RLHF shapes today's most helpful LLMs.
📑 In This Glossary Entry
What Is Reinforcement Learning?
Reinforcement Learning (RL) is a machine learning paradigm where an agent learns to make sequential decisions by interacting with an environment. The agent takes actions, observes the resulting state changes, and receives rewards (positive or negative) that signal the quality of its decisions. Through repeated trial and error, the agent learns a policy — a strategy that maps situations to actions — that maximizes cumulative reward over time.
RL is fundamentally different from supervised learning. In supervised learning, the model learns from labeled examples that show the correct answer. In RL, the agent must discover which actions are good through exploration — there's no teacher saying "this is the right move." The agent only receives a reward signal that may be delayed (you might not know if a chess move was good until many moves later) and sparse (rewards may only come at the end of a game).
Reinforcement learning is like training a dog. You can't tell the dog exactly what to do in every situation ("sit," "stay," "roll over") — instead, you give treats (positive rewards) when it does something good and withhold treats (negative reward) when it does something bad. Over time, the dog learns which behaviors lead to treats. The dog is the agent, the treats are rewards, and the learned behaviors form its policy. RL algorithms are the mathematical formalization of this learning process.
The RL Framework: Agent, Environment, and Reward
RL is formalized through Markov Decision Processes (MDPs), which provide a mathematical framework for sequential decision-making under uncertainty:
State (S)
The current situation the agent observes. In chess, the state is the board position. In self-driving, it's the sensor readings. Must satisfy the Markov property.
Action (A)
What the agent can do in a given state. Discrete actions (move left, move right) or continuous (steering angle, acceleration). The action space defines the agent's capabilities.
Reward (R)
Immediate feedback after each action. The agent's goal is to maximize cumulative (discounted) reward, not just immediate reward. The reward function defines the task.
Policy (π)
The agent's strategy: π(a|s) = probability of taking action a in state s. The policy is what the agent learns. Can be deterministic or stochastic.
🔷 The RL Loop
Agent observes State → Takes Action → Environment transitions → Agent receives Reward → Repeat
Value Functions and the Bellman Equation
A value function estimates the expected cumulative reward from a given state (or state-action pair) when following a policy. The Bellman equation is the recursive relationship that defines optimal values: V*(s) = max_a [R(s,a) + γ · Σ P(s'|s,a) · V*(s')]. This states that the optimal value of a state is the maximum over actions of the immediate reward plus the discounted expected future value. The discount factor γ (0 to 1) determines how much the agent cares about future rewards vs. immediate rewards.
Key Algorithms: Q-Learning, Policy Gradients, and PPO
| Algorithm | Type | How It Works | Best For |
|---|---|---|---|
| Q-Learning | Value-based, model-free | Learns Q(s,a) - the value of taking action a in state s. Updates Q-values using the Bellman equation. Off-policy (learns from any experience). | Discrete action spaces; tabular or small state spaces |
| DQN (Deep Q-Network) | Value-based, deep RL | Uses a neural network to approximate Q-values. Key innovations: experience replay and target network for stability. | Atari games; high-dimensional state spaces |
| Policy Gradient (REINFORCE) | Policy-based, model-free | Directly optimizes the policy by gradient ascent on expected reward. No value function needed (though actor-critic adds one). | Continuous action spaces; stochastic policies |
| Actor-Critic (A2C/A3C) | Hybrid | Combines policy-based (actor) and value-based (critic). Actor chooses actions; critic evaluates them. Reduces variance. | General-purpose; robotics; game playing |
| PPO (Proximal Policy Optimization) | Policy-based, deep RL | Clips policy updates to prevent destructive large changes. Simple, robust, and the default choice for many applications. | RLHF for LLMs; robotics; continuous control |
PPO (Schulman et al., 2017) has become the go-to RL algorithm because it's simple, robust, and works well across diverse domains. Its key innovation is a clipped objective that prevents the policy from changing too much in a single update, which was a major source of instability in earlier policy gradient methods. PPO is the algorithm used in RLHF to align ChatGPT, Claude, and other LLMs. It's also widely used in robotics and game playing.
Deep Reinforcement Learning: The Neural Network Revolution
Deep Reinforcement Learning combines RL with deep neural networks, enabling agents to learn from high-dimensional inputs like raw pixels. The breakthrough came in 2013 when DeepMind's DQN learned to play 49 Atari 2600 games directly from screen pixels, surpassing human performance on many of them. This demonstrated that neural networks could learn effective representations and policies from raw sensory input through RL alone.
Key innovations that made deep RL work include: Experience replay — storing past experiences in a buffer and sampling randomly for training, which breaks harmful temporal correlations; Target networks — using a separate, slowly-updated network for computing target Q-values, which stabilizes learning; and reward clipping — normalizing reward magnitudes to prevent gradient explosion. These techniques transformed RL from a niche field to a mainstream AI approach.
RLHF: Aligning LLMs with Human Preferences
Reinforcement Learning from Human Feedback (RLHF) is arguably the most impactful RL application in recent years. It's the technique that turns a raw language model (which just predicts next tokens) into a helpful assistant (which follows instructions, refuses harmful requests, and provides useful information).
The RLHF pipeline has three stages: (1) Supervised fine-tuning (SFT) — fine-tune the base model on high-quality human-written demonstrations of desired behavior; (2) Reward model training — collect human preference data (human raters compare two model outputs and pick the better one), then train a reward model to predict these preferences; (3) PPO fine-tuning — use the reward model to score the LLM's outputs and update the LLM via PPO to maximize the reward, while adding a KL penalty to prevent the model from diverging too far from the SFT model.
RLHF addresses the alignment problem: how do we ensure AI systems behave in ways that align with human values and intentions? A raw LLM trained on internet text might generate harmful content, refuse to follow instructions, or be unhelpful. RLHF shapes the model's behavior by rewarding outputs that humans prefer. This is why ChatGPT is polite and helpful while the base GPT-3 model was not — RLHF aligned the model's behavior with human expectations.
Landmark RL Breakthroughs
TD-Gammon (1992)
Gerald Tesauro's backgammon program learned to play at world-champion level using temporal difference learning, demonstrating RL's potential for game playing.
DQN (2013)
DeepMind's system learned to play 49 Atari games from raw pixels, surpassing human performance on many. This launched the deep RL era.
AlphaGo (2016)
Defeated world champion Lee Sedol 4-1 in Go, a game with more possible positions than atoms in the universe. Used deep neural networks + Monte Carlo Tree Search.
ChatGPT (2022)
Applied RLHF at scale to align GPT-3.5 with human preferences, creating the most rapidly adopted consumer product in history.
Frequently Asked Questions
Q: How is RL different from supervised learning?
A: In supervised learning, the model learns from labeled examples showing the correct answer. In RL, there are no labeled examples — the agent learns through interaction, receiving only a reward signal that may be delayed, sparse, and noisy. RL involves sequential decision-making where actions affect future states, requiring the agent to consider long-term consequences. Supervised learning is like learning from a textbook with answers; RL is like learning to ride a bike by falling down and getting up.
Q: What is the credit assignment problem?
A: The credit assignment problem is the challenge of determining which past actions contributed to a current reward or punishment. In chess, if you win after 40 moves, which moves were good and which were bad? The final move (checkmate) is clearly good, but what about the opening move that set up the winning position? Temporal difference learning addresses this by propagating rewards backward through time, assigning credit to states and actions that led to good outcomes.
Q: Can RL be used for training all AI models?
A: No — RL is best suited for sequential decision-making problems where there's a clear reward signal and the agent can interact with an environment. It's not well-suited for tasks like image classification (where supervised learning excels) or clustering (where unsupervised learning excels). RL is most valuable when the correct action isn't known in advance and must be discovered through trial and error. Many modern AI systems combine RL with other paradigms — for example, LLMs are pre-trained with self-supervised learning, then fine-tuned with RLHF.
Q: What is the sample efficiency problem in RL?
A: RL algorithms typically require millions or billions of interactions to learn effectively. AlphaGo played millions of games against itself during training. This is sample-inefficient compared to humans, who can learn from far fewer examples. Sample efficiency is a major barrier to applying RL in the real world (e.g., robots can't crash millions of times). Approaches to improve sample efficiency include model-based RL, offline RL (learning from fixed datasets), transfer learning, and meta-learning.
Q: What is multi-agent reinforcement learning?
A: Multi-agent RL extends RL to environments with multiple agents that may cooperate or compete. Each agent has its own policy and objectives, and the environment includes other learning agents. This creates non-stationary dynamics — what's optimal for one agent depends on what others are doing. Applications include multi-robot coordination, autonomous vehicle fleets, trading agents, and game AI. OpenAI Five (Dota 2) and DeepMind's AlphaStar (StarCraft II) are notable multi-agent RL successes.
🧠 Continue Your AI Glossary Journey
Now that you understand reinforcement learning, explore the most common ML paradigm.
Next: Supervised Learning →