AI Glossary: Pre-training
Pre-training is the initial, massive-scale learning phase where models acquire broad knowledge from vast datasets. It's the process that creates foundation models—the versatile AI systems that can be adapted to countless tasks through fine-tuning, making it the cornerstone of modern AI development.
What is Pre-training?
Pre-training is the initial, large-scale training phase where a model learns general-purpose knowledge from a massive dataset before being adapted for specific tasks. Think of it as a broad education: a student first learns general subjects (math, reading, science) before specializing in a career. Similarly, a pre-trained model learns general language understanding, visual patterns, or world knowledge from vast datasets before being specialized through fine-tuning.
Pre-training is what makes modern AI models so powerful and versatile. Instead of training a new model from scratch for every task—which would require massive labeled datasets for each task—you pre-train one large model on abundant unlabeled data, then adapt it to many tasks with relatively little labeled data. This paradigm shift from "train from scratch for each task" to "pre-train once, adapt many times" is arguably the most important practical innovation in AI over the past decade.
Core Idea
Train a model on massive, general data using self-supervised objectives. The resulting model captures broad knowledge transferable to many tasks.
Key Innovation
Self-supervised learning on unlabeled data (e.g., next token prediction, masked language modeling) eliminates the need for expensive labels.
Impact
Enabled the foundation model paradigm: one pre-trained model serves as the base for hundreds or thousands of downstream applications.
Pre-training Changed Everything
Before pre-training became dominant (around 2018 with BERT and GPT), NLP models were trained from scratch for each task using small, task-specific labeled datasets. This meant each new task required collecting expensive labeled data and training a new model. The pre-training revolution changed this: BERT showed that a single pre-trained model could be fine-tuned for 11 different NLP tasks with state-of-the-art results. GPT showed that pre-training on next-token prediction could produce a model capable of zero-shot task performance. This paradigm now dominates all of AI—language, vision, speech, robotics, and beyond.
Pre-training Objectives
The choice of pre-training objective determines what the model learns and what it can do. Here are the major approaches:
Autoregressive Language Modeling (Next Token Prediction)
Used by: GPT, LLaMA, PaLM, Claude. The model is trained to predict the next token in a sequence given all previous tokens. This is a natural self-supervised objective—every text document is its own labeled training example. The model learns grammar, facts, reasoning, and world knowledge implicitly through this simple task. The downside is that the model can only attend to previous tokens (causal attention), making it efficient for generation but requiring special handling for bidirectional understanding.
Masked Language Modeling (MLM)
Used by: BERT, RoBERTa, DeBERTa. Random tokens in the input are masked (replaced with [MASK]), and the model must predict the original tokens from the surrounding context. This forces the model to develop deep bidirectional understanding of language. MLM is excellent for understanding tasks (classification, NER, QA) but the model cannot generate text autoregressively without additional training.
Contrastive Learning
Used by: CLIP, SimCLR, DINO. The model learns to pull representations of related items closer together and push unrelated items apart. CLIP, for example, is trained on 400M image-text pairs to match images with their captions. This produces a shared embedding space where images and text can be compared, enabling zero-shot image classification and powering text-to-image models like DALL-E.
Masked Autoencoding
Used by: MAE (Masked Autoencoder). Similar to BERT's MLM but for images: random image patches are masked, and the model must reconstruct them. This forces the model to learn the visual structure of the world. The asymmetric design (encoder only sees unmasked patches, decoder sees all) makes it computationally efficient for large-scale vision pre-training.
The Foundation Model Paradigm
The term foundation model was coined by Stanford researchers to describe the new paradigm of AI development: a single, massively pre-trained model that serves as the foundation for many downstream applications. This is a fundamental shift from the old paradigm of training task-specific models from scratch.
| Dimension | Traditional ML | Foundation Model Paradigm |
|---|---|---|
| Training | Train from scratch for each task | Pre-train once, adapt many times |
| Data per task | Large labeled dataset required | Small labeled dataset (or none) |
| Model size | Small, task-specific | Massive, general-purpose |
| Task adaptation | Full training from scratch | Fine-tuning, prompting, in-context learning |
| Examples | Custom CNN for defect detection | GPT-4 → custom chatbot, code assistant, etc. |
The pre-training pipeline: a single expensive pre-training phase creates a foundation model that can be inexpensively adapted to many tasks.
Pre-training → Fine-tuning Pipeline
The standard workflow for building modern AI models follows a three-stage pipeline:
Stage 1: Pre-training
The model is trained on a massive, diverse dataset using self-supervised objectives. For language models: trillions of tokens from web pages, books, code, and scientific papers. For vision models: billions of images. This stage requires the most compute and data, but produces a model with broad knowledge. The output is a base model—powerful but not yet specialized for any particular task.
Stage 2: Instruction Tuning / Alignment (Optional)
After pre-training, models like GPT-4 and Claude undergo additional training to make them helpful, harmless, and able to follow instructions. This typically involves supervised fine-tuning (SFT) on high-quality human-written demonstrations, followed by reinforcement learning from human feedback (RLHF) or direct preference optimization (DPO). This stage aligns the model's behavior with human preferences and values.
Stage 3: Task-Specific Fine-tuning
The pre-trained (or instruction-tuned) model is further trained on a smaller, task-specific labeled dataset. This adapts the model's general knowledge to the specific task. For example, a pre-trained language model can be fine-tuned for medical diagnosis, legal document analysis, or customer support. Fine-tuning is much cheaper than pre-training—hours instead of weeks, thousands of examples instead of trillions.
The Economics of Pre-training
Pre-training is the most expensive phase of AI development, but its cost is amortized across all downstream uses:
Cost Breakdown
- Small models (BERT-base, 110M params): ~$1,000-5,000 in GPU compute. Trainable on a single GPU in days.
- Medium models (Llama 3 8B): ~$1-5M in GPU compute. Requires a cluster of GPUs for weeks.
- Large models (Llama 3 70B): ~$10-30M in GPU compute. Requires hundreds of GPUs for weeks.
- Frontier models (GPT-4, Claude 3): Estimated $50-100M+ in GPU compute. Thousands of GPUs for months.
Why Pre-training Is Worth It
The key economic insight: pre-training costs are shared across all downstream uses. If a $100M pre-trained model powers 100 different applications, each application effectively "pays" $1M for the pre-training. If each application would have required $10M to build from scratch, the pre-training approach saves $900M in total. This massive cost leverage is why the foundation model paradigm has taken over the industry.
The Scaling Trend
Pre-training costs are following a predictable trajectory: they're increasing for frontier models (each generation of GPT costs more than the last) but decreasing for equivalent capability. The cost to train a model of GPT-3.5 quality has dropped from ~$10M in 2022 to under $1M in 2024, thanks to better hardware, more efficient training algorithms, and improved data curation. This democratization of pre-training means that high-quality base models are increasingly accessible to startups and researchers, not just tech giants.
Frequently Asked Questions
Data curation is a critical and expensive part of pre-training. The process typically involves: (1) Web scraping at massive scale (Common Crawl, curated web dumps); (2) Deduplication to remove near-duplicate documents; (3) Quality filtering using classifiers to remove low-quality, spam, or toxic content; (4) Heuristic filtering (removing very short documents, documents with high perplexity under a simple model); (5) Mix determination—choosing the right proportions of web text, books, code, academic papers, and multilingual content. Data quality is often more important than data quantity—Llama 3's strong performance is attributed in part to superior data curation.
Pre-training starts from randomly initialized weights and trains on a general dataset. Continued pre-training (also called domain-adaptive pre-training or DAPT) starts from an already pre-trained model and continues training on domain-specific data (e.g., medical texts, legal documents, code). This adapts the model's knowledge to the target domain before task-specific fine-tuning. Continued pre-training is more expensive than fine-tuning but cheaper than full pre-training, and it can significantly improve performance on specialized domains.
For small models, yes. A BERT-base model (~110M parameters) can be pre-trained on a single high-end consumer GPU (RTX 4090) in a few days. GPT-2 small (~124M parameters) can be pre-trained on a single GPU. However, for models above ~1B parameters, pre-training requires multiple enterprise GPUs (A100s, H100s) with high memory bandwidth and fast interconnects. The open-source community has developed techniques like QLoRA and GaLore that enable training of larger models on consumer hardware, but full pre-training of modern LLMs (7B+) still requires significant compute infrastructure.
Pre-training is typically stopped when the validation loss plateaus or when a predetermined compute budget is exhausted. Unlike fine-tuning, where you monitor task-specific metrics, pre-training monitors the language modeling loss (perplexity) on a held-out validation set. The Chinchilla scaling laws suggest an optimal ratio of ~20 tokens per parameter (e.g., a 1B parameter model should be trained on ~20B tokens). However, many recent models are "overtrained" relative to Chinchilla-optimal—Llama 3 8B was trained on 15T tokens (~1,875 tokens per parameter)—because inference is cheaper than training, so it's worth spending more on training to get a smaller, more efficient model.
Catastrophic forgetting occurs when fine-tuning a pre-trained model causes it to lose some of its general pre-trained knowledge. For example, fine-tuning a language model on medical texts might degrade its performance on general conversation. Mitigation strategies include: (1) Mixed fine-tuning—interleaving task-specific data with general pre-training data; (2) Elastic Weight Consolidation—penalizing large changes to important pre-trained weights; (3) Multi-task fine-tuning—training on multiple tasks simultaneously; (4) Parameter-efficient fine-tuning (LoRA, adapters)—modifying only a small number of parameters, preserving the base model's knowledge.
Continue your AI learning journey
Next, explore Benchmarks—the standardized tests that measure and compare AI model performance across tasks and capabilities.
Read Next: Benchmark →