AI Glossary: Fine-tuning

What is Fine-tuning?

Fine-tuning is a transfer learning technique where you take a machine learning model that has already been pre-trained on a large, general dataset, and you continue training it on a smaller, task-specific or domain-specific dataset. The goal is to adapt the existing knowledge the model already learned to the new specific task.

During pre-training, models learn general patterns, features, and representations from massive amounts of data. For example, a large language model learns grammar, facts, reasoning patterns, and language structure from trillions of tokens. A computer vision model learns edges, textures, shapes, and object concepts from millions of images.

Fine-tuning builds on this foundation. Instead of training a model from scratch—which requires enormous amounts of data, compute, and time—you take the pre-trained model and adjust its weights slightly to perform better on your specific problem.

Why Do We Need Fine-tuning?

Pre-trained models are general-purpose. A 7B parameter language model trained on internet text knows a lot about almost everything, but it may not be good at specific tasks like:

• Legal document analysis — requires understanding legal terminology and document structure
• Medical diagnosis — requires domain-specific knowledge and accuracy constraints
• Customer support automation — requires understanding your company's products and policies
• Code generation in a specific programming language or framework

Instead of training a whole new model from scratch (which would be prohibitively expensive), you can fine-tune an existing pre-trained model on your domain-specific data to get excellent performance.

How Fine-tuning Works

The fine-tuning process follows a straightforward workflow:

1. Start with a pre-trained model — You begin with a model that's already been trained on a large general dataset (e.g., BERT, GPT, ResNet, ViT).
2. Modify the model architecture if needed — Sometimes you replace or add output layers to match your specific task requirements.
3. Initialize with pre-trained weights — The model starts with all the knowledge it already learned.
4. Train on task-specific data — You continue training using your smaller dataset, typically with a lower learning rate than pre-training.
5. Update model weights — The model adjusts its internal weights to better fit the new task.

The Learning Rate Question

A key consideration in fine-tuning is the learning rate. You typically use a much lower learning rate for fine-tuning than you would for training from scratch. Why?

• The pre-trained weights are already good
• You just need to make small adjustments
• A high learning rate would overwrite good features and cause catastrophic forgetting
• A lower learning rate preserves the general knowledge while allowing adaptation

Common Fine-tuning Approaches

There are several approaches to fine-tuning, depending on your constraints and goals:

1. Full Fine-tuning

In full fine-tuning, you update all the parameters of the pre-trained model. This gives you the most flexibility and the potentially best performance, but it also requires the most compute and memory. For large models like 70B parameter LLMs, full fine-tuning is expensive.

2. Feature Extraction (Frozen Backbone)

In this approach, you keep the entire pre-trained model frozen (no updates to weights) and only train a classification or output head on top. This is the cheapest approach but typically gives lower performance because the features aren't adapted.

3. Partial Fine-tuning

You freeze the bottom layers (which capture general low-level features) and only fine-tune the top layers (which capture more task-specific high-level features). This is a good compromise between compute and performance.

4. Parameter-Efficient Fine-tuning (PEFT)

PEFT techniques keep the base model frozen and only train a small number of additional parameters. This gives performance comparable to full fine-tuning with much lower compute requirements.

5. Progressive Unfreezing

This approach starts with the entire model frozen and gradually unfreezes layers from top to bottom during training. It helps preserve low-level features while adapting higher-level features gradually.

Best Practices for Fine-tuning

Following these best practices will help you get better results and avoid common pitfalls:

Data Preparation

• Clean your data — Fine-tuning is only as good as your data. Remove bad examples, correct errors, deduplicate.
• Use consistent formatting — Match the formatting the model expects in terms of prompt structure, special tokens, etc.
• Have a validation set — Always monitor performance on a held-out validation set to detect overfitting.
• Mix in some general data — If possible, include a small amount of general data to reduce catastrophic forgetting.

Training Hyperparameters

• Use smaller learning rates — Typically 1-10x smaller than pre-training. Common values: 1e-5 to 5e-5 for transformers.
• Use early stopping — Stop training when validation performance plateaus to avoid overfitting.
• Use batch size as large as your GPU allows — Larger batches give more stable gradients.
• Keep gradient accumulation in mind — If batch size is small, accumulate gradients over multiple steps.

Handling Catastrophic Forgetting

Catastrophic forgetting happens when fine-tuning causes the model to lose the general knowledge it had before. Strategies to mitigate:

• Use smaller learning rates
• Use parameter-efficient methods that only update a small subset of parameters
• Add regularization (weight decay, dropout)
• Use elastic weight consolidation to protect important weights
• Mix pre-training data into your fine-tuning batches

Fine-tuning vs Other Approaches

It's useful to understand how fine-tuning compares to other common approaches for leveraging pre-trained models:

Fine-tuning vs RAG

A common question is when to use fine-tuning vs retrieval-augmented generation (RAG). The answer: they solve different problems.

• Use RAG when — You need the model to access specific, frequently changing information (e.g., your company's documentation, latest product specs). RAG retrieves relevant information at inference time and doesn't require changing model weights.
• Use Fine-tuning when — You need to adapt the model's behavior, output format, writing style, or to teach it new skills that require changing how the model thinks. Fine-tuning changes the model weights.

You can also use them together: fine-tune the model to follow instructions well, and use RAG to provide it with up-to-date specific information.