Distillation
Full Form: Knowledge Distillation
Category: AI Techniques
📖 Definition
Distillation trains a smaller 'student' model to mimic a larger 'teacher' model. The student learns to approximate the teacher's outputs, resulting in a compact model with good performance.
🔑 Key Points
- Creates smaller, faster models from larger ones
- Student learns from teacher's probability distributions
- Enables deploying capable models on limited hardware
- Key technique for making AI efficient
💡 Why It Matters
Distillation makes AI more accessible by creating efficient models that can run on devices without powerful GPUs.