AI Glossary: Zero-shot Learning
Zero-shot learning is the ability of AI models to recognize or classify objects, concepts, or categories they were never explicitly trained on. It leverages semantic knowledge to bridge the gap between seen and unseen classes.
What is Zero-shot Learning?
Zero-shot learning (ZSL) is a machine learning paradigm where a model can correctly identify, classify, or understand objects that it was never trained on. Unlike traditional supervised learning, which requires labeled examples of every class, ZSL models can generalize to completely new categories by leveraging auxiliary information about those categories.
The fundamental idea is that knowledge about classes can be decomposed into semantic properties or attributes that are shared across both seen and unseen classes. For example, if a model has been trained to recognize zebras (striped, horse-like) and tigers (striped, cat-like), it might be able to recognize an okapi (striped, giraffe-like) without ever seeing one, by combining the shared attribute "striped" with the general shape of giraffes.
Core Concept
Recognize unseen classes by leveraging semantic descriptions, attributes, or embeddings shared between seen and unseen categories.
Key Distinction
Unlike traditional ML, training and test classes are completely disjoint—the model is tested on classes it has never seen.
Real-world Analogy
A child who has never seen a zebra but has seen horses and knows what "stripes" are can identify a zebra from a description.
The Semantic Gap Problem
At its heart, zero-shot learning is about bridging the gap between visual or textual features and semantic meaning. The model must learn a mapping from low-level features (pixels, tokens) to a high-level semantic space where similarity between classes can be measured. This semantic space is typically defined by:
- Attributes — Human-defined properties like "has stripes," "is furry," "can fly"
- Word embeddings — Dense vector representations of class names (e.g., Word2Vec, GloVe)
- Text descriptions — Natural language descriptions of each class
- Class hierarchies — Taxonomic relationships between classes
Key Insight
Zero-shot learning works because the world is compositional. Concepts are made of shared building blocks (attributes, features, properties). By learning these building blocks, a model can combine them in novel ways to understand unseen concepts.
How Zero-shot Learning Works
The standard ZSL pipeline involves three key components:
- Feature Extractor — A neural network (e.g., CNN, ViT, BERT) that converts raw input (images, text) into a feature vector.
- Semantic Space — A shared embedding space where both class labels and features are represented. Each class has a semantic representation (e.g., attribute vector, word embedding).
- Compatibility Function — A function that measures how well a feature vector matches a semantic class representation. The class with the highest compatibility score is the prediction.
The ZSL pipeline: extract features from unseen input, then find the closest matching class in the shared semantic space.
Generalized Zero-shot Learning (GZSL)
In practice, the most realistic setting is Generalized Zero-shot Learning (GZSL). In standard ZSL, the test set only contains unseen classes. In GZSL, the test set contains both seen and unseen classes—just like the real world. This is much harder because models tend to be biased toward seen classes. GZSL requires the model to balance classification accuracy across both seen and unseen categories.
Main Approaches and Methods
1. Attribute-based Methods
The earliest and most intuitive approach. Human annotators define a set of attributes that describe each class (e.g., "has fur," "is brown," "has hooves"). The model learns to predict these attributes from raw input, then matches the predicted attributes to the attribute vectors of all classes.
2. Embedding-based Methods
These methods learn a mapping from the feature space to the semantic embedding space directly. The model is trained so that the image/text embedding of a sample is close to the semantic embedding of its class label. Popular approaches include DeViSE, ALE, and SJE.
3. Generative Methods
A more recent and powerful approach: instead of learning a direct mapping, these methods use generative models (GANs, VAEs, or diffusion models) to synthesize features for unseen classes. The synthetic features are then used to train a standard classifier. This effectively turns the ZSL problem into a supervised learning problem.
GAN-based ZSL
Uses conditional GANs to generate visual features for unseen classes, conditioned on their semantic embeddings. The generator learns to produce realistic features that the discriminator can't distinguish from real features.
VAE-based ZSL
Uses variational autoencoders to learn a latent distribution for each class from its semantic embedding. New features for unseen classes are sampled from their learned distributions.
Diffusion-based ZSL
Uses diffusion models to generate features for unseen classes by iteratively denoising random noise. These models often produce higher-quality features than GANs or VAEs.
4. Transductive ZSL
In transductive ZSL, the model has access to unlabeled examples from unseen classes during training (but not their labels). This additional information allows the model to better understand the distribution of unseen classes and reduce domain shift.
Zero-shot in Large Language Models
In the context of large language models (LLMs), zero-shot learning takes on a slightly different meaning. Here, it refers to the model's ability to perform a task based solely on natural language instructions, without any task-specific examples in the prompt.
For example, a zero-shot prompt might be: "Translate the following text to French: 'Hello, how are you?'" The LLM has never been fine-tuned on translation specifically, but it can perform the task because it learned about translation during its pre-training on internet text.
Why LLMs Excel at Zero-shot
- Massive pre-training — LLMs are trained on trillions of tokens covering diverse tasks, so they've seen examples of almost everything during pre-training.
- Instruction following — Models like GPT-4 and Claude are specifically trained to follow instructions, making them naturally good at zero-shot task execution.
- Emergent abilities — As models scale up, they develop abilities that weren't explicitly programmed, including zero-shot reasoning, translation, summarization, and code generation.
Zero-shot vs In-context Learning
Zero-shot in LLMs means providing only the instruction with no examples. Few-shot (or in-context learning) means providing a few examples in the prompt. The key difference: zero-shot relies entirely on the model's pre-existing knowledge, while few-shot provides additional context to guide the model's behavior.
Challenges and Limitations
Despite its impressive capabilities, zero-shot learning faces several significant challenges:
The Hubness Problem
In high-dimensional spaces, some points become "hubs"—they are the nearest neighbor to many other points. This causes the model to predict the same few classes for many different inputs, severely degrading performance. This is a fundamental issue in high-dimensional nearest-neighbor search.
Domain Shift
The distribution of features for seen and unseen classes can be very different. The model learns to map seen classes to the semantic space, but this mapping may not work well for unseen classes if their feature distributions are different.
Bias Toward Seen Classes
In the GZSL setting, models tend to classify inputs as seen classes because those are what they were trained on. This bias toward seen classes is a major challenge that researchers actively work to mitigate.
Semantic Representation Quality
The quality of zero-shot predictions depends heavily on the quality of the semantic representations. Poorly defined attributes or noisy word embeddings will lead to poor predictions.
Zero-shot vs Few-shot vs Supervised
| Property | Zero-shot | Few-shot | Supervised |
|---|---|---|---|
| Training examples per class | 0 | 1-10 | Hundreds to millions |
| Relies on semantic info | Yes (essential) | Partially | No |
| Accuracy on unseen classes | Moderate | Better than ZSL | Best (but needs training data) |
| Domain shift sensitivity | High | Medium | Low |
| Setup complexity | High (needs semantic data) | Medium | Low |
| Best use case | Many classes, no data | Few examples available | Abundant labeled data |
Frequently Asked Questions
In zero-shot learning, the model has zero training examples of the target class—it must rely entirely on semantic descriptions. In one-shot learning, the model is given exactly one training example of the target class, which it can use to learn about that class. One-shot learning is easier because the model has at least one concrete example to work with, while zero-shot requires pure generalization from semantic knowledge.
Zero-shot learning can work for any data type as long as there's a meaningful semantic space connecting the classes. In practice, it works best when: (1) there's a well-defined semantic representation for classes, (2) the seen and unseen classes share meaningful attributes, and (3) the feature space captures the relevant semantic properties. It has been successfully applied to images, text, audio, and video.
In standard ZSL, evaluation uses top-1 or top-5 accuracy on unseen classes only. In GZSL, the harmonic mean of seen and unseen class accuracy is the standard metric, as it penalizes models that favor one over the other. The harmonic mean H = 2 * (acc_seen * acc_unseen) / (acc_seen + acc_unseen) provides a balanced measure of performance across both sets.
CLIP (Contrastive Language-Image Pre-training) by OpenAI is a landmark model in zero-shot learning. It's trained on 400 million image-text pairs to align visual and textual representations in a shared embedding space. This allows CLIP to perform zero-shot image classification by simply comparing an image embedding to text embeddings of class names like "a photo of a dog" or "a photo of a cat." CLIP's success demonstrated that large-scale multimodal pre-training is a powerful approach to zero-shot learning.
No, they are different. Transfer learning involves adapting a model trained on one task to perform well on a different but related task (typically using additional training data from the target task). Zero-shot learning, by contrast, requires no task-specific training data at all. The model must perform the new task using only its pre-existing knowledge and semantic information about the target classes.
Continue your AI learning journey
Next, explore Few-shot learning—where models learn from just a handful of examples.
Read Next: Few-shot Learning →