AI Glossary: Transformer Architecture

What Is the Transformer Architecture?

The Transformer architecture is a deep learning model design introduced in 2017 that relies entirely on a mechanism called self-attention to process sequential data. Unlike previous architectures — such as recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) — the Transformer does not process tokens one at a time in order. Instead, it computes relationships between every pair of tokens in the sequence simultaneously.

This parallel processing capability is what makes Transformers radically more efficient and scalable than their predecessors. A Transformer can process an entire sentence, paragraph, or document in a single forward pass rather than iterating token by token. This architectural innovation directly enabled the training of models with hundreds of billions of parameters on internet-scale datasets.

The Transformer architecture consists of two main components: an encoder that processes the input sequence into a rich contextual representation, and a decoder that generates output tokens one at a time, attending to both the encoder's output and previously generated tokens. However, many of today's most famous models (like GPT) use only the decoder portion, while others (like BERT) use only the encoder.

Origins: The "Attention Is All You Need" Paper

In June 2017, eight researchers at Google Brain and Google Research — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin — published a paper titled "Attention Is All You Need" at the NeurIPS conference. The paper's title was a bold statement: it claimed that attention mechanisms alone, without any recurrence or convolution, were sufficient to achieve state-of-the-art results on machine translation tasks.

The paper demonstrated that a Transformer model trained on the WMT 2014 English-to-German translation task achieved a BLEU score of 28.4, surpassing all previous models including ensembles. On English-to-French, it reached 41.0 BLEU. More importantly, the Transformer trained in just 3.5 days on 8 GPUs — dramatically faster than the RNN-based alternatives which required weeks of training.

While the paper initially focused on machine translation, researchers quickly recognized that the Transformer architecture was far more general. Within months, encoder-only variants (BERT) revolutionized natural language understanding, and decoder-only variants (GPT) began the generative AI revolution that continues to this day.

Core Components of a Transformer

Understanding the Transformer requires breaking it down into its constituent building blocks. Each component plays a specific role in the overall information processing pipeline:

1. Input Embedding

Before any attention computation happens, input tokens (words, subwords, or characters) are converted into dense vectors of fixed dimensionality — typically 512, 768, 1024, or more dimensions. These embedding vectors capture semantic meaning: similar words end up with similar vector representations. The embedding layer is learned during training and becomes a rich semantic map of the model's vocabulary.

2. Positional Encoding

Because the Transformer processes all tokens simultaneously (rather than sequentially), it has no inherent sense of word order. The sentence "The cat sat on the mat" and "mat the on sat cat The" would look identical to a Transformer without positional information. Positional encoding solves this by adding unique position-dependent values to each token's embedding. The original paper used sinusoidal functions of different frequencies, but modern models often use learned positional embeddings.

3. Multi-Head Self-Attention

This is the heart of the Transformer. Multi-head attention runs multiple attention operations in parallel, each with its own learned Query, Key, and Value projections. Each "head" can learn to attend to different types of relationships — one head might focus on syntactic structure, another on semantic similarity, another on long-range dependencies. The outputs of all heads are concatenated and projected to produce the final attention output.

4. Feed-Forward Network (FFN)

After attention, each token's representation passes through a position-wise feed-forward network — the same two-layer MLP applied independently to each position. This introduces non-linearity and increases the model's capacity to learn complex transformations. Typically, the FFN expands the dimension (e.g., from 512 to 2048), applies a ReLU or GELU activation, then projects back down.

5. Residual Connections and Layer Normalization

Each sub-layer (attention and FFN) is wrapped with a residual connection: the input is added to the sub-layer's output before normalization. This helps gradients flow during backpropagation, enabling the training of very deep networks. Layer normalization stabilizes training by normalizing activations across the feature dimension.

How Self-Attention Works: A Step-by-Step Guide

Self-attention is the mechanism that gives the Transformer its power. Here's exactly how it computes contextual representations:

Step 1: Create Query, Key, and Value Vectors

For each input token, the model creates three vectors by multiplying the token's embedding by learned weight matrices W_Q, W_K, and W_V. The Query represents "what am I looking for?", the Key represents "what do I contain?", and the Value represents "what information do I offer?"

Step 2: Compute Attention Scores

For each pair of tokens (i, j), the model computes a score by taking the dot product of token i's Query vector and token j's Key vector. A higher score means token j is more relevant to token i's context.

Step 3: Scale and Normalize

The scores are divided by √d_k (where d_k is the dimension of the Key vectors) to prevent the dot products from growing too large, which would push the softmax into regions of extremely small gradients. Then, a softmax function converts the scaled scores into a probability distribution — attention weights that sum to 1.

Step 4: Weighted Sum of Values

The final output for each token is the weighted sum of all Value vectors, where the weights are the attention probabilities computed in Step 3. Tokens that are highly relevant to the current token contribute more to its output representation.

Why Self-Attention Beats Recurrence

Encoder, Decoder, and Hybrid Architectures

Not all Transformers are the same. The original encoder-decoder design has spawned three major architectural families:

Encoder-Only Models

Example: BERT (Bidirectional Encoder Representations from Transformers). These models use only the encoder stack and are designed for understanding tasks. They process the entire input bidirectionally — every token attends to every other token, both left and right. This makes them ideal for classification, named entity recognition, question answering, and semantic search. BERT's bidirectional nature means it cannot generate text autoregressively; it's a "reader," not a "writer."

Decoder-Only Models

Example: GPT (Generative Pre-trained Transformer). These models use only the decoder stack with causal (masked) self-attention, meaning each token can only attend to previous tokens (and itself). This autoregressive constraint makes them natural text generators: they predict the next token given all previous tokens. GPT-4, Claude, LLaMA, Mistral, and virtually all modern chatbots use decoder-only architectures. Their strength is fluent, coherent text generation at scale.

Encoder-Decoder Models

Example: T5 (Text-to-Text Transfer Transformer). These retain the full original architecture. The encoder processes the entire input sequence into a dense representation, and the decoder generates output autoregressively while attending to the encoder's representation via cross-attention. This design shines for tasks where the output is a transformation of the input: translation, summarization, and text-to-SQL conversion.

Beyond Text: Vision Transformers and Multimodal Models

The Transformer architecture has proven remarkably versatile. Vision Transformers (ViT) apply self-attention to image patches instead of text tokens, achieving state-of-the-art results on image classification. Multimodal models like GPT-4V, Gemini, and Claude combine Transformer-based text processing with vision encoders to understand images alongside text. Even audio models like Whisper and video models use Transformer backbones — the architecture's core insight (parallel attention over a sequence of representations) applies to any modality.

Why Transformers Changed AI Forever

The Transformer's impact on AI cannot be overstated. It is arguably the single most important architectural innovation in the history of machine learning. Here's why:

Ongoing Research and Future Directions

While Transformers dominate, researchers are actively exploring alternatives and improvements. State Space Models (like Mamba) aim to achieve linear complexity in sequence length (versus the Transformer's quadratic cost in self-attention). Mixture of Experts (MoE) routes tokens to specialized sub-networks, enabling larger models without proportional compute increases. Retrieval-Augmented Generation (RAG) augments Transformers with external knowledge bases to reduce hallucination. The Transformer may eventually be superseded, but its core principles — attention, parallelization, and scalable architecture — will influence AI design for decades.

Frequently Asked Questions