AI Glossary: Tokenizer

What Is a Tokenizer?

A tokenizer is the component that converts raw text into a sequence of tokens — the discrete units that a language model can process. Before any neural network computation happens, every input to an LLM passes through a tokenizer. It is the first and most fundamental preprocessing step in every NLP pipeline.

Tokens are not necessarily words. A token might be a whole word ("the"), a subword piece ("tion"), a single character ("a"), or even a byte. The tokenizer's job is to segment text into these units and map each one to a unique integer ID from a fixed vocabulary. The model never sees raw text — it only sees sequences of token IDs.

The tokenization process has two directions: encoding (text → token IDs) and decoding (token IDs → text). Both must be lossless and deterministic — the same input should always produce the same tokens, and decoding those tokens should reconstruct the original text. The tokenizer is trained once on a large corpus, then frozen and used consistently for all subsequent model training and inference.

Types of Tokenization: Word, Character, and Subword

Tokenization strategies exist on a spectrum from coarse (whole words) to fine (individual bytes). Each approach has distinct tradeoffs:

Word-Level Tokenization

The simplest approach: split on whitespace and punctuation. "The cat sat." → ["The", "cat", "sat", "."]. This is intuitive but has severe limitations. The vocabulary must contain every possible word form — including plurals, conjugations, and misspellings — leading to vocabularies of millions of entries. Any word not in the vocabulary becomes an [UNK] (unknown) token, losing all information. For morphologically rich languages like Turkish or Finnish, where a single word can have hundreds of forms, word-level tokenization is essentially unusable.

Character-Level Tokenization

The opposite extreme: split into individual characters. "cat" → ["c", "a", "t"]. The vocabulary is tiny (26 letters + punctuation for English, ~150K for all Unicode), guaranteeing no unknown tokens. However, sequences become very long — a 10-word sentence might produce 50+ tokens. This strains the attention mechanism (which has O(n²) complexity) and makes it harder for the model to learn word-level semantics, since words are split across many tokens.

Subword Tokenization

The goldilocks solution used by all modern LLMs. Subword tokenization splits text into meaningful subword units: common words stay whole ("the", "cat"), while rare words are decomposed into frequent pieces ("uncommon" → ["un", "common"]). This balances vocabulary size (typically 32K–256K), sequence length, and coverage. The key insight: frequent substrings should be their own tokens. This is determined by statistical analysis of the training corpus.

Byte Pair Encoding (BPE) in Depth

Byte Pair Encoding (BPE) is the most widely used tokenization algorithm, powering GPT models, LLaMA, Mistral, and many others. Originally a data compression algorithm (Gage, 1994), it was adapted for NLP by Sennrich et al. (2016). Here's how it works:

The BPE Training Algorithm

1. Initialize: Start with a vocabulary containing every unique character (or byte) in the training corpus. Split every word into characters.
2. Count pairs: Count the frequency of every adjacent pair of tokens across the entire corpus.
3. Merge: Find the most frequent pair and merge it into a single new token. Add this token to the vocabulary.
4. Repeat: Go back to step 2. Repeat for a pre-specified number of merges (e.g., 50,000 merges produces a vocabulary of base characters + 50,000).

For example, if "l" and "o" appear together very frequently (in "low", "hello", "follow"), BPE will merge them into "lo". Later, if "lo" and "w" appear together frequently, they merge into "low". Common words like "the" and "and" survive as single tokens; rare words are split into frequent subword pieces.

Byte-Level BPE

Modern BPE implementations (like GPT's tokenizer) operate on bytes rather than Unicode characters. This means the base vocabulary is just 256 tokens (one per byte value), and the tokenizer can handle any input — emoji, code, mathematical symbols, even binary data — without ever producing an unknown token. GPT-4's cl100k_base tokenizer starts with 256 byte tokens and applies ~100,000 merges to produce a vocabulary of ~100,256 tokens.

WordPiece, Unigram, and SentencePiece

While BPE dominates, several other subword tokenization algorithms are important:

Algorithm	Used By	Selection Criterion	Key Difference from BPE
BPE	GPT, LLaMA, Mistral, RoBERTa	Merge most frequent pair	Greedy; merges are never undone
WordPiece	BERT, DistilBERT	Merge pair that maximizes training data likelihood	Uses likelihood rather than raw frequency; uses "##" prefix for continuation
Unigram	T5, mBART, XLNet	Prune tokens that least affect likelihood	Starts with large vocabulary and prunes; probabilistic rather than deterministic
SentencePiece	LLaMA, T5, ALBERT	Framework, not algorithm	Treats input as raw text (no pre-tokenization); handles whitespace as tokens; language-agnostic

SentencePiece: Language-Agnostic Tokenization

SentencePiece (Kudo & Richardson, 2018) is not a tokenization algorithm itself but a framework that implements BPE or Unigram with a crucial design choice: it treats the input as a raw byte stream with no language-specific preprocessing. Whitespace is treated as a regular character (represented as "▁", a special underscore), and there's no assumption about word boundaries. This makes SentencePiece truly language-agnostic — it works equally well for English (space-delimited), Japanese (no spaces), and Thai (no spaces). LLaMA models use SentencePiece with BPE, which is why you see "▁" tokens in LLaMA tokenizer outputs.

Practical Impact: Cost, Context, and Fairness

Tokenization choices have real-world consequences that affect every LLM user:

API Cost

LLM APIs charge per token. If your text tokenizes to 1,000 tokens vs. 800 tokens (due to tokenizer differences), you pay 25% more for the same content. This is especially significant for non-English languages, code, and structured data, which often tokenize less efficiently than English prose.

Context Window Utilization

When an LLM has a 128K context window, that's 128K tokens, not 128K words. A language that averages 2 tokens per word can fit twice as much content as one that averages 4 tokens per word. This is the tokenization tax: non-English users effectively have smaller context windows.

Numbers and Math

Tokenizers often struggle with numbers. GPT-4's tokenizer might split "1234567890" into ["123", "456", "7890"] or other arbitrary chunks, making it harder for the model to perform arithmetic. This is why LLMs sometimes fail at simple math — the number "123" and "456" being separate tokens means the model must learn to compose them, rather than inherently understanding the numeric value.

Tokenizers Across Major LLMs

Here's how the major LLM families handle tokenization:

Model Family	Tokenizer	Algorithm	Vocab Size	Special Notes
GPT-4 / GPT-3.5	cl100k_base (tiktoken)	Byte-level BPE	~100,256	Byte-fallback ensures no UNK tokens; widely used as reference
Claude (Anthropic)	Custom BPE	BPE	~100,000	Proprietary; details not fully public
LLaMA 2/3 (Meta)	SentencePiece BPE	BPE	32,000	Smaller vocabulary; uses SentencePiece framework
Mistral	SentencePiece BPE	BPE	32,000	Similar to LLaMA; open-source
BERT (Google)	WordPiece	WordPiece	30,522	Uses "##" prefix for subword continuations
Gemini (Google)	SentencePiece	Unigram	256,000	Large vocabulary for multilingual support

Frequently Asked Questions