AI Glossary: BLEU Score
BLEU (Bilingual Evaluation Understudy) is the standard metric for evaluating machine translation quality. By comparing n-gram overlap between machine output and human reference translations, BLEU provides an automatic, reproducible measure of translation quality that has driven progress in machine translation for over two decades.
What is BLEU?
BLEU (Bilingual Evaluation Understudy) is an automatic metric for evaluating the quality of machine-translated text by comparing it to one or more human-generated reference translations. Introduced by IBM researchers Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu in 2002, BLEU was designed to provide a fast, inexpensive, and language-independent alternative to human evaluation of machine translation output.
The core idea: a good translation should share many words and phrases (n-grams) with professional human translations. BLEU counts how many n-grams (sequences of 1, 2, 3, and 4 words) in the machine output also appear in the reference translations. Higher overlap = higher BLEU score = better translation quality. Crucially, BLEU includes a brevity penalty to prevent systems from gaming the metric by producing very short (high-precision) translations.
Purpose
Automatically evaluate machine translation quality by comparing n-gram overlap with human reference translations.
Score Range
0 to 100 (or 0 to 1). Higher is better. Scores of 30-40 are considered good for similar language pairs.
Key Components
Modified n-gram precision (unigrams through 4-grams) + Brevity Penalty (penalizes short translations).
How BLEU is Calculated
Step 1: Modified N-gram Precision
For each n (1 to 4), count the number of n-grams in the candidate translation that appear in any reference translation. But crucially, clip each n-gram's count to the maximum number of times it appears in any single reference. This prevents the system from repeating a correct word over and over to inflate the score.
Example: Candidate: "the the the the". Reference: "the cat sat on the mat". Without clipping, unigram precision would be 4/4 = 1.0. With clipping, unigram "the" appears at most twice in the reference, so precision = 2/4 = 0.5.
Step 2: Geometric Mean
Compute the modified precision for n=1,2,3,4 and take the weighted geometric mean: BLEU = exp(¼ Σ_{n=1}^{4} log p_n), where p_n is the modified precision for n-grams of length n.
Step 3: Brevity Penalty (BP)
If the candidate translation is shorter than the reference: BP = e^(1 − r/c), where r = reference length and c = candidate length. If the candidate is longer or equal: BP = 1. The final score is: BLEU = BP × exp(¼ Σ log p_n).
Why the Brevity Penalty Matters
Without the brevity penalty, a translation system could achieve high BLEU by outputting only the words it's very confident about and skipping the rest. A system that translates a 30-word sentence as just "The cat sat" would have high precision (all 3 words might be correct) but terrible recall (it missed 27 words). The brevity penalty corrects for this by penalizing outputs shorter than the reference. This is crucial because BLEU is fundamentally a precision metric—it doesn't directly measure recall. The brevity penalty is the mechanism that indirectly accounts for recall.
Interpreting BLEU Scores
BLEU scores are highly context-dependent. What counts as "good" varies by language pair, domain, and the number of reference translations:
| Score Range | Interpretation | Typical Context |
|---|---|---|
| 50-70 | Near human quality | Similar languages (EN-FR, EN-ES), professional translators |
| 30-50 | Good translation | News translation, similar languages |
| 20-30 | Adequate, understandable | Dissimilar languages (EN-JA, EN-ZH), technical domain |
| 10-20 | Poor, gist-level | Very dissimilar languages, low-resource |
| 0-10 | Unusable | Failed translation, wrong language |
Important caveat: BLEU is a corpus-level metric—it should be computed over at least 100+ sentences for reliable results. Single-sentence BLEU scores are highly noisy and not meaningful.
Limitations of BLEU
Despite its widespread use, BLEU has significant limitations that researchers have been addressing for years:
- Penalizes legitimate paraphrasing: A translation that uses different words but conveys the same meaning gets a low BLEU score. "The feline was seated on the mat" would score poorly against "The cat sat on the mat," even though it's a perfectly valid translation.
- No semantic understanding: BLEU only counts word overlap. It doesn't know that "purchase" and "buy" are synonyms, or that "not good" is the opposite of "good."
- Reference-dependent: The quality of BLEU evaluation depends entirely on the quality and quantity of reference translations. One reference can't capture all valid translation possibilities.
- No fluency or grammaticality: A translation can be grammatically terrible but have high BLEU if it uses the right words in roughly the right order. Conversely, a fluent, natural translation can have low BLEU if it uses different phrasing.
- Domain sensitivity: BLEU scores are not comparable across different domains, language pairs, or even different test sets within the same domain.
BLEU vs ROUGE
| Feature | BLEU | ROUGE |
|---|---|---|
| Primary metric | Precision (how much of candidate is in reference) | Recall (how much of reference is in candidate) |
| Designed for | Machine translation | Text summarization |
| N-gram range | 1-4 grams (equally weighted) | 1-2 grams typically, plus LCS variant |
| Brevity handling | Brevity penalty (penalizes shorter output) | No penalty (recall naturally penalizes shorter) |
| Score range | 0-100 | 0-1 (F1 typically reported) |
Frequently Asked Questions
BLEU uses up to 4-gram precision because it balances word choice (unigrams), phrase accuracy (bigrams), and fluency (trigrams, 4-grams). Using only unigrams would miss word order errors ("dog bites man" vs "man bites dog" look identical with unigrams). Using very long n-grams (5-grams, 6-grams) would be too strict—most n-grams wouldn't match even in good translations. Four-gram precision was found to correlate best with human judgments in the original BLEU paper. The geometric mean of 1-4 gram precisions provides a balanced measure of translation quality from word choice through phrase-level fluency.
SacreBLEU is a standardized implementation that produces consistent, reproducible BLEU scores. The original BLEU metric has many implementation details (tokenization method, smoothing technique, reference processing) that can cause the same system to get different scores. SacreBLEU standardizes all these details and includes a hash signature documenting exactly how the score was computed. It has become the de facto standard in NLP research, and using it ensures your scores are comparable to those reported in papers. Most top NLP conferences now expect SacreBLEU scores.
Modern neural metrics address BLEU's limitations: COMET (Crosslingual Optimized Metric for Evaluation of Translation) uses multilingual BERT to predict human quality judgments, achieving much higher correlation with human evaluation; BLEURT uses BERT fine-tuned on human ratings to score translations; BERTScore computes similarity using BERT embeddings rather than exact n-gram matching; METEOR was an earlier improvement that added synonym matching and stemming. These neural metrics are increasingly replacing BLEU in research, though BLEU remains widely used in industry due to its simplicity, speed, and interpretability.
BLEU works with a single reference translation, but accuracy improves significantly with more references. With 1 reference, BLEU captures only one valid way to translate. With 4 references, BLEU captures much more of the valid translation space. The original BLEU paper used 4 references. In practice, most benchmarks use 1 reference due to the cost of creating multiple references. When evaluating a system, using more references always produces more reliable BLEU scores. If you only have 1 reference, interpret BLEU scores cautiously—a low score might reflect the reference's style rather than the translation's quality.
Yes, BLEU is language-independent—it can be used for any language pair. However, BLEU scores are not comparable across different language pairs. A BLEU of 30 for English-French is not equivalent to a BLEU of 30 for English-Japanese. This is because: (1) Different languages have different amounts of word order variation; (2) Morphologically rich languages (Finnish, Turkish) have more word forms, making exact n-gram matches harder; (3) Writing systems differ (Chinese doesn't have word boundaries in the same way). Always compare BLEU scores within the same language pair and domain.
Continue your AI learning journey
Next, explore ROUGE Score—the recall-focused metric for evaluating text summarization and the complement to BLEU in NLP evaluation.
Read Next: ROUGE Score →