⌗ LECTURE 19 · MACHINE TRANSLATION METRICS · INTUITION FIRST, MATH STEP BY STEP
Who grades
the machine?
Suppose your neural network translates “The cat sat on the mat” into Urdu. How does a computer program — which does not speak Urdu or English — decide if the translation is a 100/100 masterpiece or 0/100 gibberish? This tutorial builds every major evaluation metric from scratch: first the human intuition, then the flaw in the naive idea, then the step-by-step logic, and finally the hand-computed English → Urdu arithmetic.
EXPERIENCE THE PARADOX FIRST · YOU ARE THE BILINGUAL JUDGE
HUMAN REFERENCE: بلی چٹائی پر بیٹھ گئیBoth candidates below translate “The cat sat on the mat” using the exact same five Urdu words. Click the one you think is a valid translation — and see how five famous metrics react:
Start at the very beginning: why is grading a translation so much harder than grading a multiple-choice quiz or a spam classifier?
02 THE PROBLEM — JUDGING AT SCALE
Why translation has no single right answer
In image classification, a photo of a cat has one true label: "cat". You grade the model with prediction == label. Try that on English-to-Urdu translation and your evaluation collapses immediately — because language allows many ways to express the exact same thought.
THE MENTAL MODEL · THE TEACHER’S GRADING DILEMMA
Imagine grading 10,000 Urdu essays in 2 seconds
Suppose you are training an English→Urdu Transformer. Every 30 minutes, your training loop produces a new checkpoint and translates 2,000 test sentences. You need to know: did Checkpoint #42 translate better than Checkpoint #41?
You cannot hire human translators every 30 minutes. So you give the computer a human answer key (the Reference translation). Now the computer must compare the Machine Candidate against the Human Reference. Immediately, it hits three walls of human language:
Different words, same meaning
For “beautiful country”, Reference says خوبصورت ملک (Persian root), while the machine writes حسین ملک (Arabic root). Both are 100% natural Urdu, yet character-by-character they share zero letters in the adjective!
Flexible Urdu phrase order
Because Urdu attaches postpositions like پر (“on”) directly to the noun (چٹائی پر), you can place چٹائی پر before or after بلی without changing who sat on what.
One moved particle inverts reality
Move پر just one slot to the right — بلی پر چٹائی بیٹھ گئی — and the sentence now means “The mat sat on the cat”! Five identical words, 100% opposite meaning.
Verb endings carry gender & aspect
In Urdu, بیٹھی (sat · fem), بیٹھ گئی (sat down · fem), and بیٹھ گیا (sat down · masc) share the root بیٹھ, but a naive computer sees three unrelated strings.
WHAT EVERY GOOD METRIC MUST MEASURE · THE TWO AXES
When human translators grade a sentence, they score two separate qualities: (1) Adequacy (Meaning Preservation) — did all the facts, actors, and actions from the English sentence arrive in Urdu without omission or addition? — and (2) Fluency (Grammatical Naturalness) — are the Urdu words in a grammatical order with proper gender/number agreement? Every metric we study below is a clever mathematical trick to approximate Adequacy + Fluency.
THE 4-GENERATION ROADMAP · HOW THE FIELD SOLVED THESE WALLS (2002 → TODAY)
Let’s try to invent our own metric from scratch. What is the simplest possible way to compare a machine’s Urdu sentence against a human’s Urdu reference?
03 THE DATA CONTRACT — SOURCE, REFERENCE, CANDIDATE
Naive Idea #1: Just count how many words match!
Before we look at BLEU, let’s understand the setup and test the simplest idea anyone thinks of first: “Break the candidate and the reference into individual words (1-grams), and count how many candidate words appear in the reference!”
THE LOGIC OF THE SETUP · WHY DON’T CLASSIC METRICS READ THE ENGLISH SOURCE?
Turning a bilingual problem into a monolingual comparison
Notice a surprising fact: BLEU, ROUGE, METEOR, TER, and chrF never look at the English source sentence (“The cat sat on the mat”)! Why? Because if an algorithm knew how to judge whether an Urdu sentence matches an English sentence directly, that algorithm would already be a full machine translation system.
Instead, we ask one or more human translators to translate the English sentence into Urdu once, producing our Reference Set (R = {r₁, r₂}). Now the metric only has to compare Urdu string vs Urdu string. And by providing multiple human references — such as r₁: بلی چٹائی پر بیٹھ گئی (compound verb) and r₂: بلی چٹائی پر بیٹھی (simple past) — the metric learns that either verb form is valid!
5/5 words match (100%).
5/5 words match (100%).
5/5 (100%)! (Trap #1: Order Blindness)
4/4 = 100%), even though the sentence is unfinished! (Trap #3: Short-Output Exploit)
4/5 = 80%).
INTERACTIVE LAB · CLICK EACH CANDIDATE TO SEE WHICH TOKENS MATCH THE REFERENCE POOL
We just saw three fatal traps in Naive Word Counting: (1) it can’t see word order, (2) it gets tricked by repeated words, and (3) it gets tricked by short fragments. Let’s fix Trap #1 first: how do we make a computer see word order?
04 THE RULER — N-GRAMS
How a sliding window checks word order
How can a metric tell that بلی چٹائی پر (“cat on the mat”) is right, while بلی پر چٹائی (“mat on the cat”) is backwards — without knowing a single rule of Urdu grammar? By sliding a window of n consecutive words across both sentences and comparing the chunks!
THE INTUITION · VELCRO STRIPS OF LENGTH n
Why n = 1 tests vocabulary, while n = 2, 3, 4 test grammar & order
Think of each n-gram as a Velcro strip holding n adjacent words together in exact order:
• 1-grams (Unigrams, n = 1) cut the sentence into single loose beads: [بلی], [چٹائی], [پر], [بیٹھ], [گئی]. Comparing 1-grams only asks: “Did the translator bring the right individual words?”
• 2-grams (Bigrams, n = 2) glue every pair of neighbors together: [بلی چٹائی], [چٹائی پر], [پر بیٹھ], [بیٹھ گئی]. Look at the second bigram: [چٹائی پر] (“mat on”). In the garbled Candidate C (بلی پر چٹائی بیٹھ گئی), the bigrams are [بلی پر], [پر چٹائی], [چٹائی بیٹھ], [بیٹھ گئی] — 3 out of 4 bigrams shattered immediately!
• 3-grams & 4-grams (n = 3, 4) test multi-word phrases and full clauses. A sentence of L words always produces L − n + 1 contiguous n-grams.
LAB 4A · BILINGUAL N-GRAM WINDOW SLICER
LAB 4B · SIDE-BY-SIDE CANDIDATE VS REFERENCE N-GRAM MATCHER
N-GRAM ORDER
CANDIDATE N-GRAMS (WHAT THE MODEL WROTE)
REFERENCE r₁ N-GRAMS (بلی چٹائی پر بیٹھ گئی)
N-grams solved Trap #1 (word order). Now what about Trap #2 — where a stuck decoder repeats “بلی بلی بلی” or “the the the” and claims 100% precision?
05 BLEU STEP 1 — CLIPPED PRECISION
The Voucher Rule: Never pay for more copies than the reference has
In a normal classification problem, precision is simply matches / total_predictions. But in text generation, a model can print the same common reference word ten times in a row. To stop repetition from inflating the score, BLEU introduces Modified (Clipped) n-gram Precision.
THE INTUITION · THE VOUCHER BUDGET ON THE TABLE
Each reference word is a single-use ticket
Imagine laying the human reference بلی چٹائی پر بیٹھ گئی on a table as five physical vouchers: one voucher for بلی, one for چٹائی, one for پر, one for بیٹھ, and one for گئی.
Now Candidate D walks in with 7 words: بلی بلی بلی چٹائی پر بیٹھ گئی (three copies of بلی):
• Copy #1 of بلی takes the single بلی voucher from the table → 1 match.
• Copy #2 of بلی looks at the table — the بلی voucher is already used up! → 0 credit (Clipped!).
• Copy #3 of بلی also gets 0 credit (Clipped!).
• The remaining 4 words (چٹائی پر بیٹھ گئی) each claim their voucher → 4 matches.
• Total clipped score: (1 + 4) / 7 = 5/7 ≈ 71.4% instead of the naive 7/7 = 100%!
Countcand(g)
For each unique n-gram g in the candidate translation, count how many times the machine produced it. In Candidate D, Countcand(بلی) = 3.
maxr ∈ Refs Countr(g)
Check each human reference separately and find the highest count of g in any single reference. (Never add counts across r₁ and r₂!). Here MaxRef(بلی) = 1.
Countclip(g) = min(Countcand, MaxRef)
Take the smaller of the two numbers: min(3, 1) = 1. Any repetitions above the reference ceiling are shaved off.
pn = ∑ Countclip(g) ÷ ∑ Countcand(g)
Sum the clipped credits over all unique n-grams (1 + 1 + 1 + 1 + 1 = 5) and divide by the total unclipped n-grams in the candidate (7) → p₁ = 5/7.
Now we have four clipped precision numbers: p₁ (words), p₂ (pairs), p₃ (triples), and p₄ (4-word phrases). How do we combine them into one final score — and stop the model from gaming precision by outputting a 1-word translation?
06 BLEU STEP 2 — THE FULL SCORE
Two design puzzles: Geometric Mean & Brevity Penalty
To finish designing BLEU, Papineni et al. (2002) had to solve two remaining exploits: (1) how to combine p₁, p₂, p₃, p₄ so a model cannot hide zero fluency behind high word overlap, and (2) how to punish a lazy model that outputs just two correct words (بلی چٹائی) and claims 100% precision.
DESIGN PUZZLE 1 · WHY GEOMETRIC MEAN INSTEAD OF ARITHMETIC MEAN?
The “AND” gate vs the “OR” gate
Look at Garbled Candidate C (بلی پر چٹائی بیٹھ گئی — “the mat sat on the cat”). Its four precisions are:
p₁ = 5/5 (1.00), p₂ = 1/4 (0.25), p₃ = 0/3 (0.00), p₄ = 0/2 (0.00).
If we used a regular Arithmetic Mean (1.00 + 0.25 + 0 + 0) / 4 = 0.3125 (31.3%), this completely inverted sentence would still get a respectable 31.3% score just for having the right vocabulary!
Instead, BLEU uses the Geometric Mean: multiply the four numbers together and take the 4th root, (p₁ · p₂ · p₃ · p₄)1/4. Multiplication acts like a logical AND gate: if any n-gram order drops to zero, the product drops to zero! Moreover, because precision decays roughly exponentially with n, taking the average in log-space (exp(¼ ∑ ln pₙ)) weights proportional improvements equally across all four orders.
DESIGN PUZZLE 2 · WHY A BREVITY PENALTY INSTEAD OF RECALL?
The 1-word cowardice exploit — and why recall breaks with multiple references
Suppose a lazy model translates “The cat sat on the mat” as just بلی چٹائی پر بیٹھ (4 tokens, dropping گئی). Its precisions against r₁ are p₁=4/4, p₂=3/3, p₃=2/2, p₄=1/1 — all 100%! Precision only checks whether the words you wrote are right; it never checks what you left out.
Why didn’t BLEU just use standard Recall? Because when you have multiple references (r₁: بیٹھ گئی and r₂: بیٹھی), a good translation should only use one phrasing, not both! Standard recall against the union of references would punish every normal sentence for not including both بیٹھ گئی and بیٹھی. So BLEU replaces word recall with a length check: if candidate length c is at least the closest reference length r, BP = 1; if c < r, multiply by e1 − r/c!
INTERACTIVE INTUITION LAB · THE BREVITY PENALTY CURVE (BP = e1 − r/c WHEN c ≤ r)
p₁ = (1 + 1 + 1 + 1 + 1) / 7 = 5/7 ≈ 0.7143.
p₂ = 4/6 ≈ 0.6667.
p₃ = 3/5 = 0.6000. Of 4 four-grams, 2 match → p₄ = 2/4 = 0.5000.
GM = (5/7 × 4/6 × 3/5 × 2/4)1/4 = (0.14286)0.25 = 0.6148.
c = 7 is greater than reference length r = 5, so BP = 1.0000. Final score: BLEU-4 = 1.0000 × 0.6148 = 0.6148 (61.5%)!
| Candidate (Urdu) | c / r | p₁ (1-gram) | p₂ (2-gram) | p₃ (3-gram) | p₄ (4-gram) | GM | BP | BLEU-4 (raw) |
|---|---|---|---|---|---|---|---|---|
| A · بلی چٹائی پر بیٹھ گئی (Perfect) | 5 / 5 | 5/5 (1.00) | 4/4 (1.00) | 3/3 (1.00) | 2/2 (1.00) | 1.0000 | 1.0000 | 1.0000 (100%) |
| B · چٹائی پر بلی بیٹھ گئی (Reorder) | 5 / 5 | 5/5 (1.00) | 2/4 (0.50) | 0/3 (0.00) | 0/2 (0.00) | 0.0000 | 1.0000 | 0.0000 (0%) |
| C · بلی پر چٹائی بیٹھ گئی (Garbled) | 5 / 5 | 5/5 (1.00) | 1/4 (0.25) | 0/3 (0.00) | 0/2 (0.00) | 0.0000 | 1.0000 | 0.0000 (0%) |
| D · بلی بلی بلی چٹائی پر بیٹھ گئی (Babble) | 7 / 5 | 5/7 (0.71) | 4/6 (0.67) | 3/5 (0.60) | 2/4 (0.50) | 0.6148 | 1.0000 | 0.6148 (61.5%) |
| E · بلی چٹائی پر بیٹھ (Truncated, vs r₁ only) | 4 / 5 | 4/4 (1.00) | 3/3 (1.00) | 2/2 (1.00) | 1/1 (1.00) | 1.0000 | e−1/4 = 0.7788 | 0.7788 (77.9%) |
| G · بلی چٹائی پر بیٹھ گیا (Gender Slip) | 5 / 5 | 4/5 (0.80) | 3/4 (0.75) | 2/3 (0.67) | 1/2 (0.50) | 0.6687 | 1.0000 | 0.6687 (66.9%) |
Now test the complete BLEU machine interactively — and see how “smoothing” rescues single sentences when p₃ or p₄ hits zero.
07 THE BLEU MACHINE
Drive the BLEU-N calculator and test Add-1 Smoothing
Why did Candidate B (چٹائی پر بلی بیٹھ گئی) get 0% in the table above? Because on a single short sentence, one legitimate phrase move left zero 3-grams (p₃ = 0/3), and multiplying by zero destroys the Geometric Mean! In practice, BLEU is designed to be computed over a whole corpus of 1,000+ sentences pooled together. When engineers must compute BLEU on a single sentence, they turn on Smoothing (adding +1 to the numerator and denominator for n ≥ 1) so a zero in p₃ or p₄ doesn’t erase p₁ and p₂.
1. PICK CANDIDATE
2. REFERENCE POOL
CLIPPED PRECISIONS p₁ … pN
Even with smoothing, BLEU has deep structural blind spots — especially when translating from English into Urdu. Let’s examine all four blind spots side by side.
08 WHAT BLEU MISSES
Why exact word n-grams struggle on English → Urdu
Every new metric invented after 2002 was born to fix a specific failure mode of BLEU. In Urdu, four linguistic realities expose BLEU’s limits immediately. Read the logic of each failure below, then click the four interactive diagnostic cards:
Urdu allows free phrase fronting
In Urdu, case markers (نے، کو، پر، میں) mark who did what to whom. Moving چٹائی پر to the front of the sentence is 100% grammatical, yet it breaks the 3-gram bridge across the middle of the sentence — causing BLEU to punish valid Urdu syntax.
Zero credit for Persian/Arabic synonyms
Urdu draws vocabulary from Prakrit/Sanskrit, Persian, and Arabic. Replacing خوبصورت with حسین (“beautiful”) not only loses the unigram credit for that word, it also breaks every 2-gram, 3-gram, and 4-gram that touches that position — cutting BLEU-4 in half (48.9%)!
All-or-nothing word boundaries
To BLEU, گئی (went · fem) and گیا (went · masc) are just as different as گئی and ہوائی جہاز (“airplane”). Worse, because گیا sits at the very end of the sentence, the first 4 words (بلی چٹائی پر بیٹھ) form a matching 4-gram — so the grammar error scores 66.9% while valid reordering scores 0%!
Function words = Content words
BLEU gives the exact same weight to matching a filler auxiliary (ہے) as it does to matching the core verb or negation (نہیں). Dropping “not” barely dents BLEU, even though it reverses the truth of the sentence!
CLICK EACH FAILURE MODE TO COMPARE HUMAN JUDGMENT VS BLEU, chrF, AND METEOR
How do we fix the Reordering Cliff so that moving a phrase doesn’t shatter our score to zero? Replace rigid contiguous n-grams with the Longest Common Subsequence (ROUGE-L).
09 RECALL & ROUGE
ROUGE-L: Follow the in-order stream, skip the detours
While BLEU asks a Precision question (“How much of the candidate is in the reference?”), ROUGE (Lin, 2004) starts from Recall (“How much of the reference survived in the candidate?”) and combines both into an F₁ score. Most importantly, ROUGE-L throws away rigid contiguous n-gram windows and measures the Longest Common Subsequence (LCS).
THE INTUITION · SKIPPING STONES ACROSS A RIVER
What is a “Subsequence” vs a “Substring (n-gram)”?
• An n-gram (substring) requires words to stand shoulder-to-shoulder with zero gaps. In Reference 1 (بلی چٹائی پر بیٹھ گئی), the words چٹائی پر and بیٹھ گئی touch. When Candidate B (چٹائی پر بلی بیٹھ گئی) slips بلی in between them, the 4-gram is broken.
• A Subsequence (LCS) only requires words to appear in the same relative order from start to finish — you are allowed to skip over inserted words! Walk through Candidate B (چٹائی پر بلی بیٹھ گئی) against Reference 1 (بلی چٹائی پر بیٹھ گئی):
1. Match چٹائی (token #1) →
2. Match پر (token #2) →
3. Skip over the relocated بلی! →
4. Match بیٹھ (token #4) →
5. Match گئی (token #5).
That is a 4-word in-order chain (چٹائی پر … بیٹھ گئی, LCS = 4)! So Candidate B earns ROUGE-L = 4/5 = 0.80 (80%) instead of BLEU-4’s 0%!
Compare every (i, j) word pair
Make a grid with Candidate words down the rows (i = 1..c) and Reference words across the columns (j = 1..r). If cand[i] == ref[j], add 1 to the top-left diagonal: dp[i][j] = dp[i−1][j−1] + 1. Otherwise, carry forward the best score so far: max(dp[i−1][j], dp[i][j−1]).
LCS(c, r) = dp[c][r]
The bottom-right cell of the table holds the length of the longest in-order chain shared by both sentences. For Candidate B, LCS = 4.
PLCS = LCS / c · RLCS = LCS / r
Divide LCS by candidate length c for Precision (4/5 = 0.80), and by reference length r for Recall (4/5 = 0.80).
ROUGE-L = 2 PLCS RLCS / (PLCS + RLCS)
Combine Precision and Recall into a single balanced F₁ score: 2(0.8)(0.8) / (0.8 + 0.8) = 0.80.
INTERACTIVE LCS DYNAMIC PROGRAMMING TABLE · PICK A CANDIDATE
LCS DYNAMIC PROGRAMMING GRID (rows = candidate · cols = reference r₁ · teal = LCS path)
ROUGE-1 · ROUGE-2 · ROUGE-L (F₁ SCORES)
ROUGE-L solved word-order detours, but it still fails when a translator uses a synonym like حسین instead of خوبصورت. How does METEOR solve synonyms and word order at the same time?
10 METEOR — BEYOND EXACT MATCH
Multi-stage word alignment plus a chunk fragmentation penalty
Designed by Banerjee & Lavie (2005), METEOR throws away blind n-gram slicing altogether. Instead, it grades a translation the way a human teacher grades a sentence: (1) draw 1-to-1 alignment lines between candidate words and reference words using exact spelling, word stems, and synonyms, then (2) check how many contiguous “chunks” those aligned words form.
THE INTUITION · HOW “CHUNKS” MEASURE WORD ORDER WITHOUT N-GRAMS
Imagine marching in formation with a partner
Look at our Urdu synonym example for “He lives in a beautiful country”:
• Reference: وہ ایک خوبصورت ملک میں رہتا ہے (7 words)
• Candidate: وہ ایک حسین ملک میں رہتا ہے (7 words — replaces خوبصورت with its synonym حسین).
What happens if we only allow Exact Match (Stage 1)?
Words 1–2 (وہ ایک) align, Word 3 (حسین) is left unaligned in the middle like a roadblock, and Words 4–7 (ملک میں رہتا ہے) align after the gap. Only m = 6 words matched, and because of the gap in the middle, the aligned words are split into 2 separate contiguous chunks!
What happens when we turn on Synonym Matching (Stage 2)?
Now حسین ↔ خوبصورت aligns at position 3! All m = 7 words are aligned, and as you walk from word 1 to word 7 in the candidate, their partners in the reference are 1, 2, 3, 4, 5, 6, 7 — 1 single unbroken chunk (chunks = 1)!
Exact → Stem → Synonym
Match each candidate word to at most one reference word: first identical strings, then shared morphological roots, then dictionary synonyms (حسین ↔ خوبصورت). Count total matches m.
Fmean = 10 P R / (R + 9 P)
Compute P = m/c and R = m/r. Why 9P in the denominator? Human correlation studies showed that Recall (not dropping meaning) matters 9× more than Precision!
How many adjacent groups?
Two aligned candidate words belong to the same chunk if they are adjacent in the candidate AND their aligned partners are adjacent in the same order in the reference. Fewer chunks = better word order.
Pen = 0.5 · (chunks / m)³
Multiply Fmean × (1 − Pen). With 1 unbroken chunk of 7 words, Pen = 0.5 × (1/7)³ = 0.0015, giving METEOR = 0.9985! Even if every word is scrambled (chunks = m), the penalty caps at 50% — never 0%.
INTERACTIVE METEOR ALIGNER · TOGGLE STAGE 1 VS STAGE 2
Watch how matching حسین ↔ خوبصورت heals the gap and merges the chunks into 1:
CANDIDATE (c) — colored dots show which reference word each token aligned to
REFERENCE (r)
◈ DEEP DIVE AVAILABLE — Want METEOR completely: all four matching modules (exact → stem → synonym → paraphrase), precision and recall by hand, the α-slider behind Fmean = 10PR/(R+9P), a scramble-it-yourself chunk lab, γ/β penalty curves, the BLEU face-off, the negation blind spot — every number machine-verified and cross-checked against a real NLTK run?
Open the METEOR deep dive →METEOR is smart, but it requires a synonym dictionary and stemmer for every target language — which many low-resource languages lack. How can we handle reordering and Urdu morphology with ZERO dictionaries? Meet TER and chrF.
11 TER & chrF — THE ENGINEER’S METRICS
Two dictionary-free workhorses: Cut-and-Paste Edits & Character Slices
When MT engineers evaluate English → Urdu systems in practice (such as with sacrebleu), they pair BLEU with two dictionary-free metrics that solve reordering and morphology directly from the raw text: TER and chrF.
INTUITION 11A · HOW TER (TRANSLATION EDIT RATE) WORKS
The Human Post-Editor’s Keyboard: Insert, Delete, Substitute, and SHIFT
Imagine a translation agency where a human Urdu editor fixes the machine’s draft. How much work did the machine save them? Standard Word Error Rate (WER / Levenshtein distance) counts single-word Insertions, Deletions, and Substitutions. But look at Reordered Candidate B (چٹائی پر بلی بیٹھ گئی vs بلی چٹائی پر بیٹھ گئی): to WER, بلی was deleted at slot 1 (+1 edit) and re-inserted at slot 3 (+1 edit) = 2 edits (40% error)!
Snover et al. (2006) realized humans don’t retype moved phrases — they Cut & Paste (SHIFT)! TER adds a 4th edit operation: moving any contiguous phrase to a new spot counts as just 1 Shift edit, cutting Candidate B’s error rate from 40% down to 1/5 = 20%. (Because TER measures error, 0% is perfect and lower is better.)
INTUITION 11B · HOW chrF (CHARACTER N-GRAM F-SCORE) WORKS
Zooming inside the word so Urdu suffixes don’t erase the stem
Why did BLEU fail on Candidate G (بلی چٹائی پر بیٹھ گیا vs بیٹھ گئی)? Because word tokenization treats گیا and گئی as atomic bricks: if 1 letter changes, 100% of the word token fails. In Urdu, nouns and verbs constantly change suffixes (چٹائی / چٹائیاں / چٹائیوں, بیٹھی / بیٹھا / بیٹھیں).
Popović (2015) asked: why slice by words at all? chrF slices words into character 1-grams through 6-grams! Even when بیٹھ گئی becomes بیٹھ گیا, the character n-grams inside بیٹھ and the shared prefix letter گ still match! It computes Precision and Recall at each character order n = 1..6 and combines them with recall weighted twice as much as precision (β = 2).
LAB 11A · TER VS WER POST-EDITING SIMULATOR
PICK CANDIDATE TO SEE MINIMUM EDITS
LAB 11B · chrF CHARACTER 1..6-GRAM INSPECTOR
◈ DEEP DIVE AVAILABLE — Want TER & chrF completely: the four-edit toolbox with phrase shifts priced at one edit, the WER-vs-TER face-off (16.7% vs 100% on the same sentence), a repair-ledger calculator including the unbounded 133% case, shared/exclusive bigram labs, the β-dial behind chrF₂, the کتاب / کتابیں Urdu walkthrough — every number machine-verified and cross-checked against a real sacrebleu 2.6.0 run?
Open the TER & chrF deep dive →Wait: what if a bad translation writes بدصورت (“ugly”) instead of خوبصورت (“beautiful”)? Because both words end in صورت, character n-grams (chrF) give “ugly” high overlap with “beautiful”! To truly judge meaning, the metric must understand what words mean.
12 LEARNED METRICS — BERTSCORE & COMET
Grade meaning in neural vector space — and read the Source
Consider the ultimate trap for surface string metrics in Urdu: خوبصورت (khoobsurat = “beautiful”) vs بدصورت (badsurat = “ugly”) vs حسین (haseen = “beautiful”). In character spelling, “beautiful” (خوبصورت) shares 4 letters (صورت) with “ugly” (بدصورت) and 0 letters with its true synonym (حسین)! String metrics are blind to meaning. Neural metrics fix this by replacing character counting with pretrained Transformer embeddings.
Embed Candidate & Reference
Pass the Urdu candidate and Urdu reference through a pretrained multilingual Transformer (like XLM-RoBERTa). Every token becomes a rich contextual vector eᵢ where synonyms (حسین and خوبصورت) point in almost the exact same direction!
Measure semantic similarity
Compute the cosine similarity cos(ex, ey) between every candidate token x and every reference token y. Even though حسین and خوبصورت share zero letters, their cosine similarity is 0.987!
PBERT, RBERT, and FBERT
For Precision, each candidate token greedily picks its single highest-cosine partner in the reference and averages the scores. For Recall, each reference token picks its closest candidate token. Then take their harmonic mean F₁.
Read (Source + Candidate + Reference)
Why does BERTScore still give 1.000 to Garbled Candidate C (بلی پر چٹائی بیٹھ گئی)? Because greedy max-cosine matching is still a bag-of-vectors check! COMET (Rei et al., 2020) feeds the English Source, Urdu Candidate, and Urdu Reference together into XLM-R with a regression head trained directly on human Direct Assessment & MQM ratings.
INTERACTIVE BERTSCORE GREEDY COSINE MATCHER
CANDIDATE TOKENS (NUMBER = MAX COSINE SIMILARITY TO ANY REFERENCE TOKEN)
REFERENCE TOKENS (COLORED DOT = MATCHED PARTNER)
HOW WELL DO THESE METRICS AGREE WITH HUMANS? (WMT19 KENDALL’S τ)
Segment-level correlation with human judgments (higher τ = closer agreement with human translators, Rei et al. 2020):
THE ARCHITECTURAL DIFFERENCE · WHY COMET BEATS BERTSCORE
BERTScore only compares Candidate ↔ Reference via an untrained max-cosine pooling rule. COMET encodes Source (English) + Candidate (Urdu) + Reference (Urdu) jointly and feeds the combined difference vectors [hcand; href; |hcand − href|; |hcand − hsrc|] into a neural network trained on tens of thousands of human translator scores — so it learns that moving پر inverts the English source meaning.
◈ DEEP DIVE AVAILABLE — Want BERTScore completely: contextual embeddings with the live “bank” context demo, cosine similarity derived identifier-by-identifier (8/√65 ≈ 0.992 by hand), the greedy-matching heatmap with a precision/recall view toggle, the F1 playground with its imbalance tax, the caution ledger (negation, role flips, five vs fifteen) — and a real bert-score run where the negation sentence out-scores the synonym sentence?
Open the BERTScore deep dive →◈ DEEP DIVE AVAILABLE — Want COMET completely: the judgment gap on the dropped-“not” doctor example scored live by overlap engines, the three inputs as a courtroom, q̂ = fθ(s, r, h) identifier by identifier, the training loop with a toy judge you can interrogate, score interpretation (why 0.91 ≠ 91% correct), reference-free COMETKiwi, the BLEU→COMET ladder with the WMT19 τ table — and the walkthrough’s exact script run on a real wmt22-comet-da checkpoint?
Open the COMET deep dive →Where did the human scores that trained COMET come from? Let’s step into the shoes of a professional MQM human auditor.
13 HUMANS IN THE LOOP — DA & MQM
How humans grade: Direct Assessment & MQM Error Spans
Automatic metrics never replace humans completely — they are calibrated against humans. Today’s MT benchmarks use two human protocols: Direct Assessment (DA), where raters drag a 0–100 slider for adequacy, and MQM (Multidimensional Quality Metrics), where expert linguists click the exact offending words in the translation and tag their error severity.
THE LOGIC OF MQM · WHY HIGHLIGHTING SPANS BEATS GUESSING A NUMBER
Start at 100, and deduct fixed points for every tagged error span
If you ask two human translators to rate بلی پر چٹائی بیٹھ گئی on a 0–100 slider, Rater A might give it 25 while Rater B gives it 60 because the words look familiar. That subjective noise hurts evaluation.
MQM fixes this by turning grading into a concrete audit checklist: every translation starts at 100 points. Whenever the translator spots a mistake, they highlight the exact word span and tag its severity:
• Minor Error (−1 point): A slight stylistic awkwardness or punctuation slip that does not confuse the reader.
• Major Error (−5 points): A grammar error (like masculine گیا with feminine بلی) or a role shift (پر attached to the wrong noun) that changes or obscures meaning.
• Critical Error (−25 points): A catastrophic mistranslation that could cause real-world harm (e.g., dropping نہیں “not” in a medical dosage instruction).
INTERACTIVE MQM AUDITOR · GRADE GARBLED CANDIDATE C (بلی پر چٹائی بیٹھ گئی)
Click any Urdu word below to cycle its error severity: None (0) → Minor (−1) → Major (−5) → Critical (−25).
| Metric | Core Mechanism & Logic | Uses Source? | Handles Synonyms? | Handles Urdu Morphology? | Sees Word Order? | When to Use It in Practice |
|---|---|---|---|---|---|---|
| BLEU-4 (SacreBLEU) | Clipped word 1..4-gram precision × Brevity Penalty | No (Ref only) | No (exact token) | Poor (1 letter change = 0 match) | Yes (via 2..4-grams) | Fast corpus-level baseline & comparing against published papers |
| ROUGE-L | Longest Common Subsequence (in-order skipping chain) | No (Ref only) | No | Poor | Soft (in-order subsequence) | Summarization & checking if reference sequence survived |
| METEOR | Exact→Stem→Synonym 1-to-1 alignment × Chunk penalty | No (Ref only) | Yes (lexicon) | Partial (with stemmer) | Yes (chunk count) | Synonym-aware evaluation when stemmers/lexicons exist |
| TER | Min word edits (Ins, Del, Sub + whole-phrase Shifts) ÷ |r| | No (Ref only) | No | Poor | Yes (1-cost phrase shifts) | Estimating human post-editing effort in translation workflows |
| chrF / chrF2 | Character 1..6-gram F-score (recall weighted β = 2) | No (Ref only) | No | Excellent (stem overlap) | Local (within words) | Best fast string metric for Urdu & morphologically rich languages |
| BERTScore | Greedy max-cosine match over XLM-R token embeddings | No (Ref only) | Yes (vector cos) | Good (subwords) | Weak (bag of vectors) | Semantic paraphrase overlap without a supervised metric model |
| COMET | XLM-R encoder on (Src, Cand, Ref) trained on human DA/MQM | Yes (Src+MT+Ref) | Yes | Excellent (XLM-R) | Yes (role-aware) | Gold-standard neural evaluation before final release |
14 CODE LAB
Run every metric in Python
Three ready-to-run scripts on our exact English → Urdu examples: (1) a 25-line pure-Python implementation of BLEU-4 from scratch so you can see every line of the algorithm, (2) the industry-standard sacrebleu library computing BLEU, chrF2, and TER, and (3) neural evaluation with unbabel-comet and bert_score.
15 MASTERY CHECK
Eight questions — grade the metrics
Test your intuition on clipped precision, the brevity penalty, the reordering cliff, character n-grams in Urdu, and why COMET sees what BERTScore misses.