CAS logo CAS NLP LABInteractive Tutorial Series ← Series

◈ DEEP DIVE — METEOR · match → measure → penalize · every number verified

Judge the translation
like a teacher

A good machine translation should not be judged only by whether its words exactly match the reference — it should also earn credit when different words carry the same meaning. METEOR — the Metric for Evaluation of Translation with Explicit ORdering (Banerjee & Lavie, 2005) — grades the way a human teacher does: it aligns words flexibly (stems, synonyms, paraphrases), weighs what was captured (recall-weighted F), and prices word order (a fragmentation penalty). This page builds all of it, number by number.

STAGE 2 · MEASURE (§05)

\[ F_{\text{mean}} = \frac{10\,P\,R}{R + 9P} \]

STAGE 3 · PENALIZE (§07)

\[ \text{Pen} = 0.5\left(\frac{ch}{m}\right)^{3} \]

THE SCORE (§08)

\[ \text{METEOR} = F_{\text{mean}} \times (1 - \text{Pen}) \]

THE WHOLE STORY IN ONE SWITCH — Reference: “The boy is playing football.”

Candidate: “The child is playing soccer.” — a faithful translation that shares only 3 of 5 words with the reference.

CANDIDATE — dots mark aligned words

REFERENCE

FULL MODULES · stem + synonyms ON
0.0000

4MATCHING MODULESexact → stem → synonym → paraphrase
11LIVE LABSevery stage is touchable
30+VERIFIED NUMBERSNode + real NLTK cross-check
8QUIZ CHECKmastery before you leave
~40MINUTEScompanion to Lecture 16 §10

Lecture 16 §10 showed METEOR rescuing one Urdu synonym pair. But how does the score actually come together — and why 9, why cubed, why 0.5? Let's build every piece from the beginning.

01 WHY METEOR

The exact-match trap

Start as the teacher. Here is one reference sentence and four machine translations of it. Three of the four deserve real credit — but an exact-spelling metric sees only spelling. Click through what each judge says, then watch METEOR disagree with the exact-match column in exactly the places humans do.

REFERENCE: “THE BOY IS PLAYING FOOTBALL.” EXACT-ONLY METEOR FULL METEOR

QUESTION 1 · PRECISION

How much of the output is supported?

Of all the words the machine produced, how many match the reference? This is P = m/c — the machine’s precision (§03).

QUESTION 2 · RECALL

How much of the meaning was captured?

Of all the words in the human reference, how many did the machine cover? This is R = m/r — recall (§04). METEOR weights it 9× heavier than precision (§05).

QUESTION 3 · WORD ORDER

Are matched words in the same order?

Are the matched words arranged in coherent runs, or scattered? METEOR counts chunks of contiguous matches and penalizes fragmentation (§06–07) — the “Explicit ORdering” in its name.

THE CONTRACT

Like every classic metric — no source sentence

METEOR never reads the English input. It compares the candidate translation against a human reference of the same language (Lecture 16 §02) — a monolingual string comparison standing in for bilingual judgment.

WHERE IT CAME FROM · BANERJEE & LAVIE, ACL WORKSHOP 2005

Born from BLEU’s blind spots

By 2005, BLEU (2002) was the industry standard — but it was built for whole corpora, and at the single-sentence level it correlated poorly with human judgment: it has no recall, no stemming, no synonyms, and its n-gram windows shatter on small differences. Satanjeev Banerjee & Alon Lavie (CMU) designed METEOR to track human adequacy scores at sentence level — and reported exactly that: substantially improved correlation with human judgments. Two decades later it remains the classic “second opinion” to BLEU, from shared-task leaderboards to Hugging Face evaluate.

Everything begins with the alignment. Before any formula fires, METEOR must decide which candidate word “answers” which reference word — and it is allowed to look beyond spelling.

02 STAGE 1 · THE MATCHING MODULES

Four ways two words can “be the same”

METEOR aligns each candidate word to at most one reference word, using a strict cascade of modules — cheaper modules first, dictionaries only when spelling fails. A pair aligns at the first stage that accepts it; a pair no module accepts stays unaligned.

StageModuleReference wordCandidate wordWhy they match2005-era resource
1ExactplayplayIdentical strings — no dictionary needed—
2StemplayingplayedBoth reduce to the stem playPorter stemmer
3SynonymbiglargeSame meaning, different spellingWordNet 2.0
4ParaphrasepurchasebuyEquivalent in context — not strict synonyms≈2M learned phrase pairs

MODULE CHAIN EXPLORER · CLICK A PAIR, WATCH THE CASCADE

RULE 1 · ONE-TO-ONE

Every word aligns at most once

A candidate word claims at most one reference word, and vice versa. If two candidate words both want the, only one gets it — the alignment is a set of pairs, not a free-for-all.

RULE 2 · STAGE ORDER

Exact eats first

Modules fire in cascade order. Exact matches are consumed before the stemmer runs; stems before synonyms. This keeps the score deterministic — and cheap: dictionaries open only for the leftovers.

RULE 3 · THE HONEST TRAP

Related ≠ matched

playing/played share a stem — but play and game share nothing a module can see. They are semantically related, and that is a different relationship. Unless a dictionary lists them, they stay unaligned. Click the trap pair above to watch every module pass.

RULE 4 · RESOURCES DECIDE

Your dictionary is your ceiling

Lecture 16 §10 aligned the Urdu synonyms خوبصورت ↔ حسین (“beautiful”) — because we gave the page that dictionary. Real METEOR scores only as well as its resources: many low-resource languages (Urdu included) lack the stemmers and synonym nets English enjoys.

Alignment done — the count of matched words is written m. Now the first number: how much of the machine’s output is defensible?

03 STAGE 2a · PRECISION

Of the words the machine wrote, how many hold up?

Our worked example for the next four sections — the sentence pair every number on this page descends from:

REFERENCE (r = 7 words)

The | cat | is | sitting | on | the | mat.

CANDIDATE (c = 6 words)

The | cat | sat | on | the | mat.

The candidate compresses is sitting into sat. Assume the stem module matches sat ↔ sitting (a lemmatizer makes this pair; see the honesty note in §08 for how real libraries do it). Then every candidate word finds a partner: m = 6.

CANDIDATE — colored dot = the reference word it claimed

REFERENCE — grey dot = never captured

THE FORMULA · PRECISION

\[ P \;=\; \frac{\text{matched words}}{\text{candidate words}} \;=\; \frac{m}{c} \;=\; \frac{6}{6} \;=\; 1.0 \]

Precision = 100%. Every single word the machine produced is defensible against the reference. But notice what precision cannot say: it grades only what was written — never what is missing. Switch the lab above to the two-word coward and watch precision stay perfect while the translation collapses.

A machine that writes two safe words and stops also earns P = 1.0. The second question closes that loophole.

04 STAGE 2b · RECALL

How much of the reference did the machine capture?

Recall flips the denominator: now the reference sets the standard, and the candidate is audited against it.

CANDIDATE

REFERENCE — count the grey dots

THE FORMULA · RECALL

\[ R \;=\; \frac{\text{matched words}}{\text{reference words}} \;=\; \frac{m}{r} \;=\; \frac{6}{7} \;\approx\; 0.857 \]

Recall ≈ 85.7%. Why not 100%? The reference word is never found a partner — the candidate folded it into sat. A teacher would say the same: good answer — one word of content missing.

THE TEACHER METAPHOR · PRECISION

How much of the student’s answer is supported?

Scan the student’s page line by line: is every claim backed by the reference? That’s precision — it audits the output.

THE TEACHER METAPHOR · RECALL

How much of the material did the student cover?

Scan the answer key: which points never appeared? That’s recall — it audits the coverage. METEOR’s classic weighting (§05) says coverage matters 9× more.

Two numbers now — P = 1.0 and R = 0.857 — and a design decision: how do you merge them without letting either hide the other?

05 STAGE 2c · F-MEAN

A harmonic mean that trusts recall

An arithmetic mean is a cheater’s friend: P = 1.0 with R = 0.286 (our two-word coward) still averages to 0.64 — a passing grade for translating two words out of seven. A harmonic mean punishes imbalance, and METEOR’s classic version tilts it toward recall.

THE FORMULA · CLASSIC F-MEAN (BANERJEE & LAVIE 2005)

\[ F_{\text{mean}} \;=\; \frac{10\,P\,R}{R + 9P} \qquad\text{equivalently}\qquad F_{\text{mean}} \;=\; \frac{P\,R}{\alpha P + (1-\alpha)R},\;\; \alpha = 0.9 \]

The 10 and the 9 are not decoration: in the parameterized form, α = 0.9 makes the denominator 0.9·P + 0.1·R — recall enters with 9× the weight of precision. Why? Because human-correlation studies in the 2005 paper found that dropped content damages perceived quality far more than extra or substituted words. Recall is where meaning lives.

SUBSTITUTING OUR NUMBERS · P = 1, R = 6/7

CLASSIC 2005 · α = 0.9
\[ F_{\text{mean}} = \frac{10 \cdot 1 \cdot \tfrac{6}{7}}{\tfrac{6}{7} + 9 \cdot 1} = \frac{\tfrac{60}{7}}{\tfrac{6}{7} + \tfrac{63}{7}} = \frac{\tfrac{60}{7}}{\tfrac{69}{7}} = \frac{60}{69} \approx \mathbf{0.8696} \]

Lower than precision alone (1.0) — because the candidate never captured is. The weighting also means a tiny safe fragment cannot farm a big score: recall drags it down 9× harder.

α = 0.5 — vanilla F1

Weights equal: the plain harmonic mean used across NLP. Our pair scores 12/13 ≈ 0.923 — already recall-friendlier than the arithmetic 0.929.

α = 0.9 — the classic

The 2005 default: 60/69 ≈ 0.870 for our pair, and only 0.308 for the two-word coward (P = 1, R = 2/7) — versus 0.64 arithmetic. The loophole closes.

Why not α = 1.0?

That is pure recall — it would reward a candidate that copies the reference plus invents nothing… but also one that pads with matches and ignores precision entirely. 0.9 keeps a 10% voice for precision.

F-mean is blind to one last thing: order. “The cat sat on the mat” and “The mat sat the cat on” contain the same six words — identical P, identical R. The third stage prices the difference.

06 STAGE 3a · CHUNKS

Word order, made countable

A chunk is a maximal run of matched words that are adjacent in the candidate and whose partners are adjacent, in the same order, in the reference. Walk the candidate left to right; every place the “one step behind its predecessor” chain breaks starts a new chunk. Fewer chunks = a more faithful word order.

THE INTUITION · MARCHING IN FORMATION

Imagine marching in formation with a partner

Each candidate word holds hands with its aligned reference partner. As long as the column advances — candidate word i+1 paired with reference word j+1, exactly one step behind — the formation is one unbroken chunk. The moment someone’s partner is two steps ahead, or behind, or out of line, the chain snaps: new chunk. A perfect translation marches as one; a scrambled one disintegrates into singletons.

SCRAMBLE LAB · CLICK ANY TWO WORDS TO SWAP THEM

Reference: The cat sat on the mat. — every word matches; only the ORDER changes. Each colored underline is one chunk.

CANDIDATE — swap words and watch the chunks re-form

REFERENCE (fixed)

Candidate A — “The cat sat on the mat.”

Identical: one unbroken chunk of 6. The formation marches as one.

Candidate C — “The cat on the mat sat.”

sat exiled to the end: three chunks (2 + 3 + 1). Meaning survives; order limps.

Candidate B — “The mat sat the cat on.”

Full scramble: six chunks of one word each. Every word matched — and order destroyed.

Now the chunk count becomes a number the score can feel — softly, with a ceiling.

07 STAGE 3b · THE FRAGMENTATION PENALTY

Soft punishment with a hard cap

Chunk count alone can’t enter the score — it must be normalized (6 chunks of 6 words is chaos; 6 chunks of 60 is nothing) and scaled. The 2005 paper’s choice:

THE FORMULA · FRAGMENTATION PENALTY

\[ \text{Pen} \;=\; \gamma \left( \frac{ch}{m} \right)^{\beta} \qquad\xrightarrow{\;\text{classic}\;}\qquad \text{Pen} \;=\; 0.5 \left( \frac{ch}{m} \right)^{3} \]

ch = number of matched chunks, m = number of matched words. γ = 0.5 sets the maximum penalty — a fully scrambled translation loses at most half its score, never all of it (matched meaning keeps credit). β = 3 makes the curve forgiving at the start: a little fragmentation is nearly free, and the pain grows steeply only as ch/m approaches 1.

PENALTY PLAYGROUND · DRAG γ AND β, WATCH THE CURVE BEND

COMPUTED · ONE CHUNK OF SIX

Pen = 0.5·(1/6)³ ≈ 0.0023

Candidate A, the identical sentence. One chunk means the cube of a small fraction — the penalty is nearly invisible. Good order is almost free credit.

COMPUTED · TWO CHUNKS OF SIX

Pen = 0.5·(2/6)³ ≈ 0.0185

Our §03 spine example: the dropped is splits the alignment into [The cat][sat on the mat] — two chunks. Four times the penalty of one chunk: the cube amplifies fast.

COMPUTED · THREE CHUNKS OF SIX

Pen = 0.5·(3/6)³ = 0.0625

Candidate C’s exiled sat: eight times the one-chunk penalty. Fragmentation is starting to hurt.

COMPUTED · SIX CHUNKS OF SIX

Pen = 0.5·(6/6)³ = 0.5

The cap. Full scramble costs exactly half the score: METEOR never pays a matched translation to zero — that restraint is a design decision, and β’s steepness is why.

Every piece is built. Time to multiply it all together — and to settle, honestly, the question the workbooks get wrong.

08 THE FINAL SCORE

F-mean × (1 − Pen) — the full calculator

Two factors, two lessons: Fmean grades content coverage, (1 − Pen) grades order coherence. Pick a candidate and watch all six stages assemble.

THE FORMULA · FINAL

\[ \text{METEOR} \;=\; F_{\text{mean}} \times (1 - \text{Pen}) \;=\; \frac{60}{69} \times \left(1 - 0.5\left(\tfrac{2}{6}\right)^{3}\right) \;=\; 0.8696 \times 0.9815 \;=\; \mathbf{0.8535} \]

Strict arithmetic on our spine example: the dropped is makes two chunks, so Pen = 0.0185 and the final score is 0.8535. METEOR is conventionally reported on a 0–1 scale; a “86” you may see elsewhere is just this number rescaled ×100 — nothing more.

THE METEOR CALCULATOR · SIX STAGES, LIVE

AN HONESTY NOTE · THE WORKBOOK VARIANT YOU WILL MEET ELSEWHERE

Why many tutorials print 0.8676 — and what the library actually says

Walk through most METEOR explainers and you will find our same example ending at 0.8676. That version assumes the six matched words form one chunk (Pen = 0.5·(1/6)³ ≈ 0.0023). It is a convenient simplification — but under the actual chunk rule, the unmatched reference word is sits between cat and sitting, snapping the run: the strict count is 2 chunks and the strict score is 0.8535.

Which is right? We ran the pair through real nltk.translate.meteor_score (NLTK 3.9.2, WordNet resources): it prints 0.8535. The library agrees with the strict rule — this page computes it that way, and the calculator above shows the chunk underlines so you can count them yourself.

You now know both classics. Put them side by side on the same sentences and the philosophical gap becomes measurable.

09 METEOR vs BLEU

Two theories of “good translation”

BLEU asks: how many matching word sequences did the system produce? METEOR asks: how much of the reference did it capture, how flexibly did the words match, and how coherently are they ordered? Same sentence, same reference — different verdicts:

FACE-OFF · ALL AGAINST “THE BOY IS PLAYING FOOTBALL.” (BLEU-2 SHOWN — BLEU-4 WOULD BE EVEN CRUELER)

THE FULL LEDGER · CLICK ANY ROW FOR THE WHY

The simplest way to remember BLEU

“How many matching word sequences did the system produce?” — precision of n-gram windows, geometrically combined, length-penalized.

The simplest way to remember METEOR

“How much of the reference did the system capture, how well did its words match, and how coherently were they ordered?” — alignment, recall-heavy F, chunk penalty.

What each is used for

BLEU: fast, corpus-level system comparison — the shared-task default for 20 years. METEOR: sentence-level judgment, richer resources, better human correlation when you can afford the dictionaries.

All this flexibility might suggest METEOR “understands” meaning. It does not — and the failure mode is worth seeing live.

10 LIMITS

Does METEOR understand meaning like a human?

Not completely. METEOR measures lexical correspondence — same words, same stems, listed synonyms, contiguous order. Meaning operations that leave the words intact are invisible to it. The clearest demonstration is one small word: not.

THE NEGATION TRAP · THE SENTENCE THAT MEANS THE OPPOSITE

Reference: “The patient does not have a fever.” Candidate: “The patient has a fever.”

CANDIDATE

REFERENCE — note which words never aligned

LIMIT 1 · NEGATION

The opposite sentence scores 0.55–0.71

Dropping does not inverts the clinical meaning, yet five content words align beautifully — even has stems to have. No stage of the cascade can see that the meaning flipped.

LIMIT 2 · CONTEXT

Synonyms are context-blind

Cold means temperature or illness; a dictionary lists both matches. Synonym alignment resolves spelling, not intent — the resource can’t tell which cold this is.

LIMIT 3 · LANGUAGE RESOURCES

Urdu pays the dictionary tax

English METEOR ships with Porter, WordNet and paraphrase tables. For English → Urdu evaluation, stemming (e.g. بیٹھ گئی vs بیٹھ گیا) and synonymy need Urdu resources many systems lack — Lecture 16 §11’s dictionary-free chrF exists precisely for this gap.

LIMIT 4 · SAFETY

Never the last judge in high-stakes domains

Factual correctness and clinical safety cannot be certified by word overlap. When errors cost lives, the metric is a first filter — humans with MQM error spans (Lecture 16 §13) make the call.

Enough theory — run it. Two code paths: the formula from scratch, and the library way, both with outputs you have already seen computed on this page.

11 CODE LAB

METEOR in Python — from scratch and from NLTK

Both scripts are runnable as printed; both terminals show real, verified output — the scratch version matches this page’s engine exactly, and the NLTK terminal is a genuine run of nltk 3.9.2 with the wordnet resource (the same numbers this page cross-checked in §08 and §10).

METEOR · TWO WAYS
Why .split()?

NLTK’s meteor_score expects tokenized input — lists of strings, not raw sentences. That’s why every example ends in .split().

The dictionary tax, live

Watch the child + soccer row: 0.5111 from NLTK versus 0.996 from our classroom dictionary — WordNet simply doesn’t list child ↔ boy. A METEOR score is only as rich as its resources.

For English → Urdu

Do not assume English-oriented stemmers and WordNet carry over. Language-specific tokenization, morphology and synonym resources decide what your METEOR can see — test with real Urdu references before trusting the number.

12 CLASSROOM SUMMARY

METEOR in three moves

STAGE 1 · MATCH

Align words beyond exact matches

Run the cascade — exact → stem → synonym → paraphrase — one word to one word, and count the matches m. This is where meaning-flexibility enters.

STAGE 2 · MEASURE

Precision, recall, recall-weighted F

P = m/c, R = m/r, then Fmean = 10PR/(R+9P) — the harmonic mean that trusts recall 9× because humans do.

STAGE 3 · PENALIZE

Discount fragmented alignments

Count contiguous chunks; Pen = 0.5·(ch/m)³ — capped at half, so matched meaning never scores zero, but chaos always pays.

THE ONE EQUATION TO KEEP

\[ \text{METEOR} \;=\; \underbrace{F_{\text{mean}}}_{\text{precision + recall}} \;\times\; \underbrace{(1 - \text{Pen})}_{\text{word-order coherence}} \]

The big lesson: METEOR moves machine-translation evaluation beyond exact word matching — crediting stems, synonyms and paraphrases, weighing what was captured, pricing word order softly. But it remains an automatic approximation of translation quality, built from dictionaries and counts: not a replacement for human judgment.

Continue the series

TER & chrF → dictionary-free answers to reordering and Urdu morphology, and BERTScore & COMET → embeddings that finally see past spelling.

The original paper

S. Banerjee & A. Lavie (ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for MT, 2005), “METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments”.

The reference implementation

CMU’s METEOR page — the metric’s home across its 1.0–2.0 evolution (function-word discount, tuneable parameters, many languages); also wrapped by Hugging Face evaluate.

13 MASTERY CHECK

Eight questions — grade the grader

Matching modules, the 9-to-1 weighting, the chunk rule, the 0.5 cap, and the traps. Answer all eight to close the page.

YOUR SCORE

0 / 8