CAS logo CAS NLP LABInteractive Tutorial Series ← Series

◈ DEEP DIVE — TER & chrF · repair work & character overlap · every number verified

Metrics that think
like an editor

BLEU counts matching word sequences. METEOR aligns words and weighs recall. But an engineer shipping an English → Urdu system asks two blunter questions: how much rework would this output need? — that is TER, the Translation Edit Rate (Snover et al., 2006) — and how close are the words even when they miss? — that is chrF, the character n-gram F-score (Popović, 2015). One prices repairs, the other prices surface similarity. This page builds both, edit by edit and bigram by bigram.

TER · AN ERROR RATE (§03)

\[ \mathrm{TER} = \frac{\#\,\text{edits}}{\text{reference words}} \]

THE FOUR EDITS (§02)

\[ \text{insert} \;\; \text{delete} \;\; \text{substitute} \;\; \text{shift} \;\;\; (\text{each} = 1) \]

CHR F₂ · SIMILARITY (§07)

\[ \mathrm{chrF}_{\beta} = (1+\beta^2)\,\frac{P_{\text{char}}\,R_{\text{char}}}{\beta^2 P_{\text{char}} + R_{\text{char}}} \]

THE WHOLE LECTURE IN ONE SWITCH — Reference: “The cat sat on the mat.”

Candidate: “On the mat the cat sat.” — every word correct, one phrase relocated.

CANDIDATE — amber = the traveling phrase

REFERENCE (fixed)

TER · SHIFTS ALLOWED
0.0%

4EDIT TYPESshift moves a whole phrase for 1
11LIVE LABSledger, bigrams, β-dial, Urdu RTL
2REAL LIBRARY RUNSsacrebleu 2.6.0 · python scratch
8QUIZ CHECKmastery before you leave
~40MINUTEScompanion to Lecture 19 §11

Lecture 19 §11 met TER and chrF as “the dictionary-free workhorses.” Here we open the machines: where 16.7% comes from, why one shift beats six edits, and how characters rescue Urdu morphology — with every number computed live.

01 WHY TER & CHR F

Two questions engineers actually ask

A metric is a theory of what “good” means. TER and chrF answer shippable questions: TER estimates the post-editing effort — how many repairs until the machine’s sentence equals the reference. chrF measures surface similarity — how much of the reference’s character stream the candidate reproduces, catching spelling, word-form and suffix errors that whole-word metrics zero out.

QUESTION 1 · POST-EDIT EFFORT

How much work to fix this output?

If a human editor must touch every second word, the system isn’t saving anyone time. TER counts the repairs: insert, delete, substitute — and the phrase shift that makes word-order fixes affordable.

QUESTION 2 · PARTIAL MATCHES

How close are the words that miss?

کتاب vs کتابیں (book / books) is a whole-word miss but a character near-hit. chrF pays partial credit through character n-grams — no stemmer, no dictionary, no tokenizer.

DIRECTION MATTERS

One is an error rate, one is a score

TER: lower is better, 0 = perfect, unbounded above (yes, it can exceed 100%). chrF: higher is better, 1 = perfect, capped. Never average the two; never compare their numbers directly.

WHY ENGINEERS PAIR THEM

They fail in opposite places

TER punishes any rewrite — even a fluent synonym — as edits. chrF forgives rewrites but can’t see moved words (same characters!). Together with BLEU they triangulate quality; alone, each can be gamed.

REFERENCE: “I LIKE APPLES.” (r = 3) EDITS TER — CLICK A ROW

Every TER number is built from four — and only four — repair operations. Meet the toolbox.

02 THE EDIT TOOLBOX

Four repairs, each costs one

TER lives in edit-distance land, but with a crucial extra move. Insertion, deletion and substitution behave exactly as in Levenshtein distance — one word, one edit. The shift is new: a contiguous phrase of any length travels to a new position and the entire move costs exactly one edit.

EDIT EXPLORER · CLICK A REPAIR, WATCH BEFORE → AFTER

BEFORE (candidate)

AFTER (reference)

THE FORMULA · TRANSLATION EDIT RATE

\[ \mathrm{TER} \;=\; \frac{\text{minimum number of edits (insert + delete + substitute + shift)}}{\text{number of words in the reference}} \]

Fewer edits → better translation. Zero edits → exact match. And because the numerator counts all repairs while the denominator is fixed, a bad enough candidate pushes TER past 1.0 — it is a rate, not a probability (Snover et al., AMTA 2006).

Formula in hand — run it on the classic worked examples, including the one that breaks the 100% ceiling.

03 TER LAB

Worked examples, ledger style

Pick a candidate; the repair ledger runs live — the Levenshtein core, the shifts the search finds, and both rates side by side. All numbers are computed by the same engine this page verified against Python and sacrebleu.

THE TER CALCULATOR · CANDIDATE VS “I LIKE APPLES” / THE 6-WORD MAT SENTENCE

One row of that ledger deserves its own section — the row where WER says 100% and TER says 16.7%.

04 WHY SHIFTS MATTER

Humans repair order by moving phrases

Plain word error rate (WER) knows only insert/delete/substitute — so a relocated phrase is demolished and rebuilt, paying two edits per word. Real editors don’t do that: they select the phrase and drag it. TER’s shift operation prices the drag at one edit, which is why TER tracks post-editing effort while WER inflates it.

FACE-OFF · SAME SENTENCES, BOTH EDIT MODELS (LIVE-COMPUTED)

THE SEARCH PROBLEM

Finding good shifts is a search

Which phrase, moved where, helps most? Real TER implementations search for beneficial shifts (with pruning) — the optimal sequence isn’t trivial to compute. Our lab runs a small greedy search: try every span ≤ 4 words at every position, keep the best improvement, repeat.

READING A TER NUMBER

Edits per 100 reference words

System A at TER 0.12 ≈ 12 repairs per 100 words; System B at 0.28 ≈ 28. A post-editor facing B works more than twice as hard — that is the decision TER supports.

THE HONEST CAVEAT

TER measures reference-matching, not taste

TER counts edits toward the reference — not every edit a human would actually make for fluency. A perfect paraphrase can score badly; a stilted word-for-word match scores 0. It prices effort against one gold answer, assuming that answer is worth reproducing.

WHERE YOU’VE SEEN IT

Lecture 19’s Urdu candidates

The reordered candidate چٹائی پر بلی بیٹھ گئی cost 1 shift → TER 20% where WER paid 2 substitutions → 40%. Same lesson as the English rotation above — now in a language where word order is genuinely free.

TER is done. Now stop counting words and start looking inside them.

05 CHARACTER N-GRAMS

Look inside the words

An n-gram is a contiguous run of \(n\) units — for chrF the units are characters. The word cat holds 1-grams \(\{c,a,t\}\), 2-grams \(\{ca, at\}\), the 3-gram \(\{cat\}\). Add one plural s and cats holds \(\{ca, at, ts\}\): two of three bigrams still match. A whole-word metric calls that a miss; characters call it two-thirds of a hit.

BIGRAM LAB · TEAL = SHARED WITH THE OTHER WORD · GREY = EXCLUSIVE

CANDIDATE — bigrams

REFERENCE — bigrams

Why morphology matters

Morphologically rich languages — Urdu among them — change words through prefixes, suffixes and inflections. Whole-word metrics zero these near-misses; character n-grams pay partial credit automatically.

No tokenizer required

chrF’s basic computation reads raw characters — a decisive practical feature where tokenization is contested or inconsistent between systems.

The honest limit

Character similarity is surface similarity. Two words can share most of their letters and mean entirely different things — credit for characters is not credit for meaning.

Those shared/exclusive bigram chips are all chrF needs. First the two classic measures, then the combine.

06 CHR F PRECISION & RECALL

Two directions of the same overlap

Exactly the precision/recall idea you met in METEOR — but the units are character n-grams, not words. Switch pairs and watch which side pays for the exclusive grey bigrams.

CANDIDATE — bigrams (grey hurts P)

REFERENCE — bigrams (grey hurts R)

THE TWO FORMULAS · ON THE cat / cats EXAMPLE

\[ P_{\text{char}} = \frac{\text{matching candidate n-grams}}{\text{candidate n-grams}} = \frac{2}{3} \approx 0.667 \qquad\qquad R_{\text{char}} = \frac{\text{matching reference n-grams}}{\text{reference n-grams}} = \frac{2}{2} = 1.0 \]

Precision asks: how much of what the candidate wrote is backed by the reference? — the exclusive ts drags it down. Recall asks: how much of the reference’s character stream was captured? — both of ca, at were, so R = 1. The plural s cost the candidate precision, and the reference nothing.

Two numbers, one combine — with a dial that decides how much recall matters.

07 THE F-β COMBINE

chrF₂ — where the 2 means something

chrF merges \(P_{\text{char}}\) and \(R_{\text{char}}\) with the weighted harmonic mean. The parameter \(\beta\) sets how much recall matters: at \(\beta = 1\) weights are equal; the standard configuration — chrF2, the sacrebleu default — runs \(\beta = 2\), weighting recall \(4\times\) heavier. Same recall-favoring philosophy as METEOR’s 9:1.

THE FORMULA · SUBSTITUTING cat / cats

\[ \mathrm{chrF}_{2} = (1+2^2)\,\frac{\tfrac{2}{3}\cdot 1}{2^2\cdot\tfrac{2}{3} + 1} = 5\cdot\frac{\tfrac{2}{3}}{\tfrac{11}{3}} = \frac{10}{11} \approx \mathbf{0.9091} \]

A 90.9% score despite 66.7% precision — because every reference bigram was captured and this configuration trusts recall. The metric forgives extra characters; it punishes missing ones.

β DIAL · P = 2/3 AND R = 1 FIXED, β SWEEPS 0.5 → 3

CHR F2 · β = 2 · SACREBLEU DEFAULT
β = 1 — vanilla F1

Equal weights: cat/cats scores 0.80. Already harmonically punished for the missing third of precision.

β = 2 — the standard chrF2

Recall weighted 4×: 0.9091. Chosen because character recall correlates better with human judgment — missing text is worse than extra text.

The full configuration

Real chrF2 doesn’t use one order — it averages P and R over character n-gram orders 1 through 6, β = 2, whitespace stripped. Our hand example isolates order 2 so every digit is checkable; §11 shows the full-order number on a sentence.

Now the reason this metric exists for Urdu — watch a plural suffix become partial credit.

08 THE URDU INSIGHT

کتاب / کتابیں — book becomes books

Urdu plurals live in suffixes: کتاب (kitab, book) → کتابیں (kitabein, books). A whole-word metric sees two different tokens. chrF sees the shared stem کت+اب — and pays for it, with no Urdu stemmer anywhere in sight.

RTL BIGRAM LAB · TEAL = SHARED STEM BIGRAMS · GREY = THE PLURAL SUFFIX

CANDIDATE — کتابیں (kitabein, books)

REFERENCE — کتاب (kitab, book)

The hand math

Candidate bigrams \(\{\)کت، تا، اب، بی، یں\(\}\) — three of five shared → P = 3/5. Reference bigrams all captured → R = 1. chrF2 = 5·(0.6·1)/(4·0.6+1) = 3/3.4 ≈ 0.882. A near-hit, priced fairly.

The gender-slip case

Lecture 19’s candidate G wrote بیٹھ گیا for بیٹھ گئی (he sat / she sat): chrF scored 66.7 — partial credit for the shared verb stem, where BLEU’s whole-word match scored zero.

When engineers reach for chrF

Hard tokenization, rich morphology, inconsistent spellings, cross-language evaluation without per-language tools — chrF runs on raw characters and needs none of them. Include it in an En→Urdu suite; never use it alone.

Three siblings on the bench — TER, chrF, chrF++. What exactly separates them?

09 THE FAMILY

TER vs chrF vs chrF++

chrF++ (Popović 2017) interleaves word n-grams (order ≤ 2) into the character stream — buying back the word-order sensitivity plain chrF lacks, while keeping the morphological credit. Click any row for the why.

THE FULL LEDGER · CLICK ANY ROW FOR THE WHY

Use TER when…

You care about post-editing effort: estimating how much human repair a system’s output will need against its references.

Use chrF when…

You need a robust, tokenizer-free surface measure — especially for morphologically rich or low-resource target languages like Urdu.

Use chrF++ when…

You want chrF’s flexibility plus word-order sensitivity in one number — char ≤ 6 and word ≤ 2, β = 2, one F-score.

Enough theory — run both metrics, two ways, with terminals you can trust.

10 CODE LAB

TER & chrF in Python — library and scratch

Both terminals are genuine runs: the sacrebleu tab is a real sacrebleu 2.6.0 execution of the corpus evaluation; the scratch tab reproduces this page’s engines — greedy phrase shifts and clipped bigrams — digit for digit.

TER & CHR F · TWO WAYS
0–100, not 0–1

SacreBLEU reports TER, chrF and chrF++ on a 0–100 scale. TER’s 23.53 means 23.53 edits per 100 reference words — still an error rate, just rescaled.

Corpus ≠ sentence average

corpus_ter pools all edits and all reference words across the corpus — not the mean of per-sentence TERs. Sentence-level functions (sentence_ter) exist separately.

Record your config

For reproducible results, record the sacrebleu version and metric settings (β, char order, word order) alongside every score — the library prints them in each metric’s signature line.

11 LIMITS

What neither metric sees

TER counts repairs and chrF counts characters — neither reads meaning. The demonstration is one small word again: not.

THE NEGATION TRAP · Reference: “The medicine is safe.” (r = 4)

CANDIDATE — the opposite claim

REFERENCE

THE ONE-LINE LESSON

\[ \textbf{Reference similarity} \;\neq\; \textbf{semantic correctness} \]

A fluent translation that uses very different words can score badly; a meaning-inverting sentence can score well. For research and production alike: run multiple metrics — and where stakes are real, add human evaluation (MQM, Lecture 19 §13).

The three to remember

TER — edits ÷ reference length, lower better, shifts included. chrF — character n-gram F, higher better, β = 2 standard. chrF++ — adds word n-grams (≤ 2) to chrF.

Continue the series

BERTScore & COMET → embeddings and trained judges that finally see past surface forms — the sequel section in Lecture 19.

The papers

M. Snover, B. Dorr, R. Schwartz, L. Micciulla & J. Makhoul (AMTA 2006), “A Study of Translation Edit Rate”; M. Popović (WMT 2015), “chrF: character n-gram F-score for automatic MT evaluation”; chrF++ (Popović 2017, RANLP).

12 MASTERY CHECK

Eight questions — repairs and characters

The four edits, the 16.7% vs 100% story, the unbounded rate, bigram P & R, β weighting, and what chrF++ adds. Answer all eight to close the page.

YOUR SCORE

0 / 8