CAS logo CAS NLP LABInteractive Tutorial Series ← Series

◈ DEEP DIVE — COMET · the metric that was taught to judge · real wmt22 run

From similarity
to judgment

Every metric in this series computes — counts, alignments, edits, cosines. COMET — the Crosslingual Optimized Metric for Evaluation of Translation (Rei et al., EMNLP 2020) — was trained to judge: feed it the source, a reference and the machine’s translation, and a neural network supervised by thousands of human quality ratings predicts how good the translation is. The question changes one final time: from “how similar are these sentences?” to “how good is this translation likely to be?”

BERTSCORE ASKS (§01)

\[ \text{similarity}(h, r) \;\text{— fixed formula} \]

COMET ASKS (§03)

\[ \hat{q} = f_\theta(s, r, h) \;\text{— a trained judge} \]

THE PAYOFF (§09 · WMT19 τ)

\[ \text{COMET } 0.445 \;>\; \text{chrF } 0.371 \;>\; \text{BLEU } 0.321 \]

THE JUDGMENT GAP, IN ONE SENTENCE — Source/Reference: “The doctor did not prescribe the medicine.”

Hypothesis: “The doctor prescribed the medicine.” — the opposite medical fact, wearing the reference’s words.

3INPUTSsource · reference · hypothesis
7LIVE LABSincl. a toy judge + the τ ladder
1REAL MODEL RUNunbabel-comet · wmt22-comet-da
8QUIZ CHECKmastery before you leave
~35MINUTEScompanion to Lecture 16 §12

The BERTScore dive ended on its caution: similarity is not judgment. This page builds the judge — and shows what training on humans actually buys.

01 WHY COMET

The judgment gap

One source sentence, three candidates — including the dropped not that reverses a medical instruction. The two overlap columns are computed live by this page’s BLEU-1 and chrF engines; both are generous to the meaning-reversing candidate.

SOURCE: “THE DOCTOR DID NOT PRESCRIBE THE MEDICINE.” WORD OVERLAP CHAR OVERLAP — CLICK A ROW

WHAT OVERLAP SEES

Words, not facts

4 of the 5 hypothesis tokens are exact reference words; the characters overlap even more. To BLEU and chrF the reversed sentence is a decent translation with a brevity problem.

WHAT BERTSCORE ADDS — AND DOESN'T

Similarity, still not judgment

Contextual embeddings soften the miss, but the bertscore dive showed the real library scoring the negation above the synonym rewrite (0.9839 vs 0.9626). Nothing in its formula was ever told what humans care about.

WHAT COMET CHANGES

Supervision from humans

COMET is trained on human quality ratings (Direct Assessment, MQM). Dropped negations are heavily penalized in that data — so the learned function penalizes them too. Judgment is inherited from the raters.

THE HONEST CAVEAT

Learned ≠ infallible

A learned estimator can be wrong — domain terminology, rare languages, numerals, subtle shifts. Safety-critical use pairs it with targeted checks and human review (§09).

Before the formula, the evidence: three inputs, each carrying something the others can’t.

02 THE THREE INPUTS

s, r, h — three kinds of evidence

Every metric before COMET compared two texts. Reference-based COMET reads three — and the third input, the source, is the one that carries the original meaning.

Read the triple like a courtroom. The source is the fact being translated; the reference is one witness of what a good translation looks like (note its synonym medication — the reference is an answer, not the answer); the hypothesis is on trial. Overlap metrics cross-examine only the witness; COMET reads all three statements.

Three inputs, one learned function. The whole metric is one line — and every symbol earns its place.

03 THE FORMULA

One line, five identifiers

THE FORMULA · A REFERENCE-BASED COMET MODEL, ABSTRACTED

\[ \hat{q} = f_\theta(s, r, h) \]

\(s\) — the source sentence · \(r\) — the reference translation · \(h\) — the hypothesis (the translation being graded) · \(f_\theta\) — a trained neural network · \(\theta\) — its learned parameters · \(\hat{q}\) — the predicted quality score. Feed in the evidence; the judge returns a verdict. (An abstraction, not one fixed architecture — checkpoints differ in encoders, representations and heads.)

FeatureBERTScoreCOMET
Main approachContextual token similarityLearned quality prediction
Typical inputsCandidate + referenceSource + candidate + reference
Human judgments in trainingNot used by the standard calculationUsed as supervision
OutputPrecision, recall, F1Predicted quality score
Main advantageDirect semantic-similarity comparisonLearns quality patterns from human judgments

Neither metric guarantees perfect semantic or factual correctness — the comparison is about mechanism, not infallibility.

“Trained on human judgments” is the sentence to unpack — this is where COMET stops being a formula and becomes a student.

04 LEARNING TO JUDGE

The training loop

Imagine thousands of (source, translation, human-rating) triples. The illustrative set below shows the shape of the data; the loop beneath it is what every COMET checkpoint went through — at scale.

ILLUSTRATIVE TRAINING RECORDS — NOT ACTUAL COMET DATA

LOOP 1 · ENCODE

Represent the evidence

The neural encoder turns each (source, translation) — plus reference, when present — into representations of the sentences and their relationships.

LOOP 2 · PREDICT

The head guesses quality

A prediction head maps the combined features to a score \(\hat{q}\) — scalar regression for DA-style checkpoints; other variants train on pairwise rankings instead.

LOOP 3 · COMPARE & UPDATE

Loss against the human rating

The prediction is compared with the human assessment; the loss updates \(\theta\). Repeat across thousands of examples and the model internalizes the raters’ standards.

THE POINT

Humans grade once, the model grades forever

At scoring time there is no human in the loop — the trained \(f_\theta\) predicts alone. The judgments were baked in during training, which is what makes COMET cheap and what makes it only as wise as its raters.

TOY JUDGE · WHAT A “LEARNED PATTERN” LOOKS LIKE — PICK A CASE (a rules-stand-in for learned weights, honestly labeled)

Inside the network, four steps — and one distinction people get wrong.

05 INSIDE THE NETWORK

Four steps — and the real difference

THE PIPELINE (COMMON REFERENCE-BASED ARCHITECTURES)

THE DISTINCTION PEOPLE GET WRONG

Both use neural networks — only one was trained to judge

BERTScore also runs a pretrained language model. The difference is not “deep learning vs no deep learning” — it is what happens after the encoder. BERTScore applies a fixed, hand-designed procedure: max-cosine matching, then P/R/F1. COMET adds a trained prediction head whose parameters were fitted against human ratings. Same raw material; different contract. One measures similarity because that is what the formula says. The other estimates quality because that is what it was taught.

So you ran COMET and got a number. What is it — and what is it not?

06 READING A SCORE

A ranking — not a percentage of correctness

Suppose a model returns these values. They are illustrative — they demonstrate how a trained judge ranks translations; the code lab runs the real checkpoint on the same sentences.

The real checkpoint, for contrast. Running §08’s script (Unbabel/wmt22-comet-da, CPU): the synonym swap scores 0.9583, the dropped-“not” sentence 0.8610, system 0.9096. The ranking matches the illustrative table’s direction — and beats BERTScore, which had ranked the negation above the synonym (0.9839 vs 0.9626). But note the honest gap: 0.86 is still a respectable-looking number for a reversed medical fact. Three rules for reading \(\hat{q}\): (1) 0.91 ≠ 91% correct — scores live on a checkpoint-specific scale (some variants even leave [0, 1]). (2) Same checkpoint, same config whenever you compare numbers. (3) Judge many sentences — one example is an anecdote; a test set is evidence.

One more axis of freedom: what if there is no reference at all?

07 VARIANTS

Reference-based vs reference-free

The reference is expensive — someone bilingual must write it. The COMET family includes estimators that skip it entirely.

Run the real thing — the walkthrough’s exact script, on the doctor example, with a genuine wmt22-comet-da checkpoint.

08 CODE LAB

unbabel-comet — the real checkpoint

This is the source walkthrough’s script unchanged. The download needs internet (the checkpoint is ≈2.3 GB); gpus=0 runs on CPU. The model’s actual scores must come from running the code — never hardcode the illustrative numbers from §06, and check the model card for intended language pairs before production use.

COMET · WMT22-COMET-DA
Why wmt22-comet-da

Trained on WMT22 Direct Assessment ratings — the standard reference-based checkpoint for the “Comet” column in modern MT papers. The da is the human-score type used as supervision.

English → Urdu

Check multilingual coverage on the model card before trusting scores for your pair; the same family powers reference-free estimation where no Urdu reference exists (COMETKiwi).

Record your setup

Checkpoint name, package version, and settings belong in every results table — a COMET score without its checkpoint is not a number, it’s a rumor.

09 THE FULL LADDER

Six metrics, six questions, one table of truth

The series in one breath — each metric asking a different question, and (where measured) how well each answer agrees with human judgment.

SIX QUESTIONS · τ BARS = WMT19 SEGMENT-LEVEL AGREEMENT WITH HUMANS

The empirical payoff. COMET 0.445 > BLEURT 0.436 > YiSi-1 0.426 > chrF 0.371 > BLEU 0.321 (Rei et al. 2020, Table 2 — WMT19 Kendall’s τ, segment level). The learned judges top the table; BLEU brings up the rear. Not because BLEU is broken — because counting matches was only ever a proxy for the thing COMET was trained on directly: human judgment.

WHEN COMET SHINES

System comparison · ranking · research

Comparing systems over a large test set, ranking candidate translations, correlating with human judgment in research — and reference-free scoring where no reference exists.

WHEN TO BE CAREFUL

Domain terms · rare languages · numbers

A learned estimator inherits its raters’ blind spots. Terminology, low-resource languages, numerals and subtle meaning shifts remain risk zones.

FOR SAFETY-CRITICAL MT

Metric + checks + humans

Medical, legal, safety-critical translation: pair COMET with targeted checks (negation, dosage, entities) and human MQM review where stakes demand — Lecture 16 §13 is that human standard.

THE SERIES' LAST WORD

Learned judges, human judges

BERTScore and COMET complement each other — and neither is a perfect substitute for a human. The ladder ends where it began: meaning is judged by those who understand it.

10 MASTERY CHECK

Eight questions — the judge on trial

The three inputs, the learned parameters, what a score means (and doesn’t), COMETKiwi, and the τ table that settles the ladder.

YOUR SCORE

0 / 8