◈ DEEP DIVE — COMET · the metric that was taught to judge · real wmt22 run
From similarity
to judgment
Every metric in this series computes — counts, alignments, edits, cosines. COMET — the Crosslingual Optimized Metric for Evaluation of Translation (Rei et al., EMNLP 2020) — was trained to judge: feed it the source, a reference and the machine’s translation, and a neural network supervised by thousands of human quality ratings predicts how good the translation is. The question changes one final time: from “how similar are these sentences?” to “how good is this translation likely to be?”
BERTSCORE ASKS (§01)
\[ \text{similarity}(h, r) \;\text{— fixed formula} \]COMET ASKS (§03)
\[ \hat{q} = f_\theta(s, r, h) \;\text{— a trained judge} \]THE PAYOFF (§09 · WMT19 τ)
\[ \text{COMET } 0.445 \;>\; \text{chrF } 0.371 \;>\; \text{BLEU } 0.321 \]THE JUDGMENT GAP, IN ONE SENTENCE — Source/Reference: “The doctor did not prescribe the medicine.”
Hypothesis: “The doctor prescribed the medicine.” — the opposite medical fact, wearing the reference’s words.
The BERTScore dive ended on its caution: similarity is not judgment. This page builds the judge — and shows what training on humans actually buys.
01 WHY COMET
The judgment gap
One source sentence, three candidates — including the dropped not that reverses a medical instruction. The two overlap columns are computed live by this page’s BLEU-1 and chrF engines; both are generous to the meaning-reversing candidate.
Words, not facts
4 of the 5 hypothesis tokens are exact reference words; the characters overlap even more. To BLEU and chrF the reversed sentence is a decent translation with a brevity problem.
Similarity, still not judgment
Contextual embeddings soften the miss, but the bertscore dive showed the real library scoring the negation above the synonym rewrite (0.9839 vs 0.9626). Nothing in its formula was ever told what humans care about.
Supervision from humans
COMET is trained on human quality ratings (Direct Assessment, MQM). Dropped negations are heavily penalized in that data — so the learned function penalizes them too. Judgment is inherited from the raters.
Learned ≠ infallible
A learned estimator can be wrong — domain terminology, rare languages, numerals, subtle shifts. Safety-critical use pairs it with targeted checks and human review (§09).
Before the formula, the evidence: three inputs, each carrying something the others can’t.
02 THE THREE INPUTS
s, r, h — three kinds of evidence
Every metric before COMET compared two texts. Reference-based COMET reads three — and the third input, the source, is the one that carries the original meaning.
Three inputs, one learned function. The whole metric is one line — and every symbol earns its place.
03 THE FORMULA
One line, five identifiers
| Feature | BERTScore | COMET |
|---|---|---|
| Main approach | Contextual token similarity | Learned quality prediction |
| Typical inputs | Candidate + reference | Source + candidate + reference |
| Human judgments in training | Not used by the standard calculation | Used as supervision |
| Output | Precision, recall, F1 | Predicted quality score |
| Main advantage | Direct semantic-similarity comparison | Learns quality patterns from human judgments |
Neither metric guarantees perfect semantic or factual correctness — the comparison is about mechanism, not infallibility.
“Trained on human judgments” is the sentence to unpack — this is where COMET stops being a formula and becomes a student.
04 LEARNING TO JUDGE
The training loop
Imagine thousands of (source, translation, human-rating) triples. The illustrative set below shows the shape of the data; the loop beneath it is what every COMET checkpoint went through — at scale.
ILLUSTRATIVE TRAINING RECORDS — NOT ACTUAL COMET DATA
Represent the evidence
The neural encoder turns each (source, translation) — plus reference, when present — into representations of the sentences and their relationships.
The head guesses quality
A prediction head maps the combined features to a score \(\hat{q}\) — scalar regression for DA-style checkpoints; other variants train on pairwise rankings instead.
Loss against the human rating
The prediction is compared with the human assessment; the loss updates \(\theta\). Repeat across thousands of examples and the model internalizes the raters’ standards.
Humans grade once, the model grades forever
At scoring time there is no human in the loop — the trained \(f_\theta\) predicts alone. The judgments were baked in during training, which is what makes COMET cheap and what makes it only as wise as its raters.
TOY JUDGE · WHAT A “LEARNED PATTERN” LOOKS LIKE — PICK A CASE (a rules-stand-in for learned weights, honestly labeled)
Inside the network, four steps — and one distinction people get wrong.
05 INSIDE THE NETWORK
Four steps — and the real difference
THE PIPELINE (COMMON REFERENCE-BASED ARCHITECTURES)
THE DISTINCTION PEOPLE GET WRONG
Both use neural networks — only one was trained to judge
BERTScore also runs a pretrained language model. The difference is not “deep learning vs no deep learning” — it is what happens after the encoder. BERTScore applies a fixed, hand-designed procedure: max-cosine matching, then P/R/F1. COMET adds a trained prediction head whose parameters were fitted against human ratings. Same raw material; different contract. One measures similarity because that is what the formula says. The other estimates quality because that is what it was taught.
So you ran COMET and got a number. What is it — and what is it not?
06 READING A SCORE
A ranking — not a percentage of correctness
Suppose a model returns these values. They are illustrative — they demonstrate how a trained judge ranks translations; the code lab runs the real checkpoint on the same sentences.
One more axis of freedom: what if there is no reference at all?
07 VARIANTS
Reference-based vs reference-free
The reference is expensive — someone bilingual must write it. The COMET family includes estimators that skip it entirely.
Run the real thing — the walkthrough’s exact script, on the doctor example, with a genuine wmt22-comet-da checkpoint.
08 CODE LAB
unbabel-comet — the real checkpoint
This is the source walkthrough’s script unchanged. The download needs internet (the checkpoint is ≈2.3 GB); gpus=0 runs on CPU. The model’s actual scores must come from running the code — never hardcode the illustrative numbers from §06, and check the model card for intended language pairs before production use.
Trained on WMT22 Direct Assessment ratings — the standard reference-based checkpoint for the “Comet” column in modern MT papers. The da is the human-score type used as supervision.
Check multilingual coverage on the model card before trusting scores for your pair; the same family powers reference-free estimation where no Urdu reference exists (COMETKiwi).
Checkpoint name, package version, and settings belong in every results table — a COMET score without its checkpoint is not a number, it’s a rumor.
09 THE FULL LADDER
Six metrics, six questions, one table of truth
The series in one breath — each metric asking a different question, and (where measured) how well each answer agrees with human judgment.
SIX QUESTIONS · τ BARS = WMT19 SEGMENT-LEVEL AGREEMENT WITH HUMANS
System comparison · ranking · research
Comparing systems over a large test set, ranking candidate translations, correlating with human judgment in research — and reference-free scoring where no reference exists.
Domain terms · rare languages · numbers
A learned estimator inherits its raters’ blind spots. Terminology, low-resource languages, numerals and subtle meaning shifts remain risk zones.
Metric + checks + humans
Medical, legal, safety-critical translation: pair COMET with targeted checks (negation, dosage, entities) and human MQM review where stakes demand — Lecture 16 §13 is that human standard.
Learned judges, human judges
BERTScore and COMET complement each other — and neither is a perfect substitute for a human. The ladder ends where it began: meaning is judged by those who understand it.
10 MASTERY CHECK
Eight questions — the judge on trial
The three inputs, the learned parameters, what a score means (and doesn’t), COMETKiwi, and the τ table that settles the ladder.