◈ DEEP DIVE — BERTSCORE · embed → compare → match · every number verified
Grade the vectors,
not the spelling
Every metric so far compared strings — words, stems, characters, edits. BERTScore (Zhang et al., ICLR 2020) swaps the string for a contextual embedding: each token becomes a vector from a pretrained language model, tokens are compared by cosine similarity, and every token greedily claims its best counterpart. The question changes from “are the words the same?” to “how similar are the words in context?” — and kid finally gets credit for meaning child.
STAGE 2 · SIMILARITY (§04)
\[ \operatorname{sim}(c_i, r_j) = \frac{\mathbf{c}_i \cdot \mathbf{r}_j}{\lVert \mathbf{c}_i \rVert \, \lVert \mathbf{r}_j \rVert} \]STAGE 4 · PRECISION (§06)
\[ P_{\text{BERT}} = \frac{1}{|C|} \sum_{i} \max_{j} \operatorname{sim}(c_i, r_j) \]STAGE 6 · F1 (§08)
\[ F_1 = \frac{2\,P\,R}{P+R} \]THE WHOLE METRIC IN ONE PICTURE — Reference: “The child is happy.”
Candidate: “The kid is joyful.” — zero content words match by spelling; every one matches by meaning.
CANDIDATE — dot color = the reference token it claimed
REFERENCE
Lecture 19 §12 introduced BERTScore as “greedy embedding matching — it compares BAGS of meaning vectors.” This page opens the bag: where each number comes from, why greedy beats exact matching, and exactly where the meaning-metric still fails.
01 WHY BERTSCORE
The overlap wall
One reference sentence, three candidates — the same trio that breaks every string metric in a different place. The scores below are computed live by this page’s toy engine on the §03 embeddings.
kid ≠ child
Word-overlap metrics (BLEU, exact METEOR stages) refuse the synonym pair: different strings, zero credit — Candidate B scores like a half-wrong translation.
happy ≈ unhappy
chrF reads 5 of 6 character bigrams shared — character overlap LOVES antonyms that share a prefix. Surface similarity mistakes negation for kinship.
Compare meanings in context
Contextual embeddings put kid beside child and — in a real model — push unhappy away from happy. Vector similarity is flexible in exactly the way strings are not.
Similarity is not truth
BERTScore does not verify facts, logic or roles: negation flips, number changes (“five” vs “fifteen”) and actor reversals can all hide inside a high score (§09 — demonstrated, not asserted).
Three stages, one pipeline. Let’s meet the machine before opening each gear.
02 THE MACHINE
Three stages, no dictionaries
Notice what’s missing: no stemmer, no synonym table, no paraphrase database. Everything METEOR needed as a resource, BERTScore gets from a pretrained model — the same models the series has been training since Lecture 11.
Words → contextual vectors
A pretrained language model (BERT / RoBERTa / XLM-R class) reads each sentence and produces one vector per token — a vector that depends on the surrounding words (§03).
Every token × every token
All-pairs cosine similarity builds the similarity matrix — every candidate token scored against every reference token (§04–05).
Greedy best matches → P, R, F1
Each candidate token keeps its best reference score (precision), each reference token keeps its best candidate score (recall), and the harmonic mean seals it (§06–08).
Where it sits in the story
Zhang, Kishore, Wu, Weinberger & Artzi (ICLR 2020) — the metric that let evaluation ride the pretrained-model wave. English → Urdu? Use a multilingual checkpoint (XLM-R) — the default English model won’t see Urdu tokens.
Stage 1: where do the numbers on this page come from — and what makes an embedding “contextual”?
03 STAGE 1 · CONTEXTUAL EMBEDDINGS
Words become directions
A computer cannot compare meanings directly — so a pretrained model converts each token into a vector. Our teaching toys are 3-dimensional; real models emit 768–1024 dimensions. The geometry is identical: similar meaning → similar direction.
| Word | Illustrative embedding | Nearest neighbor in the table | Cosine (live) |
|---|---|---|---|
| child | [0.82, 0.31, 0.65] | kid | 0.9994 |
| kid | [0.80, 0.34, 0.63] | child | 0.9994 |
| happy | [0.15, 0.91, 0.42] | joyful | 0.9989 |
| joyful | [0.18, 0.89, 0.45] | happy | 0.9989 |
| the | [0.90, 0.12, 0.20] | is (far away) | 0.3350 |
| is | [0.10, 0.15, 0.92] | the (far away) | 0.3350 |
LIVE COSINE LAB · CLICK A PAIR — computed from the table above, not hardcoded
WHY “CONTEXTUAL”? · ONE WORD, TWO MEANINGS, TWO VECTORS
The string bank is identical in both sentences — its vector is not.
CONTEXTUAL VECTOR FOR “BANK”
Vectors in hand — the comparison operator. One formula, four identifiers, one tiny worked example.
04 STAGE 2 · COSINE SIMILARITY
Comparing directions, not lengths
Cosine similarity asks: do two vectors point the same way? \(1\) = aligned, \(0\) = orthogonal, \(-1\) = opposite. Vector length (how strongly the token fires) is normalized away — only the direction of meaning survives.
WORKED-EXAMPLE STEPPER · PICK A PAIR, WATCH THE SUBSTITUTION
c = [1, 2], r = [2, 3]: dot = 1·2 + 2·3 = 8, magnitudes √5 and √13, cosine = 8/√65 ≈ 0.992. Nearly the same direction — a very high similarity.
Raw dot products grow with vector length — frequent, high-energy tokens would dominate. Normalizing by both magnitudes makes every comparison purely about direction.
The same formula runs on real contextual token embeddings (768+ dims) — with optional IDF weighting so rare, informative tokens count more than “the” (see the code lab notes).
Now multiply that one number across every pair of tokens — the similarity matrix, and the greedy matching rule that reads it.
05 STAGE 3 · THE SIMILARITY MATRIX
Every token meets every token
Four candidate tokens × four reference tokens = sixteen cosine similarities. The values below are the illustrative matrix from the source walkthrough — plausible numbers for the toy embeddings of §03, chosen so every later digit is checkable by hand.
HEATMAP EXPLORER · GREEN INTENSITY = SIMILARITY · CLICK THE DIRECTION TOGGLE
Best match wins — no exclusivity
Look at kid’s row: [0.22, 0.92, 0.18, 0.42]. The max is 0.92 → kid matches child. Likewise joyful → happy at 0.91. Nothing is consumed: two candidate tokens may lean on the same reference token.
Greedy ≠ one-to-one
METEOR’s alignment is a matching: each word claims at most one partner. BERTScore’s is a covering: every token independently takes its max. Cheaper to compute — and it is why order blindness creeps in (§09).
The matrix has no word order
Row order and column order carry no constraint — token 1 may match token 4 as freely as token 1. Word order survives only implicitly, inside the contextual vectors themselves.
The price of flexibility
|C| × |R| cosine comparisons per sentence pair, plus a transformer forward pass for both sentences — vastly more than BLEU’s n-gram counting. This is why BERTScore is batched offline, not computed per-token in a decoder loop.
Read the matrix in the precision direction: rows, best matches, one average.
06 STAGE 4 · PRECISION
Is everything the candidate wrote defensible?
SUBSTITUTING THE MATRIX · “THE KID IS JOYFUL” VS “THE CHILD IS HAPPY”
Flip the direction — and drop one small word from the reference to see recall do its job.
07 STAGE 5 · RECALL
How much of the reference was covered?
THE “VERY” OMISSION · REFERENCE: “THE CHILD IS VERY HAPPY.” · CANDIDATE: “THE CHILD IS HAPPY.”
Two directions, one seal — and the imbalance tax that makes the harmonic mean the right blender.
08 STAGE 6 · F1
The harmonic seal — and a playground
On the headline example P = R = 0.9525, so \(F_1 = \frac{2PR}{P+R} = 0.9525\) — the harmonic mean of equal values is that value. The interesting behavior appears when they disagree.
F1 PLAYGROUND · DRAG P AND R, WATCH THE HARMONIC MEAN RESIST
BALANCEDFlexibility has a price. Five sentences BERTScore grades dangerously well — one of them live.
09 LIMITS
What it detects — and what it misses
BERTScore measures how similar the contextual token representations are. It does not prove that the translation preserved every fact, relationship or logical detail — and it is worth seeing exactly where that bites.
THE CAUTION LEDGER · CLICK ANY ROW FOR THE WHY
THE NEGATION TRAP, SCORED · “THE CHILD IS UNHAPPY.” VS “THE CHILD IS HAPPY.”
Swap subject and object — “the doctor treated the patient” ↔ the reverse — and every token still finds a perfect partner: F1 stays near 1.0. Lecture 19 §12 showed this exact failure on the Urdu role flip.
Five vs fifteen tablets: both are numbers beside “tablets”, so their vectors sit close. Dosage errors need rule-based checks, not embedding similarity.
COMET → feeds (source, translation, reference) into a model trained on human judgments — it learns that role flips and negations are errors, because raters told it so. Similarity becomes prediction.
Run it two ways: the library with a real pretrained model, and the page’s engine from scratch.
10 CODE LAB
BERTScore in Python — library and scratch
The scratch tab reproduces this page’s toy engine exactly (real cosines, real argmax matching — genuine run). The library tab is the source walkthrough’s script; run it yourself — do not assume the synonym example must beat the negation example until you have seen the numbers.
Beyond real embeddings: IDF weighting (rare tokens count more), baseline rescaling (scores mapped against a chance-similarity baseline), batching, and multilingual checkpoints via lang= or model_type=.
Record the model, package version and settings with every number — rescaling and IDF change results materially. Our recorded run (roberta-large, lang="en"): the synonym sentence F1 0.9626, the negation F1 0.9839 — the trap, confirmed by the library itself.
Use a multilingual model covering both languages (e.g. XLM-RoBERTa class) — the default English model cannot embed Urdu tokens. Lecture 19 §12 pairs BERTScore with COMET for exactly this pipeline.
11 CLASSROOM SUMMARY
BERTScore in five moves
A language model reads both sentences
Each token becomes a contextual vector — same word, different sentence, different vector.
Cosine over every pair
The similarity matrix: \(\operatorname{sim}(c_i,r_j) = \frac{\mathbf{c}_i\cdot\mathbf{r}_j}{\lVert\mathbf{c}_i\rVert\lVert\mathbf{r}_j\rVert}\).
Greedy best matches
Every token keeps its maximum-similarity partner — no exclusivity, no order constraint.
P, R — then the harmonic F1
Mean of row maxima (precision), mean of column maxima (recall), blended by \(F_1 = \frac{2PR}{P+R}\).
T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger & Y. Artzi (ICLR 2020), “BERTScore: Evaluating Text Generation with BERT” — IDF weighting, baseline rescaling and the correlation tables that made it a standard.
Tiiiger/bert_score — the reference implementation behind pip install bert-score.
COMET → — from similarity to prediction — then MQM error spans, the human standard every metric is chasing.
12 MASTERY CHECK
Eight questions — vectors, matrices, traps
Contextual embeddings, the cosine, greedy matching, the F1 seal — and the five sentences that still get past it.