CAS logo CAS NLP LABInteractive Tutorial Series ← Series

◈ DEEP DIVE — BERTSCORE · embed → compare → match · every number verified

Grade the vectors,
not the spelling

Every metric so far compared strings — words, stems, characters, edits. BERTScore (Zhang et al., ICLR 2020) swaps the string for a contextual embedding: each token becomes a vector from a pretrained language model, tokens are compared by cosine similarity, and every token greedily claims its best counterpart. The question changes from “are the words the same?” to “how similar are the words in context?” — and kid finally gets credit for meaning child.

STAGE 2 · SIMILARITY (§04)

\[ \operatorname{sim}(c_i, r_j) = \frac{\mathbf{c}_i \cdot \mathbf{r}_j}{\lVert \mathbf{c}_i \rVert \, \lVert \mathbf{r}_j \rVert} \]

STAGE 4 · PRECISION (§06)

\[ P_{\text{BERT}} = \frac{1}{|C|} \sum_{i} \max_{j} \operatorname{sim}(c_i, r_j) \]

STAGE 6 · F1 (§08)

\[ F_1 = \frac{2\,P\,R}{P+R} \]

THE WHOLE METRIC IN ONE PICTURE — Reference: “The child is happy.”

Candidate: “The kid is joyful.” — zero content words match by spelling; every one matches by meaning.

CANDIDATE — dot color = the reference token it claimed

REFERENCE

3STAGESembed → cosine → greedy match
9LIVE LABScosine lab, heatmap, F1 dial
2REAL LIBRARY RUNSbert-score · python scratch
8QUIZ CHECKmastery before you leave
~40MINUTEScompanion to Lecture 19 §12

Lecture 19 §12 introduced BERTScore as “greedy embedding matching — it compares BAGS of meaning vectors.” This page opens the bag: where each number comes from, why greedy beats exact matching, and exactly where the meaning-metric still fails.

01 WHY BERTSCORE

The overlap wall

One reference sentence, three candidates — the same trio that breaks every string metric in a different place. The scores below are computed live by this page’s toy engine on the §03 embeddings.

REFERENCE: “THE CHILD IS HAPPY.” TOY BERTSCORE F1 — CLICK A ROW

WHERE EXACT MATCH FAILS

kid ≠ child

Word-overlap metrics (BLEU, exact METEOR stages) refuse the synonym pair: different strings, zero credit — Candidate B scores like a half-wrong translation.

WHERE CHARACTERS FAIL

happy ≈ unhappy

chrF reads 5 of 6 character bigrams shared — character overlap LOVES antonyms that share a prefix. Surface similarity mistakes negation for kinship.

WHAT BERTSCORE ADDS

Compare meanings in context

Contextual embeddings put kid beside child and — in a real model — push unhappy away from happy. Vector similarity is flexible in exactly the way strings are not.

THE STANDING CAUTION

Similarity is not truth

BERTScore does not verify facts, logic or roles: negation flips, number changes (“five” vs “fifteen”) and actor reversals can all hide inside a high score (§09 — demonstrated, not asserted).

Three stages, one pipeline. Let’s meet the machine before opening each gear.

02 THE MACHINE

Three stages, no dictionaries

Notice what’s missing: no stemmer, no synonym table, no paraphrase database. Everything METEOR needed as a resource, BERTScore gets from a pretrained model — the same models the series has been training since Lecture 11.

STAGE 1 · EMBED

Words → contextual vectors

A pretrained language model (BERT / RoBERTa / XLM-R class) reads each sentence and produces one vector per token — a vector that depends on the surrounding words (§03).

STAGE 2 · COMPARE

Every token × every token

All-pairs cosine similarity builds the similarity matrix — every candidate token scored against every reference token (§04–05).

STAGE 3 · MATCH & SCORE

Greedy best matches → P, R, F1

Each candidate token keeps its best reference score (precision), each reference token keeps its best candidate score (recall), and the harmonic mean seals it (§06–08).

ERA NOTE · 2020

Where it sits in the story

Zhang, Kishore, Wu, Weinberger & Artzi (ICLR 2020) — the metric that let evaluation ride the pretrained-model wave. English → Urdu? Use a multilingual checkpoint (XLM-R) — the default English model won’t see Urdu tokens.

Stage 1: where do the numbers on this page come from — and what makes an embedding “contextual”?

03 STAGE 1 · CONTEXTUAL EMBEDDINGS

Words become directions

A computer cannot compare meanings directly — so a pretrained model converts each token into a vector. Our teaching toys are 3-dimensional; real models emit 768–1024 dimensions. The geometry is identical: similar meaning → similar direction.

WordIllustrative embeddingNearest neighbor in the tableCosine (live)
child[0.82, 0.31, 0.65]kid0.9994
kid[0.80, 0.34, 0.63]child0.9994
happy[0.15, 0.91, 0.42]joyful0.9989
joyful[0.18, 0.89, 0.45]happy0.9989
the[0.90, 0.12, 0.20]is (far away)0.3350
is[0.10, 0.15, 0.92]the (far away)0.3350

LIVE COSINE LAB · CLICK A PAIR — computed from the table above, not hardcoded

WHY “CONTEXTUAL”? · ONE WORD, TWO MEANINGS, TWO VECTORS

The string bank is identical in both sentences — its vector is not.

CONTEXTUAL VECTOR FOR “BANK”

Vectors in hand — the comparison operator. One formula, four identifiers, one tiny worked example.

04 STAGE 2 · COSINE SIMILARITY

Comparing directions, not lengths

Cosine similarity asks: do two vectors point the same way? \(1\) = aligned, \(0\) = orthogonal, \(-1\) = opposite. Vector length (how strongly the token fires) is normalized away — only the direction of meaning survives.

THE FORMULA · EVERY IDENTIFIER

\[ \operatorname{sim}(c_i, r_j) = \frac{\mathbf{c}_i \cdot \mathbf{r}_j}{\lVert \mathbf{c}_i \rVert \, \lVert \mathbf{r}_j \rVert} \]

\(c_i\) — the \(i\)-th candidate token · \(r_j\) — the \(j\)-th reference token · \(\mathbf{c}_i \cdot \mathbf{r}_j\) — the dot product (sum of element-wise products) · \(\lVert \mathbf{c}_i \rVert\) — the vector’s magnitude \(\sqrt{\sum_k c_{i,k}^2}\). Divide the dot by both magnitudes and only the angle remains.

WORKED-EXAMPLE STEPPER · PICK A PAIR, WATCH THE SUBSTITUTION

The article’s tiny example

c = [1, 2], r = [2, 3]: dot = 1·2 + 2·3 = 8, magnitudes √5 and √13, cosine = 8/√65 ≈ 0.992. Nearly the same direction — a very high similarity.

Why cosine, not dot product?

Raw dot products grow with vector length — frequent, high-energy tokens would dominate. Normalizing by both magnitudes makes every comparison purely about direction.

In real BERTScore

The same formula runs on real contextual token embeddings (768+ dims) — with optional IDF weighting so rare, informative tokens count more than “the” (see the code lab notes).

Now multiply that one number across every pair of tokens — the similarity matrix, and the greedy matching rule that reads it.

05 STAGE 3 · THE SIMILARITY MATRIX

Every token meets every token

Four candidate tokens × four reference tokens = sixteen cosine similarities. The values below are the illustrative matrix from the source walkthrough — plausible numbers for the toy embeddings of §03, chosen so every later digit is checkable by hand.

HEATMAP EXPLORER · GREEN INTENSITY = SIMILARITY · CLICK THE DIRECTION TOGGLE

THE GREEDY RULE

Best match wins — no exclusivity

Look at kid’s row: [0.22, 0.92, 0.18, 0.42]. The max is 0.92 → kid matches child. Likewise joyful → happy at 0.91. Nothing is consumed: two candidate tokens may lean on the same reference token.

VS METEOR’S ALIGNMENT

Greedy ≠ one-to-one

METEOR’s alignment is a matching: each word claims at most one partner. BERTScore’s is a covering: every token independently takes its max. Cheaper to compute — and it is why order blindness creeps in (§09).

POSITION-BLIND

The matrix has no word order

Row order and column order carry no constraint — token 1 may match token 4 as freely as token 1. Word order survives only implicitly, inside the contextual vectors themselves.

COMPUTATIONAL COST

The price of flexibility

|C| × |R| cosine comparisons per sentence pair, plus a transformer forward pass for both sentences — vastly more than BLEU’s n-gram counting. This is why BERTScore is batched offline, not computed per-token in a decoder loop.

Read the matrix in the precision direction: rows, best matches, one average.

06 STAGE 4 · PRECISION

Is everything the candidate wrote defensible?

THE FORMULA · PRECISION

\[ P_{\text{BERT}} = \frac{1}{|C|} \sum_{i=1}^{|C|} \max_{j} \operatorname{sim}(c_i, r_j) \]

For each candidate token \(c_i\), keep the best similarity over all reference tokens (\(\max_j\)); sum those bests and divide by the candidate length \(|C|\). A mean of maxima.

SUBSTITUTING THE MATRIX · “THE KID IS JOYFUL” VS “THE CHILD IS HAPPY”

Flip the direction — and drop one small word from the reference to see recall do its job.

07 STAGE 5 · RECALL

How much of the reference was covered?

THE FORMULA · RECALL — NOTE THE FLIPPED MAX

\[ R_{\text{BERT}} = \frac{1}{|R|} \sum_{j=1}^{|R|} \max_{i} \operatorname{sim}(c_i, r_j) \]

Now each reference token \(r_j\) keeps its best match over all candidate tokens. Precision audits the output; recall audits the coverage — the same division of labor as METEOR and chrF, one level up the abstraction ladder.

THE “VERY” OMISSION · REFERENCE: “THE CHILD IS VERY HAPPY.” · CANDIDATE: “THE CHILD IS HAPPY.”

Two directions, one seal — and the imbalance tax that makes the harmonic mean the right blender.

08 STAGE 6 · F1

The harmonic seal — and a playground

On the headline example P = R = 0.9525, so \(F_1 = \frac{2PR}{P+R} = 0.9525\) — the harmonic mean of equal values is that value. The interesting behavior appears when they disagree.

THE FORMULA · AND THE IMBALANCE TAX

\[ F_1 = \frac{2\,P\,R}{P+R} \qquad\text{vs (arithmetic)}\qquad \frac{P+R}{2} \]

Precision 0.95 with recall 0.40: arithmetic mean says 0.675 — a gentleman’s C. Harmonic says 0.563. High precision cannot hide missing content; the flatter side drags the score down. Same property that powers METEOR’s F-mean and chrF’s F-β.

F1 PLAYGROUND · DRAG P AND R, WATCH THE HARMONIC MEAN RESIST

BALANCED

Flexibility has a price. Five sentences BERTScore grades dangerously well — one of them live.

09 LIMITS

What it detects — and what it misses

BERTScore measures how similar the contextual token representations are. It does not prove that the translation preserved every fact, relationship or logical detail — and it is worth seeing exactly where that bites.

THE CAUTION LEDGER · CLICK ANY ROW FOR THE WHY

THE NEGATION TRAP, SCORED · “THE CHILD IS UNHAPPY.” VS “THE CHILD IS HAPPY.”

The order-blind twin

Swap subject and object — “the doctor treated the patient” ↔ the reverse — and every token still finds a perfect partner: F1 stays near 1.0. Lecture 19 §12 showed this exact failure on the Urdu role flip.

The numbers trap

Five vs fifteen tablets: both are numbers beside “tablets”, so their vectors sit close. Dosage errors need rule-based checks, not embedding similarity.

What comes next

COMET → feeds (source, translation, reference) into a model trained on human judgments — it learns that role flips and negations are errors, because raters told it so. Similarity becomes prediction.

Run it two ways: the library with a real pretrained model, and the page’s engine from scratch.

10 CODE LAB

BERTScore in Python — library and scratch

The scratch tab reproduces this page’s toy engine exactly (real cosines, real argmax matching — genuine run). The library tab is the source walkthrough’s script; run it yourself — do not assume the synonym example must beat the negation example until you have seen the numbers.

BERTSCORE · TWO WAYS
What the library adds

Beyond real embeddings: IDF weighting (rare tokens count more), baseline rescaling (scores mapped against a chance-similarity baseline), batching, and multilingual checkpoints via lang= or model_type=.

Reproducibility

Record the model, package version and settings with every number — rescaling and IDF change results materially. Our recorded run (roberta-large, lang="en"): the synonym sentence F1 0.9626, the negation F1 0.9839 — the trap, confirmed by the library itself.

For English → Urdu

Use a multilingual model covering both languages (e.g. XLM-RoBERTa class) — the default English model cannot embed Urdu tokens. Lecture 19 §12 pairs BERTScore with COMET for exactly this pipeline.

11 CLASSROOM SUMMARY

BERTScore in five moves

MOVE 1 · EMBED

A language model reads both sentences

Each token becomes a contextual vector — same word, different sentence, different vector.

MOVE 2 · COMPARE

Cosine over every pair

The similarity matrix: \(\operatorname{sim}(c_i,r_j) = \frac{\mathbf{c}_i\cdot\mathbf{r}_j}{\lVert\mathbf{c}_i\rVert\lVert\mathbf{r}_j\rVert}\).

MOVE 3 · MATCH

Greedy best matches

Every token keeps its maximum-similarity partner — no exclusivity, no order constraint.

MOVES 4–5 · SCORE

P, R — then the harmonic F1

Mean of row maxima (precision), mean of column maxima (recall), blended by \(F_1 = \frac{2PR}{P+R}\).

THE DISTINCTION TO TEACH

\[ \underbrace{\text{token-vector similarity}}_{\text{what BERTScore measures}} \;\;\neq\;\; \underbrace{\text{preserved facts, roles and logic}}_{\text{what translation must guarantee}} \]

The big lesson: BERTScore moves evaluation from strings to meanings-in-context — crediting kid for child, catching what BLEU and chrF structurally cannot. It remains a similarity metric: negations, role reversals and numbers can hide inside a high score. For learned judgment of those, the next step is COMET — trained directly on human ratings.

The original paper

T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger & Y. Artzi (ICLR 2020), “BERTScore: Evaluating Text Generation with BERT” — IDF weighting, baseline rescaling and the correlation tables that made it a standard.

The library

Tiiiger/bert_score — the reference implementation behind pip install bert-score.

Continue the series

COMET → — from similarity to prediction — then MQM error spans, the human standard every metric is chasing.

12 MASTERY CHECK

Eight questions — vectors, matrices, traps

Contextual embeddings, the cosine, greedy matching, the F1 seal — and the five sentences that still get past it.

YOUR SCORE

0 / 8