CAS logo CAS NLP LABInteractive Tutorial Series ← Series

⌗  BERT — BIDIRECTIONAL ENCODER REPRESENTATIONS · Devlin, Chang, Lee & Toutanova (Google AI Language, Oct 2018 · NAACL 2019)

BERT — the encoder, reading both ways

Lectures 12–13 chased the Transformer’s decoder half (GPT), which wears a causal blindfold so it can generate the next token left to right. Four months after GPT-1, Jacob Devlin and colleagues asked the mirror-image question: if your goal is to understand a text that already exists, why wear a blindfold at all? Keep only the Transformer Encoder, delete the causal mask so every word attends to both past and future simultaneously, and pretrain it by playing Cloze (fill-in-the-blank). Every figure, equation, and ablation from the 2018 paper runs live below.

SENTENCE PAIR (A, B)→ WORDPIECE (30,522)→ [CLS] + A + [SEP] + B→ E_tok + E_seg + E_pos→ BIDIRECTIONAL ENCODER ×12 / ×24→ MLM (15%) + NSP ([CLS])→ FINE-TUNE ON 11 TASKS
14SectionsPaper Figures 1–4 + Tables 1–8
80/10/10The [MASK] RuleWhy 15% Cloze doesn’t break fine-tuning
110M / 340MBASE vs LARGEL=12, H=768 vs L=24, H=1024
+10.7 F1SQuAD Bidir Gain88.5 vs 77.8 left-to-right (Table 5)
★ PAPER EXECUTIVE BRIEFING · DEVLIN ET AL. (NAACL 2019)

Top 6 Breakthroughs & Headlines that redefined NLP

Every core contribution of the 2018 BERT paper distilled into six high-impact headlines — filter by theme or click any card to jump straight to its live interactive simulator below.

THE PAPER’S CORE INTUITION IN ONE PARAGRAPH

Standard language models (GPT) can only train unidirectionally because in a multi-layer bidirectional network, each word would trivially see itself through its neighbors. BERT breaks that mirror by corrupting 15% of input tokens before Layer 1 (the Masked Language Model, inspired by Wilson Taylor’s 1953 Cloze task) and pairing it with Next Sentence Prediction (NSP) on [CLS]. One pretrained encoder body then solves classification, entailment, question answering, and named entity recognition with a single thin linear layer on top.

1. PICK A CLOZE SENTENCE (NOTICE THE PREFIX LEFT OF [MASK] IS IDENTICAL)

2. READING DIRECTION

INPUT SEQUENCE — HOVER A TOKEN TO SEE ATTENTION FLOWING INTO [MASK]

PREDICTED DISTRIBUTION FOR [MASK] — P(w | context)

Why did nobody train a deep bidirectional Transformer language model before October 2018? Because removing the causal mask without changing the objective creates a fatal two-hop mirror leak.

02 PAPER FIGURE 1 · THE SEE-YOURSELF PARADOX

Why left-to-right and shallow Bi-LSTMs aren’t enough

Figure 1 of Devlin et al. compares BERT against OpenAI GPT (Radford et al., 2018) and ELMo (Peters et al., 2018). GPT fuses context deeply across 12 layers, but only from the left. ELMo reads both left and right, but trains two separate LSTMs and only concatenates [h→ ; h←] at the very top. And if you try to train a standard multi-layer Transformer without a causal mask, every word cheats by copying itself in 2 hops! Click all four architectures below to see why:

1. SELECT ARCHITECTURE (PAPER FIGURE 1 + THE NAIVE BI-LM TRAP)

DEEP JOINT CONDITIONING

2. CLICK ANY TOKEN TO TRACE ITS RECEPTIVE FIELD ACROSS LAYERS

Sentence-level tasksSentiment (SST-2) can limp along left-to-right because the final token sees the whole review.
Token-level tasks (SQuAD / NER)Whether “bank” starts an answer span or is a B-ORG entity depends critically on the words to its right.
Why ELMo is “shallow”In ELMo, h_t^→ and h_t^← never attend to each other inside Layer 1 or Layer 2 — they only meet at the final linear readout.
BERT’s Cloze CureReplacing the target token with [MASK] at the input deletes the token’s identity from Layer 0, so multi-layer full attention cannot leak it!

With the self-leak solved by [MASK], let’s open the hood of the BERT machine — first its parameter budget (BERT_BASE vs BERT_LARGE), then every wire of its stack.

03 ENCODER-ONLY ARCHITECTURE

Keep the encoder, delete the mask — BERTBASE vs BERTLARGE

Where GPT kept the 2017 Transformer’s Decoder, BERT keeps the 2017 Encoder. Devlin et al. released two model sizes: BERTBASE (L=12, H=768, A=12 → 110M params), engineered to have the exact same footprint as OpenAI GPT-1 for a controlled scientific comparison, and BERTLARGE (L=24, H=1024, A=16 → 340M params).

LIVE PARAMETER AUDITOR — WHERE DO THE 110M / 340M WEIGHTS LIVE?

THE COMPLETE BERT ARCHITECTURE BLOCK DIAGRAM — SCALED & COLOR-CODED BY PURPOSE

Click any quick-focus button below or hover/click any block in the diagram to inspect its tensor shapes, math, and purpose.

3 Zones · Encoder-Only · Full N×N Attention · Dual Heads

Look at the bottom of the stack above: three separate embedding tables meet at a single ⊕ node. Why does BERT need three embeddings per token instead of one?

04 PAPER FIGURE 2 · INPUT REPRESENTATION

One vector, three jobs: Token + Segment + Position

Many downstream NLP tasks — Question Answering, Natural Language Inference, Paraphrase Detection — take a pair of sentences (A, B). Instead of building a two-tower architecture, BERT packs both into one flat token sequence separated by [SEP], headed by [CLS], and sums three learned H-dimensional vectors at every position i:

xᵢ = E_token(wᵢ) + E_segment(sᵢ ∈ {A, B}) + E_position(i ∈ {0 … 511})

1. SELECT SEQUENCE PAIR (INCLUDING PAPER FIGURE 2’S EXACT EXAMPLE)

2. CLICK ANY COLUMN IN THE FIGURE-2 STRIP TO COMPUTE ITS ELEMENT-WISE VECTOR SUM

In Figure 2 above, “playing” entered BERT as two columns: “play” and “##ing”. That is WordPiece — let’s run its greedy splitter and see how its merge rule differs from GPT’s BPE.

05 30,522 SUBWORD VOCABULARY

WordPiece and the “##” continuation contract

BERT uses WordPiece (Wu et al., 2016; Schuster & Nakajima, 2012) with a 30,522-token vocabulary. At tokenization time, WordPiece scans each word from left to right using greedy longest-prefix-first matching (MaxMatch), prefixing any subword that does not start a word with ## so the model knows where spaces were without wasting tokens on whitespace.

PICK A PROBE WORD OR TYPE YOUR OWN (a–z)

EMITTED WORDPIECE TOKENS

STEP-BY-STEP GREEDY LONGEST-PREFIX SEARCH TRACE

Candidate Pair (x + y) BPE Criterion: Raw Count c(xy) Individual Counts c(x) · c(y) WordPiece Criterion: c(xy) / [c(x)·c(y)] Why They Diverge

Now the centerpiece of the 2018 paper: how do you train a deep bidirectional encoder on raw text? Task #1 — the Masked Language Model and its famous 80 / 10 / 10 rule.

06 PRE-TRAINING TASK #1 · MASKED LM

Hide 15% of the tokens — and why 80% [MASK] · 10% Random · 10% Unchanged

In every sequence, BERT chooses 15% of token positions at random as prediction targets M. Only those 15% positions are graded by cross-entropy. But there is a catch: the [MASK] token never appears during downstream fine-tuning! If BERT replaced 100% of target tokens with [MASK], it would learn to build good representations only when it sees [MASK]. Devlin et al.’s solution is the 80 / 10 / 10 corruption rule:

L_MLM = − (1 / |M|) Σ_{i ∈ M} log P(wᵢ | x̃)   ·   For i ∈ M: 80% [MASK], 10% Random Word, 10% Unchanged

1. SELECT SENTENCE

2. OR INSPECT EACH CORRUPTION BRANCH DIRECTLY

STEP A · ORIGINAL SEQUENCE & 15% SELECTION

STEP B · CORRUPTED INPUT x̃ FED TO BERT LAYER 1

STEP C · CROSS-ENTROPY LABELS (85% IGNORED VIA −100)

MLM HEAD OUTPUT AT SELECTED POSITION — P(w | x̃)

PAPER APPENDIX C.2 (TABLE 8) — WHAT HAPPENS IF YOU CHANGE THE 80 / 10 / 10 MIX?

Masked LM teaches token-level representations, but what teaches BERT how two whole sentences relate in Question Answering or Entailment? Task #2 — Next Sentence Prediction.

07 PRE-TRAINING TASK #2 · NEXT SENTENCE PREDICTION

Is Sentence B the real continuation — or a random intruder?

To train sentence-level and cross-sentence reasoning before any labeled data is seen, BERT constructs every pretraining example from two text spans (A, B): 50% of the time B is the actual next span that followed A (IsNext); 50% of the time B is a random span from another document (NotNext). The [CLS] vector C predicts which is which, and BERT minimizes the joint loss L = L_MLM + L_NSP.

PICK A PRETRAINING PAIR (PAPER §3.1 EXAMPLES + THE ALBERT SOP TWIST)

SENTENCE A (SEGMENT E_A)

SENTENCE B (SEGMENT E_B)

CROSS-SENTENCE ATTENTION CLUES POOLED INTO [CLS]

[CLS] NSP HEAD OUTPUT — softmax(W_nsp · tanh(W_pool · C))

IsNext (Continuation) 96.0%
NotNext (Random Intruder) 4.0%
L_total = L_MLM (15% tokens) + L_NSP ([CLS] binary CE)

We’ve seen what enters BERT and what its two heads predict at the top. Now let’s open the N×N attention matrix inside the encoder and see what unrestricted bidirectionality looks like.

08 FULL N×N SELF-ATTENTION

What BERT’s heads look at when the future is unmasked

In GPT (Lecture 13), every row of the attention matrix hits a −∞ wall at the diagonal (j > i). In BERT, both triangles of the N×N matrix are alive: blue cells below the diagonal attend to the past, and teal cells above the diagonal attend to the future! Click any row in the matrix below, or toggle on the causal mask to watch what breaks:

1. SELECT SPECIALIZED HEAD (CLARK ET AL., ACL 2019 “WHAT DOES BERT LOOK AT?”)

CLICK ANY ROW (QUERY TOKEN) — SKY = PAST/SELF (j≤i) · TEAL = FUTURE (j>i)

SELECTED QUERY’S ATTENTION DISTRIBUTION ACROSS THE SENTENCE

Pretraining is done (1M steps on 3.3B words of BooksCorpus + English Wikipedia). How does this single pretrained encoder adapt to 11 completely different downstream tasks?

09 PAPER FIGURE 4 · UNIVERSAL FINE-TUNING

One pretrained encoder body, four thin task heads

Before BERT, every NLP benchmark had its own bespoke neural architecture (BiDAF for QA, ESIM for entailment, BiLSTM-CRF for NER). Figure 4 of the BERT paper swept all of them away: keep the exact same pretrained BERT encoder, attach at most a couple thousand new weights on top, and fine-tune all parameters end-to-end for 2–4 epochs. Explore all four Figure-4 topologies below:

SELECT DOWNSTREAM TASK TOPOLOGY (PAPER FIGURE 4a–4d)

What if you don’t even fine-tune BERT’s weights — and what is each of its 12 layers actually computing under the hood?

10 PAPER TABLE 7 & LAYER PROBING

The Classical NLP Pipeline inside BERT’s 12 layers

Two discoveries revealed how BERT organizes language internally. First, in Paper §5.3 (Table 7), Devlin et al. froze BERT completely and fed its hidden layers into a BiLSTM on CoNLL-2003 NER — proving that concatenating the last 4 layers reaches 96.1 F1 (within 0.3 of full fine-tuning!). Second, Tenney et al. (ACL 2019, “BERT Rediscovers the Classical NLP Pipeline”) probed all 12 layers and found that BERT spontaneously orders its layers from POS → Constituents → Dependencies → Entities → Semantic Roles → Coreference!

DRAG THROUGH BERT_BASE’S 12 ENCODER LAYERS Layer 7 / 12

PROBING SIGNAL STRENGTH AT THIS LAYER (TENNEY ET AL., 2019)

PAPER TABLE 7 · CoNLL-2003 NER (FROZEN FEATURES vs FINE-TUNING)

How much of BERT’s jump came from bidirectionality, how much from NSP, and how much from scale? The paper’s own Tables 1 and 5 give the exact numbers.

11 PAPER TABLES 1 & 5 · THE EMPIRICAL PROOF

Eleven records — and the ablation that settled the debate

Table 5 of Devlin et al. is one of the cleanest ablation studies in deep learning: holding architecture, tokenizer, corpus, and parameter count (110M) fixed, they stripped away NSP and then stripped away bidirectionality (LTR & No NSP). On SQuAD 1.1, dropping bidirectionality crashed F1 by −10.7 points (88.5 → 77.8)!

Every breakthrough leaves new cracks for the next generation of papers to fix. What were BERT’s blind spots, and how did RoBERTa, ALBERT, DistilBERT, and DeBERTa fix them?

12 THE HONEST LEDGER & THE BERT FAMILY TREE

What BERT got wrong — and how RoBERTa, ALBERT & DeBERTa evolved it

Within 18 months of BERT’s release, follow-up papers scrutinized every assumption in Devlin et al. Explore the five-model lineage below, then test BERT’s subtle mathematical flaw — the conditional independence assumption between multiple [MASK] tokens:

CLICK ANY ENCODER IN THE BERT LINEAGE (2018 → 2020)

THE [MASK] INDEPENDENCE FLAW (YANG ET AL. 2019 · XLNet) — “SAN YORK” IN ONE PASS

13 CODE LAB · KERAS, HUGGING FACE & PYTORCH

The BERT paper, in runnable Keras, Hugging Face & PyTorch

Four self-contained, copy-ready implementations: (1) Keras 3 / tf.keras & keras_hub building the Tri-Embedding, Bidirectional Encoder block, and Masked LM head, (2) Hugging Face transformers for Cloze MLM, contextual embeddings, and Trainer fine-tuning, (3) Hugging Face SQuAD Span Extraction & Token NER (Paper Figure 4c–d), and (4) PyTorch from scratch with the exact 80/10/10 masking function.

BERT · KERAS · HUGGING FACE · PYTORCH

14 MASTERY CHECK

Eight questions — one per mechanism of the BERT paper.

YOUR SCORE

0 / 8