⌗ BERT — BIDIRECTIONAL ENCODER REPRESENTATIONS · Devlin, Chang, Lee & Toutanova (Google AI Language, Oct 2018 · NAACL 2019)
BERT — the encoder, reading both ways
Lectures 12–13 chased the Transformer’s decoder half (GPT), which wears a causal blindfold so it can generate the next token left to right. Four months after GPT-1, Jacob Devlin and colleagues asked the mirror-image question: if your goal is to understand a text that already exists, why wear a blindfold at all? Keep only the Transformer Encoder, delete the causal mask so every word attends to both past and future simultaneously, and pretrain it by playing Cloze (fill-in-the-blank). Every figure, equation, and ablation from the 2018 paper runs live below.
Top 6 Breakthroughs & Headlines that redefined NLP
Every core contribution of the 2018 BERT paper distilled into six high-impact headlines — filter by theme or click any card to jump straight to its live interactive simulator below.
1. PICK A CLOZE SENTENCE (NOTICE THE PREFIX LEFT OF [MASK] IS IDENTICAL)
2. READING DIRECTION
INPUT SEQUENCE — HOVER A TOKEN TO SEE ATTENTION FLOWING INTO [MASK]
PREDICTED DISTRIBUTION FOR [MASK] — P(w | context)
Why did nobody train a deep bidirectional Transformer language model before October 2018? Because removing the causal mask without changing the objective creates a fatal two-hop mirror leak.
02 PAPER FIGURE 1 · THE SEE-YOURSELF PARADOX
Why left-to-right and shallow Bi-LSTMs aren’t enough
Figure 1 of Devlin et al. compares BERT against OpenAI GPT (Radford et al., 2018) and ELMo (Peters et al., 2018). GPT fuses context deeply across 12 layers, but only from the left. ELMo reads both left and right, but trains two separate LSTMs and only concatenates [h→ ; h←] at the very top. And if you try to train a standard multi-layer Transformer without a causal mask, every word cheats by copying itself in 2 hops! Click all four architectures below to see why:
1. SELECT ARCHITECTURE (PAPER FIGURE 1 + THE NAIVE BI-LM TRAP)
2. CLICK ANY TOKEN TO TRACE ITS RECEPTIVE FIELD ACROSS LAYERS
With the self-leak solved by [MASK], let’s open the hood of the BERT machine — first its parameter budget (BERT_BASE vs BERT_LARGE), then every wire of its stack.
03 ENCODER-ONLY ARCHITECTURE
Keep the encoder, delete the mask — BERTBASE vs BERTLARGE
Where GPT kept the 2017 Transformer’s Decoder, BERT keeps the 2017 Encoder. Devlin et al. released two model sizes: BERTBASE (L=12, H=768, A=12 → 110M params), engineered to have the exact same footprint as OpenAI GPT-1 for a controlled scientific comparison, and BERTLARGE (L=24, H=1024, A=16 → 340M params).
LIVE PARAMETER AUDITOR — WHERE DO THE 110M / 340M WEIGHTS LIVE?
Look at the bottom of the stack above: three separate embedding tables meet at a single ⊕ node. Why does BERT need three embeddings per token instead of one?
04 PAPER FIGURE 2 · INPUT REPRESENTATION
One vector, three jobs: Token + Segment + Position
Many downstream NLP tasks — Question Answering, Natural Language Inference, Paraphrase Detection — take a pair of sentences (A, B). Instead of building a two-tower architecture, BERT packs both into one flat token sequence separated by [SEP], headed by [CLS], and sums three learned H-dimensional vectors at every position i:
1. SELECT SEQUENCE PAIR (INCLUDING PAPER FIGURE 2’S EXACT EXAMPLE)
2. CLICK ANY COLUMN IN THE FIGURE-2 STRIP TO COMPUTE ITS ELEMENT-WISE VECTOR SUM
In Figure 2 above, “playing” entered BERT as two columns: “play” and “##ing”. That is WordPiece — let’s run its greedy splitter and see how its merge rule differs from GPT’s BPE.
05 30,522 SUBWORD VOCABULARY
WordPiece and the “##” continuation contract
BERT uses WordPiece (Wu et al., 2016; Schuster & Nakajima, 2012) with a 30,522-token vocabulary. At tokenization time, WordPiece scans each word from left to right using greedy longest-prefix-first matching (MaxMatch), prefixing any subword that does not start a word with ## so the model knows where spaces were without wasting tokens on whitespace.
PICK A PROBE WORD OR TYPE YOUR OWN (a–z)
EMITTED WORDPIECE TOKENS
STEP-BY-STEP GREEDY LONGEST-PREFIX SEARCH TRACE
| Candidate Pair (x + y) | BPE Criterion: Raw Count c(xy) | Individual Counts c(x) · c(y) | WordPiece Criterion: c(xy) / [c(x)·c(y)] | Why They Diverge |
|---|
Now the centerpiece of the 2018 paper: how do you train a deep bidirectional encoder on raw text? Task #1 — the Masked Language Model and its famous 80 / 10 / 10 rule.
06 PRE-TRAINING TASK #1 · MASKED LM
Hide 15% of the tokens — and why 80% [MASK] · 10% Random · 10% Unchanged
In every sequence, BERT chooses 15% of token positions at random as prediction targets M. Only those 15% positions are graded by cross-entropy. But there is a catch: the [MASK] token never appears during downstream fine-tuning! If BERT replaced 100% of target tokens with [MASK], it would learn to build good representations only when it sees [MASK]. Devlin et al.’s solution is the 80 / 10 / 10 corruption rule:
1. SELECT SENTENCE
2. OR INSPECT EACH CORRUPTION BRANCH DIRECTLY
STEP A · ORIGINAL SEQUENCE & 15% SELECTION
STEP B · CORRUPTED INPUT x̃ FED TO BERT LAYER 1
STEP C · CROSS-ENTROPY LABELS (85% IGNORED VIA −100)
MLM HEAD OUTPUT AT SELECTED POSITION — P(w | x̃)
PAPER APPENDIX C.2 (TABLE 8) — WHAT HAPPENS IF YOU CHANGE THE 80 / 10 / 10 MIX?
Masked LM teaches token-level representations, but what teaches BERT how two whole sentences relate in Question Answering or Entailment? Task #2 — Next Sentence Prediction.
07 PRE-TRAINING TASK #2 · NEXT SENTENCE PREDICTION
Is Sentence B the real continuation — or a random intruder?
To train sentence-level and cross-sentence reasoning before any labeled data is seen, BERT constructs every pretraining example from two text spans (A, B): 50% of the time B is the actual next span that followed A (IsNext); 50% of the time B is a random span from another document (NotNext). The [CLS] vector C predicts which is which, and BERT minimizes the joint loss L = L_MLM + L_NSP.
PICK A PRETRAINING PAIR (PAPER §3.1 EXAMPLES + THE ALBERT SOP TWIST)
SENTENCE A (SEGMENT E_A)
SENTENCE B (SEGMENT E_B)
CROSS-SENTENCE ATTENTION CLUES POOLED INTO [CLS]
[CLS] NSP HEAD OUTPUT — softmax(W_nsp · tanh(W_pool · C))
We’ve seen what enters BERT and what its two heads predict at the top. Now let’s open the N×N attention matrix inside the encoder and see what unrestricted bidirectionality looks like.
08 FULL N×N SELF-ATTENTION
What BERT’s heads look at when the future is unmasked
In GPT (Lecture 13), every row of the attention matrix hits a −∞ wall at the diagonal (j > i). In BERT, both triangles of the N×N matrix are alive: blue cells below the diagonal attend to the past, and teal cells above the diagonal attend to the future! Click any row in the matrix below, or toggle on the causal mask to watch what breaks:
1. SELECT SPECIALIZED HEAD (CLARK ET AL., ACL 2019 “WHAT DOES BERT LOOK AT?”)
CLICK ANY ROW (QUERY TOKEN) — SKY = PAST/SELF (j≤i) · TEAL = FUTURE (j>i)
SELECTED QUERY’S ATTENTION DISTRIBUTION ACROSS THE SENTENCE
Pretraining is done (1M steps on 3.3B words of BooksCorpus + English Wikipedia). How does this single pretrained encoder adapt to 11 completely different downstream tasks?
09 PAPER FIGURE 4 · UNIVERSAL FINE-TUNING
One pretrained encoder body, four thin task heads
Before BERT, every NLP benchmark had its own bespoke neural architecture (BiDAF for QA, ESIM for entailment, BiLSTM-CRF for NER). Figure 4 of the BERT paper swept all of them away: keep the exact same pretrained BERT encoder, attach at most a couple thousand new weights on top, and fine-tune all parameters end-to-end for 2–4 epochs. Explore all four Figure-4 topologies below:
SELECT DOWNSTREAM TASK TOPOLOGY (PAPER FIGURE 4a–4d)
What if you don’t even fine-tune BERT’s weights — and what is each of its 12 layers actually computing under the hood?
10 PAPER TABLE 7 & LAYER PROBING
The Classical NLP Pipeline inside BERT’s 12 layers
Two discoveries revealed how BERT organizes language internally. First, in Paper §5.3 (Table 7), Devlin et al. froze BERT completely and fed its hidden layers into a BiLSTM on CoNLL-2003 NER — proving that concatenating the last 4 layers reaches 96.1 F1 (within 0.3 of full fine-tuning!). Second, Tenney et al. (ACL 2019, “BERT Rediscovers the Classical NLP Pipeline”) probed all 12 layers and found that BERT spontaneously orders its layers from POS → Constituents → Dependencies → Entities → Semantic Roles → Coreference!
PROBING SIGNAL STRENGTH AT THIS LAYER (TENNEY ET AL., 2019)
PAPER TABLE 7 · CoNLL-2003 NER (FROZEN FEATURES vs FINE-TUNING)
How much of BERT’s jump came from bidirectionality, how much from NSP, and how much from scale? The paper’s own Tables 1 and 5 give the exact numbers.
11 PAPER TABLES 1 & 5 · THE EMPIRICAL PROOF
Eleven records — and the ablation that settled the debate
Table 5 of Devlin et al. is one of the cleanest ablation studies in deep learning: holding architecture, tokenizer, corpus, and parameter count (110M) fixed, they stripped away NSP and then stripped away bidirectionality (LTR & No NSP). On SQuAD 1.1, dropping bidirectionality crashed F1 by −10.7 points (88.5 → 77.8)!
Every breakthrough leaves new cracks for the next generation of papers to fix. What were BERT’s blind spots, and how did RoBERTa, ALBERT, DistilBERT, and DeBERTa fix them?
12 THE HONEST LEDGER & THE BERT FAMILY TREE
What BERT got wrong — and how RoBERTa, ALBERT & DeBERTa evolved it
Within 18 months of BERT’s release, follow-up papers scrutinized every assumption in Devlin et al. Explore the five-model lineage below, then test BERT’s subtle mathematical flaw — the conditional independence assumption between multiple [MASK] tokens:
CLICK ANY ENCODER IN THE BERT LINEAGE (2018 → 2020)
THE [MASK] INDEPENDENCE FLAW (YANG ET AL. 2019 · XLNet) — “SAN YORK” IN ONE PASS
13 CODE LAB · KERAS, HUGGING FACE & PYTORCH
The BERT paper, in runnable Keras, Hugging Face & PyTorch
Four self-contained, copy-ready implementations: (1) Keras 3 / tf.keras & keras_hub building the Tri-Embedding, Bidirectional Encoder block, and Masked LM head, (2) Hugging Face transformers for Cloze MLM, contextual embeddings, and Trainer fine-tuning, (3) Hugging Face SQuAD Span Extraction & Token NER (Paper Figure 4c–d), and (4) PyTorch from scratch with the exact 80/10/10 masking function.
14 MASTERY CHECK