⌗ GPT-2 — LANGUAGE MODELS ARE UNSUPERVISED MULTITASK LEARNERS · Radford, Wu, Child, Luan, Amodei & Sutskever (OpenAI, Feb 14 2019)
GPT-2 — the model that learned tasks without being taught
Four months after BERT, OpenAI asked the opposite question. BERT (Lecture 14) changed the objective — mask, fill in, read both ways — to build an understanding engine. GPT-2 changed nothing but scale and data: the same decoder-only block as GPT-1 (Lecture 13), two lines of surgery (pre-LN + 1/√N init), a byte-level tokenizer, and 40 GB of human-curated web text. Train it to predict the next token and it begins to answer questions, translate, and summarize zero-shot — with no parameter or architecture modification. Every table, ablation, and controversy of the 2019 paper runs live below.
▼ EXPLORE GPT-2 — START LESSON 1
Top 6 Breakthroughs & Headlines that started the prompting era
Every core contribution of the GPT-2 paper distilled into six high-impact headlines — filter by theme or click any card to jump straight to its live interactive section below.
PAPER TABLE 13 · THE PROMPT (HUMAN-WRITTEN)
GPT-2’s COMPLETION — THE MOST FAMOUS PARAGRAPH IN NLP HISTORY
Press the button — the completion streams in token by token.
Famous paragraphs are marketing; the paper’s actual claim is sharper: task conditioning costs nothing because it was never architectural in the first place. Let’s read the thesis precisely.
02 PAPER §2 · THE THESIS
p(output | input, task) — the supervised objective hides inside the LM
Learning a single task is estimating p(output | input). A general system must estimate p(output | input, task) — and previous efforts built task conditioning into the architecture (task-specific encoders, MAML inner loops, decaNLP’s MQAN). The paper’s move: language itself specifies the task, so one next-token objective covers them all. Click a task format below, then watch capacity do the rest:
PLAY THE OBJECTIVE — SAMPLE A CONTINUATION, TOKEN BY TOKEN
NEXT-TOKEN CANDIDATES — P(next | everything above)
1. PICK A TASK — THE PAPER’S OWN SEQUENCE FORMATS (§2, MCCANN ET AL. 2018)
2. THE TRAINING SEQUENCE — TASK, INPUT AND OUTPUT ARE ALL JUST TOKENS
3. FIGURE 1’S CLAIM — CAPACITY IS THE ONLY DIAL (ENDPOINTS = PAPER VALUES, LINE = LOG-LINEAR FIT)
Click any dot to inspect a model size.
If tasks live inside ordinary text, the dataset IS the curriculum. Common Crawl was the obvious source — and the paper opens §2.1 by explaining why it refused to use it.
03 PAPER §2.1 · THE DATASET
WebText — 45 million Reddit links, 40 GB of text
Common Crawl is orders of magnitude bigger than any LM dataset, but “it has significant data quality issues” — Trinh & Le (2018) found many documents “mostly unintelligible”. GPT-2’s authors chose curation over scale: scrape only outbound links that humans had already upvoted. Click each stage of the pipeline:
THE WEBTEXT PIPELINE — CLICK A STAGE
PAPER TABLE 1 · “NATURALLY OCCURRING DEMONSTRATIONS” — TRANSLATION PAIRS NOBODY LABELED
This is the thesis made concrete: ordinary English pages are full of French quotes with inline translations — (task, input, output) triples growing wild.
WebText is raw HTML text: Urdu comments, emoji, code, math, 130,000+ Unicode code points. A word-level tokenizer dies here. GPT-2’s answer is the quietest big idea in the paper — byte-level BPE.
04 PAPER §2.2 · INPUT REPRESENTATION
Byte-level BPE — any string ever written, zero UNK
“A general language model should be able to compute the probability of (and also generate) any string.” Word-level models can’t (out-of-vocabulary). Unicode-code-point BPE needs a 130,000+ base vocabulary. Pure byte-level models underperform. GPT-2’s compromise: BPE over UTF-8 bytes — base vocabulary 256 — but with merges forbidden across character categories so dog., dog!, dog? don’t each waste a slot.
1. PICK A TORTURE TEST (OR TYPE ANYTHING — URDU WELCOME)
2. THE CHARACTER-CATEGORY REGEX PRE-SPLIT (encoder.py, VERBATIM)
's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+
3. EACH PIECE AS UTF-8 BYTES, MAPPED BY bytes_to_unicode() (Ġ = THE SPACE BYTE)
THE PAPER’S OWN ARGUMENT, COMPUTED LIVE — NAIVE BPE vs CATEGORY-SPLIT BPE ON A TOY CORPUS (TOY CORPUS · REAL ALGORITHM)
MERGE STEP 0 — LEFT: BPE over whole words (punctuation attached) · RIGHT: BPE per character category, the GPT-2 way
NAIVE VOCABULARY
CATEGORY-SPLIT VOCABULARY (GPT-2 RULE)
THE VOCABULARY ARITHMETIC — WHERE 50,257 COMES FROM
Compare GPT-1: 40,478 code-point BPE tokens with out-of-vocabulary tokens. The byte-level move is why GPT-2 can be evaluated zero-shot on datasets with “aggressively standardized text, tokenization artifacts… and even the string <unk>, which is extremely rare in WebText — occurring only 26 times in 40 billion bytes.”
The tokenizer feeds a stack that looks exactly like GPT-1’s — almost. §2.3 changes exactly two structural things, and they are the reason a 48-layer decoder could be trained at all in 2019.
05 PAPER §2.3 · THE MODEL
Two lines of surgery: pre-LN & the 1/√N init
“The model largely follows the details of the OpenAI GPT model… with a few modifications.” Those modifications, quoted from the paper: LayerNorm moved to the input of each sub-block (like pre-activation ResNets), an additional LayerNorm after the final block, and residual weights initialized 1/√N smaller. Click the six deltas, then break a 48-layer stack yourself:
THE SIX DELTAS — GPT-1 → GPT-2 (§2.3, VERBATIM CLAIMS)
WHAT DID NOT CHANGE — THE CAUSAL MASK, REFRESHED (LECTURE 12’S GRID, VERBATIM MECHANICS)
How much did that recipe weigh? Table 2 gives four sizes on one knob — and the released checkpoints’ parameter counts can be recomputed to the last weight, live.
06 PAPER TABLE 2 · THE FOUR SIZES
117M → 1.5B — where every parameter lives
Four LMs, approximately log-uniformly spaced. The smallest equals GPT-1; the second equals BERT-LARGE; the largest — “GPT-2” proper — has over an order of magnitude more parameters than GPT. Table 2 lists just Layers and d_model — the auditor below recomputes each size’s full parameter count from those two numbers alone:
LIVE PARAMETER AUDITOR — CLICK A SIZE
—
CONTEXT OF THE MOMENT — EARLY 2019 PARAMETER COUNTS
MEMORY EXPLORER — WHAT THE SELECTED SIZE COSTS TO STORE
Four sizes, zero fine-tuning. Time for the scoreboard: Table 3 evaluates all of them zero-shot on eight language-modeling benchmarks — and wins seven.
07 PAPER §3.1 · TABLE 3
Eight benchmarks, zero fine-tuning — seven state-of-the-art records
“Since our model operates on a byte level and does not require lossy pre-processing or tokenization, we can evaluate it on any language model benchmark.” Because these datasets are wildly out-of-distribution (shuffled sentences, PTB artifacts), GPT-2 evaluates with invertible de-tokenizers — a “simple form of domain adaptation” worth 2.5–5 perplexity. Click a dataset:
THE DE-TOKENIZER DETAIL — AND THE 26 TOKENS
Perplexity is the home turf. The paper’s bold move is the away games: can raw next-token prediction play LAMBADA, the Children’s Book Test, and the Winograd Schema Challenge?
08 PAPER §3.2–3.3 · LONG-RANGE GAMES
LAMBADA & CBT — the 50-token-context test
LAMBADA asks for the final word of passages that are unpredictable from the last sentence alone — humans need ≥50 tokens of context. Play a round yourself, then watch capacity climb the curve. Below it: the Children’s Book Test, where GPT-2 scores each of 10 candidates by the LM probability of the sentence with the candidate filled in.
1. PICK A ROUND (LAMBADA-STYLE PASSAGES — FINAL WORD NEEDS THE FULL PASSAGE)
2. PREDICT THE FINAL WORD — WHAT FOLLOWS:
LAMBADA ACCURACY BY MODEL SIZE (PAPER §3.3 — REAL NUMBERS)
CHILDREN’S BOOK TEST — COMMON NOUNS vs NAMED ENTITIES (PAPER FIGURE 2)
One benchmark above all became shorthand for “does this LM reason?” — the Winograd Schema Challenge. GPT-2 answered without a single gradient step of tuning.
09 PAPER §3.4 · WINOGRAD SCHEMA CHALLENGE
Commonsense, zero-shot — 70.70% on the schema test
Following Trinh & Le (2018), GPT-2 resolves the pronoun by comparing the LM probability of the full sentence under each reading. One adjective flip changes the answer — world knowledge, not syntax, decides. Play the resolver (illustrative toy scores), then check the real numbers:
1. PICK A SCHEMA (LEVESQUE ET AL. 2012 STYLE)
2. FLIP THE ADJECTIVE
3. WHO DOES THE PRONOUN REFER TO? CLICK A CANDIDATE ROW
Three for three on benchmarks built for reasoning. Now the honest chapter: reading comprehension and factoid QA — where GPT-2’s limits show, and the paper says so plainly.
10 PAPER §3.5 & §3.8 · READING COMPREHENSION & QA
CoQA 55 F1, Natural Questions 4.1% — the honest results
Condition on a document, the conversation history, and the literal token A: — greedy decoding scores 55 F1 on CoQA, matching or beating 3 of 4 supervised baselines without their 127,000+ training pairs. On Natural Questions: 4.1% exact match — 5.3× a trivial baseline, and wildly below supervised systems. Both facts matter. Replay the paper’s own examples:
THE PAPER’S SELECTED CoQA COMPLETIONS (TABLES 16–17, VERBATIM)
DOCUMENT
CONVERSATION HISTORY — FED BACK INTO THE PROMPT
GPT-2 (GREEDY, AFTER “A:”)
FIVE HUMAN ANNOTATORS SAID
PAPER TABLE 5 · THE 30 MOST CONFIDENT NATURAL QUESTIONS ANSWERS (TOP ROWS)
sorted by GPT-2’s own probabilitySo how do you make a next-token predictor summarize and translate? You ask it — in plain English. §3.6 and §3.7 are the birthplace of prompt engineering.
11 PAPER §3.6–3.7 · SUMMARIZATION & TRANSLATION
“TL;DR:” and “= ” — task behavior, invoked by text
To induce summarization on CNN/Daily Mail, GPT-2 appends the text TL;DR: to the article and samples 100 tokens with top-k = 2. For translation, the context is seeded with pairs in the format english sentence = french sentence. Toggle the hint below and watch the metric move — this tiny switch is the ancestor of every system prompt:
THE TL;DR SWITCH — PAPER TABLE 4 LIVE
PROMPT PREVIEW
A REAL SUMMARIZATION — PAPER TABLE 14, SIDE BY SIDE WITH THE HUMAN REFERENCE
TRANSLATION BY PROMPT FORMAT — WMT-14, ZERO-SHOT (PAPER §3.7)
THE PROMPT FORMAT
PAPER TABLE 15 · REAL GPT-2 TRANSLATIONS, VERBATIM
INTERACTIVE TOOL · IN-CONTEXT LEARNING LAB — ZERO / ONE / FEW-SHOT
illustrative engine · no API · weights untouched1. SHOTS MODE
2. TASK
3. THE ASSEMBLED PROMPT — task cue · examples · query
MODEL OUTPUT
Seven records, one honest defeat, and a famous unicorn. The skeptic’s last weapon: maybe it memorized the test sets. §4 answers with Bloom filters — and the page recomputes its method live.
12 PAPER §4 · GENERALIZATION VS MEMORIZATION
Did it just memorize the test? The Bloom-filter audit
§4 builds Bloom filters of every 8-gram in WebText (lowercased, alphanumeric, single spaces; false-positive rate ≤ 10⁻⁸) and checks every benchmark’s test set. The method is three lines of code — so this page implements it. Edit the test document and watch the overlap recompute:
1. PICK A TEST DOCUMENT (OR EDIT IT — THE CALCULATOR IS REAL)
MATCHED 8-GRAMS
Toy training set = 3 documents from this page; the paper’s filter held all 40 GB of WebText. The algorithm — normalize, slice 8-grams, membership-test — is identical.
PAPER TABLE 6 — OVERLAP RATES: EACH DATASET vs ITS OWN TRAIN SET vs WEBTEXT
The science held up. Then came the part no section number covers: OpenAI judged the model too dangerous to publish — and the nine-month wait became a landmark in AI policy.
13 THE STAGED RELEASE & THE LEGACY · HISTORICAL CONTEXT BEYOND THE PAPER
“Too dangerous to release” — and what it started
On February 14, 2019, OpenAI published the results but withheld the 1.5B model, citing potential misuse. The checkpoints shipped in four waves over nine months, each accompanied by detection research. Click the timeline — then the ledger of what this single paper started:
WHERE GPT-2 SITS — THE EVOLUTION TIMELINE (2017 → 2022)
THE STAGED RELEASE — CLICK A WAVE
Expand — GPT-2 vs modern instruction-tuned LLMs: why instruction tuning and human-preference alignment became necessary
| Dimension | GPT-2 (2019) | Modern instruction-tuned LLMs (2022+) |
|---|---|---|
| Training | Next-token prediction on WebText — nothing else | + supervised instruction tuning, then human-preference alignment (RLHF) |
| How tasks are invoked | Prompt text only — format is fragile (§11’s −6.4 ROUGE with the hint removed) | Trained instruction following — robust across phrasings |
| Factual reliability | Hallucinates freely (Table 5: 4.1% EM, §13’s ledger) | Substantially fewer errors — still imperfect |
| Output control | Sampling knobs (temperature, top-k) — no refusal or steering | Safety training, refusals, system prompts |
| Context window | 1,024 tokens | 10⁵–10⁶+ tokens |
The gap this table shows is exactly why supervised instruction tuning and human-preference alignment became the next decade’s central engineering problem — GPT-2 provided the raw capability, alignment made it usable.
14 CODE LAB · TENSORFLOW 1.12, PYTORCH & HUGGING FACE
The GPT-2 paper, in runnable code
Four self-contained tabs: (1) the TensorFlow 1.12 structure of the original openai/gpt-2 release — pre-LN block, final norm, tied LM head, the 1/√N initializer; (2) a PyTorch from-scratch GPT-2 with only the paper’s six deltas; (3) Hugging Face running the paper’s four zero-shot suites verbatim; (4) the byte-level BPE encoder — bytes_to_unicode(), the category regex, and the merge loop.
15 MASTERY CHECK