CAS logo CAS NLP LABInteractive Tutorial Series ← Series

⌗  GPT-2 — LANGUAGE MODELS ARE UNSUPERVISED MULTITASK LEARNERS · Radford, Wu, Child, Luan, Amodei & Sutskever (OpenAI, Feb 14 2019)

GPT-2 — the model that learned tasks without being taught

Four months after BERT, OpenAI asked the opposite question. BERT (Lecture 14) changed the objective — mask, fill in, read both ways — to build an understanding engine. GPT-2 changed nothing but scale and data: the same decoder-only block as GPT-1 (Lecture 13), two lines of surgery (pre-LN + 1/√N init), a byte-level tokenizer, and 40 GB of human-curated web text. Train it to predict the next token and it begins to answer questions, translate, and summarize zero-shot — with no parameter or architecture modification. Every table, ablation, and controversy of the 2019 paper runs live below.

REDDIT LINKS (≥3 KARMA)→ 45M PAGES → 40 GB→ BYTE-LEVEL BPE (50,257)→ wte + wpe→ PRE-LN DECODER ×12 … ×48→ NEXT-TOKEN CROSS-ENTROPY→ ZERO-SHOT ON 8+ TASKS
15SectionsPaper §1–8 + Appendix A
1.5BParameters48 layers × d1600 — ≈12.8× GPT-1
7 / 8LM Benchmarkszero-shot SOTA (Table 3)
40 GBWebText8M docs from 45M Reddit links
LEARNING OBJECTIVES 1 GPT-1 → GPT-2 evolution 2 Unsupervised multitask learning 3 Zero / one / few-shot 4 Task conditioning 5 Scaling & capacity 6 Why GPT-2 mattered

▼ EXPLORE GPT-2 — START LESSON 1

★ PAPER EXECUTIVE BRIEFING · RADFORD ET AL. (OPENAI 2019)

Top 6 Breakthroughs & Headlines that started the prompting era

Every core contribution of the GPT-2 paper distilled into six high-impact headlines — filter by theme or click any card to jump straight to its live interactive section below.

THE PAPER’S CORE INTUITION IN ONE PARAGRAPH

Supervised NLP needs a labeled dataset per task. But a language model learns p(output | input, task) for free: task, input and output are all just tokens in one sequence — (translate to french, english text, french text) — and the supervised objective is the LM objective evaluated on the output span. So a large enough LM trained on a diverse enough corpus begins to infer and perform the tasks demonstrated in its training text, unprompted. The paper tests this with zero-shot evaluations — and to make the demo famous, it asked GPT-2 to continue a sentence about unicorns that speak English.

PAPER TABLE 13 · THE PROMPT (HUMAN-WRITTEN)

GPT-2’s COMPLETION — THE MOST FAMOUS PARAGRAPH IN NLP HISTORY

Press the button — the completion streams in token by token.

Famous paragraphs are marketing; the paper’s actual claim is sharper: task conditioning costs nothing because it was never architectural in the first place. Let’s read the thesis precisely.

02 PAPER §2 · THE THESIS

p(output | input, task) — the supervised objective hides inside the LM

Learning a single task is estimating p(output | input). A general system must estimate p(output | input, task) — and previous efforts built task conditioning into the architecture (task-specific encoders, MAML inner loops, decaNLP’s MQAN). The paper’s move: language itself specifies the task, so one next-token objective covers them all. Click a task format below, then watch capacity do the rest:

p(x) = ∏ᵢ p(sᵢ | s₁ … sᵢ₋₁)   ·   supervised loss = the same ∏, restricted to the output span

PLAY THE OBJECTIVE — SAMPLE A CONTINUATION, TOKEN BY TOKEN

NEXT-TOKEN CANDIDATES — P(next | everything above)

1. PICK A TASK — THE PAPER’S OWN SEQUENCE FORMATS (§2, MCCANN ET AL. 2018)

2. THE TRAINING SEQUENCE — TASK, INPUT AND OUTPUT ARE ALL JUST TOKENS

3. FIGURE 1’S CLAIM — CAPACITY IS THE ONLY DIAL (ENDPOINTS = PAPER VALUES, LINE = LOG-LINEAR FIT)

Click any dot to inspect a model size.

Why the “toy” framing mattersThe supervised objective’s global minimum IS the unsupervised objective’s global minimum — no new theory needed, only enough capacity and data to actually reach it.
Learning is slowerPreliminary experiments confirmed large LMs can multitask this way, “but learning is much slower than in explicitly supervised approaches.”
Zero-shot protocol“Without any parameter or architecture modification” — the model is never fine-tuned; tasks are invoked only by the text placed in the context.
The stakesdecaNLP needed 10 curated (dataset, objective) pairs. WebText contains millions of naturally occurring demonstrations — nobody had to label them.

If tasks live inside ordinary text, the dataset IS the curriculum. Common Crawl was the obvious source — and the paper opens §2.1 by explaining why it refused to use it.

03 PAPER §2.1 · THE DATASET

WebText — 45 million Reddit links, 40 GB of text

Common Crawl is orders of magnitude bigger than any LM dataset, but “it has significant data quality issues” — Trinh & Le (2018) found many documents “mostly unintelligible”. GPT-2’s authors chose curation over scale: scrape only outbound links that humans had already upvoted. Click each stage of the pipeline:

THE WEBTEXT PIPELINE — CLICK A STAGE

PAPER TABLE 1 · “NATURALLY OCCURRING DEMONSTRATIONS” — TRANSLATION PAIRS NOBODY LABELED

This is the thesis made concrete: ordinary English pages are full of French quotes with inline translations — (task, input, output) triples growing wild.

The karma filterReddit outbound links with ≥3 karma: “a heuristic indicator for whether other users found the link interesting, educational, or just funny.” Human curation at zero labeling cost.
Wikipedia deleted on purposeWikipedia powers too many test sets (enwik8, text8, WikiText-*) — keeping it would contaminate the zero-shot evaluation and the §4 overlap analysis.
Diversity is the point“Our approach motivates building as large and diverse a dataset as possible in order to collect natural language demonstrations of tasks in as varied of domains and contexts as possible.”
Era noteAll results use the preliminary WebText: links posted before Dec 2017, extracted with Dragnet + newspaper, deduplicated and heuristically cleaned → 8,013,769 docs / ~40 GB.

WebText is raw HTML text: Urdu comments, emoji, code, math, 130,000+ Unicode code points. A word-level tokenizer dies here. GPT-2’s answer is the quietest big idea in the paper — byte-level BPE.

04 PAPER §2.2 · INPUT REPRESENTATION

Byte-level BPE — any string ever written, zero UNK

“A general language model should be able to compute the probability of (and also generate) any string.” Word-level models can’t (out-of-vocabulary). Unicode-code-point BPE needs a 130,000+ base vocabulary. Pure byte-level models underperform. GPT-2’s compromise: BPE over UTF-8 bytes — base vocabulary 256 — but with merges forbidden across character categories so dog., dog!, dog? don’t each waste a slot.

1. PICK A TORTURE TEST (OR TYPE ANYTHING — URDU WELCOME)

2. THE CHARACTER-CATEGORY REGEX PRE-SPLIT (encoder.py, VERBATIM)

's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+

3. EACH PIECE AS UTF-8 BYTES, MAPPED BY bytes_to_unicode() (Ġ = THE SPACE BYTE)

THE PAPER’S OWN ARGUMENT, COMPUTED LIVE — NAIVE BPE vs CATEGORY-SPLIT BPE ON A TOY CORPUS (TOY CORPUS · REAL ALGORITHM)

MERGE STEP 0 — LEFT: BPE over whole words (punctuation attached) · RIGHT: BPE per character category, the GPT-2 way

NAIVE VOCABULARY

CATEGORY-SPLIT VOCABULARY (GPT-2 RULE)

THE VOCABULARY ARITHMETIC — WHERE 50,257 COMES FROM

Compare GPT-1: 40,478 code-point BPE tokens with out-of-vocabulary tokens. The byte-level move is why GPT-2 can be evaluated zero-shot on datasets with “aggressively standardized text, tokenization artifacts… and even the string <unk>, which is extremely rare in WebText — occurring only 26 times in 40 billion bytes.”

The tokenizer feeds a stack that looks exactly like GPT-1’s — almost. §2.3 changes exactly two structural things, and they are the reason a 48-layer decoder could be trained at all in 2019.

05 PAPER §2.3 · THE MODEL

Two lines of surgery: pre-LN & the 1/√N init

“The model largely follows the details of the OpenAI GPT model… with a few modifications.” Those modifications, quoted from the paper: LayerNorm moved to the input of each sub-block (like pre-activation ResNets), an additional LayerNorm after the final block, and residual weights initialized 1/√N smaller. Click the six deltas, then break a 48-layer stack yourself:

THE SIX DELTAS — GPT-1 → GPT-2 (§2.3, VERBATIM CLAIMS)

RESIDUAL-ACCUMULATION LAB — DRAG THE DEPTH 12 residual layers

WHAT DID NOT CHANGE — THE CAUSAL MASK, REFRESHED (LECTURE 12’S GRID, VERBATIM MECHANICS)

Why pre-LN mattersPost-LN normalizes after every residual add, so gradients must fight through a normalization at every depth. Pre-LN leaves the residual path a clean identity highway — gradients reach layer 1 unchanged.
Why the final LNThe clean highway is also unnormalized — activations grow with depth. One last LayerNorm before the LM head restores scale. GPT-1 had no final norm.
Why 1/√NN unit-variance branch outputs accumulate to √N·σ. Scaling each branch by 1/√N keeps the stream at ≈√2·σ for any depth — “accounts for the accumulation on the residual path with model depth.”
What did NOT changeThe attention itself, GELU MLPs, and the objective are untouched. GPT-2 is GPT-1’s recipe taken seriously: scale parameters, scale data, keep the objective.

How much did that recipe weigh? Table 2 gives four sizes on one knob — and the released checkpoints’ parameter counts can be recomputed to the last weight, live.

06 PAPER TABLE 2 · THE FOUR SIZES

117M → 1.5B — where every parameter lives

Four LMs, approximately log-uniformly spaced. The smallest equals GPT-1; the second equals BERT-LARGE; the largest — “GPT-2” proper — has over an order of magnitude more parameters than GPT. Table 2 lists just Layers and d_model — the auditor below recomputes each size’s full parameter count from those two numbers alone:

LIVE PARAMETER AUDITOR — CLICK A SIZE

—

CONTEXT OF THE MOMENT — EARLY 2019 PARAMETER COUNTS

MEMORY EXPLORER — WHAT THE SELECTED SIZE COSTS TO STORE

Heads come freeEvery size uses d_model / 64 attention heads: 12, 16, 20, 25. One hyperparameter (width) buys the head count — 64 dims per head, inherited from the 2017 Transformer.
Trained to underfit“All models still underfit WebText and held-out perplexity has as of yet improved given more training time.” The learning rate was manually tuned per size on a 5% held-out sample.
One context for allEvery size runs 1,024-token contexts at batch 512 — 512 × 1024 tokens per step, doubled from GPT-1’s 512-token window.
117M vs 124MPaper Table 2 says 117M (the GPT-1-equivalent count); the released checkpoints are named 124M / 355M / 774M / 1.5B. Same models — the names differ in how the embedding matrix is counted. The auditor computes the release names’ convention.

Four sizes, zero fine-tuning. Time for the scoreboard: Table 3 evaluates all of them zero-shot on eight language-modeling benchmarks — and wins seven.

07 PAPER §3.1 · TABLE 3

Eight benchmarks, zero fine-tuning — seven state-of-the-art records

“Since our model operates on a byte level and does not require lossy pre-processing or tokenization, we can evaluate it on any language model benchmark.” Because these datasets are wildly out-of-distribution (shuffled sentences, PTB artifacts), GPT-2 evaluates with invertible de-tokenizers — a “simple form of domain adaptation” worth 2.5–5 perplexity. Click a dataset:

ZERO-SHOT SOTA: —

THE ONE DEFEAT — AND WHY IT’S INFORMATIVE

The One Billion Word Benchmark is the only loss: GPT-2’s 42.16 perplexity trails the 21.8 SOTA. But 1BW shuffles sentences within documents — destroying precisely the long-range structure that lets big LMs win everywhere else. GPT-2 loses exactly where there is nothing left to be good at: “1BW’s sentence level shuffling removes all long-range structure.” The defeat confirms the thesis instead of refuting it.

THE DE-TOKENIZER DETAIL — AND THE 26 TOKENS

Perplexity is the home turf. The paper’s bold move is the away games: can raw next-token prediction play LAMBADA, the Children’s Book Test, and the Winograd Schema Challenge?

08 PAPER §3.2–3.3 · LONG-RANGE GAMES

LAMBADA & CBT — the 50-token-context test

LAMBADA asks for the final word of passages that are unpredictable from the last sentence alone — humans need ≥50 tokens of context. Play a round yourself, then watch capacity climb the curve. Below it: the Children’s Book Test, where GPT-2 scores each of 10 candidates by the LM probability of the sentence with the candidate filled in.

1. PICK A ROUND (LAMBADA-STYLE PASSAGES — FINAL WORD NEEDS THE FULL PASSAGE)

2. PREDICT THE FINAL WORD — WHAT FOLLOWS:

LAMBADA ACCURACY BY MODEL SIZE (PAPER §3.3 — REAL NUMBERS)

CHILDREN’S BOOK TEST — COMMON NOUNS vs NAMED ENTITIES (PAPER FIGURE 2)

One benchmark above all became shorthand for “does this LM reason?” — the Winograd Schema Challenge. GPT-2 answered without a single gradient step of tuning.

09 PAPER §3.4 · WINOGRAD SCHEMA CHALLENGE

Commonsense, zero-shot — 70.70% on the schema test

Following Trinh & Le (2018), GPT-2 resolves the pronoun by comparing the LM probability of the full sentence under each reading. One adjective flip changes the answer — world knowledge, not syntax, decides. Play the resolver (illustrative toy scores), then check the real numbers:

1. PICK A SCHEMA (LEVESQUE ET AL. 2012 STYLE)

2. FLIP THE ADJECTIVE

3. WHO DOES THE PRONOUN REFER TO? CLICK A CANDIDATE ROW

63.7% → 70.70%GPT-2 beats the previous SOTA by 7 points using only LM probability — no pronoun-resolution training data, no coreference model.
Only 273 examples“The dataset is quite small with only 273 examples so we recommend reading Trichelair et al. (2018) to help contextualize this result.” The paper cites its own caveats.
Partial vs full scoringFigure 3 plots both: partial scores only the modified span; full scores the whole sentence. Full scoring wins — context disciplines the guess.
Contamination check (§4)Only 10 of its schemata overlap WebText by 8-grams; 2 spurious; and of the remaining 8, only 1 appeared in a context that gave the answer away.

Three for three on benchmarks built for reasoning. Now the honest chapter: reading comprehension and factoid QA — where GPT-2’s limits show, and the paper says so plainly.

10 PAPER §3.5 & §3.8 · READING COMPREHENSION & QA

CoQA 55 F1, Natural Questions 4.1% — the honest results

Condition on a document, the conversation history, and the literal token A: — greedy decoding scores 55 F1 on CoQA, matching or beating 3 of 4 supervised baselines without their 127,000+ training pairs. On Natural Questions: 4.1% exact match — 5.3× a trivial baseline, and wildly below supervised systems. Both facts matter. Replay the paper’s own examples:

THE PAPER’S SELECTED CoQA COMPLETIONS (TABLES 16–17, VERBATIM)

DOCUMENT

CONVERSATION HISTORY — FED BACK INTO THE PROMPT

GPT-2 (GREEDY, AFTER “A:”)

FIVE HUMAN ANNOTATORS SAID

PAPER TABLE 5 · THE 30 MOST CONFIDENT NATURAL QUESTIONS ANSWERS (TOP ROWS)

sorted by GPT-2’s own probability
READ THE LIMITS LIKE A RESEARCHER

The paper itself flags the trick: “inspection of its answers and errors suggests GPT-2 often uses simple retrieval based heuristics such as answer with a name from the document in response to a who question.” Zero-shot competence here is real but shallow — and the supervised BERT-based system (89 F1, human level) still laps it. Capacity, not craftsmanship, was the missing ingredient: “model capacity has been a major factor in the poor performance of neural systems on this kind of task.”

So how do you make a next-token predictor summarize and translate? You ask it — in plain English. §3.6 and §3.7 are the birthplace of prompt engineering.

11 PAPER §3.6–3.7 · SUMMARIZATION & TRANSLATION

“TL;DR:” and “= ” — task behavior, invoked by text

To induce summarization on CNN/Daily Mail, GPT-2 appends the text TL;DR: to the article and samples 100 tokens with top-k = 2. For translation, the context is seeded with pairs in the format english sentence = french sentence. Toggle the hint below and watch the metric move — this tiny switch is the ancestor of every system prompt:

THE TL;DR SWITCH — PAPER TABLE 4 LIVE

PROMPT PREVIEW

A REAL SUMMARIZATION — PAPER TABLE 14, SIDE BY SIDE WITH THE HUMAN REFERENCE

TRANSLATION BY PROMPT FORMAT — WMT-14, ZERO-SHOT (PAPER §3.7)

THE PROMPT FORMAT

PAPER TABLE 15 · REAL GPT-2 TRANSLATIONS, VERBATIM

INTERACTIVE TOOL · IN-CONTEXT LEARNING LAB — ZERO / ONE / FEW-SHOT

illustrative engine · no API · weights untouched

1. SHOTS MODE

2. TASK

3. THE ASSEMBLED PROMPT — task cue · examples · query

MODEL OUTPUT

Seven records, one honest defeat, and a famous unicorn. The skeptic’s last weapon: maybe it memorized the test sets. §4 answers with Bloom filters — and the page recomputes its method live.

12 PAPER §4 · GENERALIZATION VS MEMORIZATION

Did it just memorize the test? The Bloom-filter audit

§4 builds Bloom filters of every 8-gram in WebText (lowercased, alphanumeric, single spaces; false-positive rate ≤ 10⁻⁸) and checks every benchmark’s test set. The method is three lines of code — so this page implements it. Edit the test document and watch the overlap recompute:

1. PICK A TEST DOCUMENT (OR EDIT IT — THE CALCULATOR IS REAL)

MATCHED 8-GRAMS

Toy training set = 3 documents from this page; the paper’s filter held all 40 GB of WebText. The algorithm — normalize, slice 8-grams, membership-test — is identical.

PAPER TABLE 6 — OVERLAP RATES: EACH DATASET vs ITS OWN TRAIN SET vs WEBTEXT

The science held up. Then came the part no section number covers: OpenAI judged the model too dangerous to publish — and the nine-month wait became a landmark in AI policy.

13 THE STAGED RELEASE & THE LEGACY · HISTORICAL CONTEXT BEYOND THE PAPER

“Too dangerous to release” — and what it started

On February 14, 2019, OpenAI published the results but withheld the 1.5B model, citing potential misuse. The checkpoints shipped in four waves over nine months, each accompanied by detection research. Click the timeline — then the ledger of what this single paper started:

WHERE GPT-2 SITS — THE EVOLUTION TIMELINE (2017 → 2022)

THE STAGED RELEASE — CLICK A WAVE

Expand — GPT-2 vs modern instruction-tuned LLMs: why instruction tuning and human-preference alignment became necessary
DimensionGPT-2 (2019)Modern instruction-tuned LLMs (2022+)
TrainingNext-token prediction on WebText — nothing else+ supervised instruction tuning, then human-preference alignment (RLHF)
How tasks are invokedPrompt text only — format is fragile (§11’s −6.4 ROUGE with the hint removed)Trained instruction following — robust across phrasings
Factual reliabilityHallucinates freely (Table 5: 4.1% EM, §13’s ledger)Substantially fewer errors — still imperfect
Output controlSampling knobs (temperature, top-k) — no refusal or steeringSafety training, refusals, system prompts
Context window1,024 tokens10⁵–10⁶+ tokens

The gap this table shows is exactly why supervised instruction tuning and human-preference alignment became the next decade’s central engineering problem — GPT-2 provided the raw capability, alignment made it usable.

THE HONEST LEDGER — WHAT THE PAPER ITSELF ADMITS

Zero-shot is “still far from use-able” in practice; GPT-2 “still underfits WebText”; there are “undoubtedly many practical tasks where the performance of GPT-2 is still no better than random”; and the paper wonders whether its capacity can “overcome the inefficiencies of uni-directional representations demonstrated by BERT.” The next three years answered that last worry — but the paper’s honesty is part of why it convinced anyone.

14 CODE LAB · TENSORFLOW 1.12, PYTORCH & HUGGING FACE

The GPT-2 paper, in runnable code

Four self-contained tabs: (1) the TensorFlow 1.12 structure of the original openai/gpt-2 release — pre-LN block, final norm, tied LM head, the 1/√N initializer; (2) a PyTorch from-scratch GPT-2 with only the paper’s six deltas; (3) Hugging Face running the paper’s four zero-shot suites verbatim; (4) the byte-level BPE encoder — bytes_to_unicode(), the category regex, and the merge loop.

GPT-2 · RADFORD ET AL. 2019

15 MASTERY CHECK

Ten questions — one per contribution, plus the in-context learning ideas it pointed toward.

YOUR SCORE

0 / 8