CAS logo CAS NLP LABInteractive Tutorial Series ← Series

⌗  LECTURE 12 · DECODER-ONLY MODELS — GPT & AUTOREGRESSIVE LANGUAGE MODELING

The Decoder, Alone

Lecture 11 ended with three descendants of the Transformer. This lecture builds the one that became the large language model — from the objective up, not the architecture down. What does it mean to model language? One product of conditional probabilities. How do you learn it? By counting, then by a network you will train in this browser. What architecture computes it? The 2017 decoder with its encoder deleted — masked attention all the way down. By §08 you will have watched every mechanism of GPT run live, at doll scale.

⟨s⟩ CONTEXT→ EMBEDDINGS→ CAUSAL ATTENTION × 12→ LM HEAD→ SOFTMAX→ SAMPLE→ APPEND ↺
12Sectionsobjective → architecture → loop
11Live Demosevery probability computed on the spot
1,159Parametersa language model trained live, in §04
8Quiz Checksthe mastery gate
THE ONE FORMULA THIS WHOLE LECTURE OBEYS

P(w₁…wₙ) = ∏ₜ P(wₜ | w₁…wₜ₋₁)

The probability of any sentence is a chain of next-token guesses — P(w₁)·P(w₂|w₁)·…·P(wₙ|w₁…wₙ₋₁), minted factor by factor in §02. A decoder-only model is a function that outputs each factor — nothing more. GPT-1 (2018) computed it with 117M parameters; ChatGPT computes it with many more; the demo below computes it with 1,159, trained in your browser the moment this page loaded.

WATCH A LANGUAGE MODEL WRITE

The machine: one token at a time. At every step the model exposes a full probability distribution over its vocabulary; one token is drawn, appended, and the whole thing repeats. That loop — and nothing else — is how GPT writes anything: essays, code, dialogue, this sentence’s next word.

This toy was trained live when the page loaded (18 sentences, 1,159 parameters, deterministic seed). §04 replays the entire training run; §08 hands you the loop’s controls.

THE STORY SO FARgenerated, token by token

CONTEXT (last 2 tokens)

P( NEXT TOKEN | CONTEXT ) — LIVE

That was generation. But every honest mechanism lecture starts one step earlier: before a model can produce language, we need a mathematical object called “the probability of a sentence.” Defining it is the whole game — and it factors.

02 LANGUAGE MODELING, DEFINED

What is the probability of a sentence?

You play this game daily: someone starts a sentence and you finish it. Claude Shannon — the father of information theory — turned it into a number in 1948: score every possible sentence, higher for the ones people actually write. One application of conditional probability later, the whole task collapses into guess the next token, repeat. Below, a real sentence is minted factor by factor; every probability is on the screen.

THE SENTENCE, MINTING — each token’s conditional probability appears beneath it

P( NEXT | CONTEXT ) — THE FACTOR BEING MINTED

∏ not ΣThe chain rule MULTIPLIES conditionals — that is the “autoregressive” in the title: each factor conditions on the entire run of tokens before it.
ln PProducts of small numbers underflow every float; real models sum LOG-probabilities. Your loss curves in §04 and §07 are exactly these logs, negated.
Shannon 1951Measured English’s entropy by having humans guess text letter by letter: 0.6–1.3 bits/char. Language modeling = approximating that entropy, mechanically.

So a language model is any function that outputs those factors. The 1950s answer — and still a shockingly good one — needs no learning at all: count.

03 THE COUNTING ANSWER

N-gram models — count, normalize, done

Want P(next | “the cat”)? Find every “the cat” in your corpus, count what followed, divide. No gradients, no GPU — the entire model is a hash map. It works… until it doesn’t: the table’s emptiness grows exponentially with context length. Every number below is counted live from the page’s 18-sentence corpus.

THE WALL — slide the context length and watch coverage collapse

N-GRAM ORDER (context = n−1 tokens)3-GRAM
1 · no context5 · 4-token context

The bet that follows: a table stores the answers it has seen; a function can interpolate answers it has never seen. Bengio et al. (2003) made that bet — embed each word, feed the embeddings to a network, softmax over the vocabulary — and it took fifteen years, one Transformer (Lecture 11) and one objective (§02) to pay off completely.

Time to make the bet yourself. Below is a complete language model — embeddings, hidden layer, softmax — with 1,159 parameters and the exact GPT objective. Press TRAIN and watch it learn the corpus from §03, live, in your browser.

04 THE NEURAL BET, RUN LIVE

Train a language model — right here

The architecture of every neural LM since Bengio (2003): embed the context tokens, mix them in a hidden layer, softmax over the whole vocabulary. Below it trains with full-batch gradient descent on cross-entropy — the identical objective GPT-1 trained on, at 1/100,000th the scale. Watch the loss fall from ln(15) = 2.71 (know-nothing uniform) toward the entropy the data genuinely contains.

THE MACHINE — shapes on every wire (Lecture 10 style)

“the cat”2 tokens
embed2 × 8
concat16
W₁ · tanh16 × 32
hidden32
W₂32 × 15
logits15
softmax15
P(next)15

1,159 parameters · GPT-1: 117,000,000 · GPT-3: 175,000,000,000 — same objective, same loss, same softmax. Only the mixer between embedding and logits grows.

CROSS-ENTROPY LOSS — live, full-batch gradient descent

0gradient steps
—loss (nats / token)
—perplexity = e^loss

PROBE THE MODEL — click a context, watch its distribution sharpen as training runs

2.71 → 0.40The loss falls from know-nothing uniform to the entropy the data genuinely contains (measured: 0.395 nats, ppl 1.48) — verbs remain a real choice; §07 totals that ledger exactly.
18 sentences…are enough for this toy. GPT-1 needed ~0.8B words — the objective scales, the data is the fuel.
1 hidden layer…mixes the context. Replace it with the masked self-attention + FFN blocks of Lecture 11 and this same picture IS GPT — §05 performs that surgery.

The learner works — now scale the picture. Keep the objective (§02) and the training loop you just watched (§04); swap the one tanh layer for Lecture 11’s decoder blocks. The architecture part of “decoder-only” is pure subtraction.

05 ARCHITECTURE SURGERY

One 2017 machine, three destinies — GPT keeps one half

Lecture 11’s closing table showed encoder-only, encoder–decoder and decoder-only as three families. Here is the surgery at sublayer granularity: which stack each keeps, which mask, which training task — and then the piece every diagram forgets: the LM head, one matrix that turns a hidden state into a distribution over the entire vocabulary.

THE LM HEAD, LIVE — hidden state → logits → next-token distribution, one dot product per word

h · Wᵤ — the “unembedding”

GPT-1 (2018) vs ITS 2017 PARENT vs THIS PAGE’S TOY

The stack computes one hidden state per position — but a next-token model must never let position t’s state be influenced by token t+1. That constraint has a name, a shape, and a starring role in the single most important trick of the whole architecture.

06 THE CAUSAL MASK

The arrow of time, as a matrix

Lecture 11 §08 introduced the mask. Here it is the whole story: the lower-triangular “allowed to look” pattern that makes prediction honest, lets every position train in parallel, and doubles as the definition of “autoregressive.” Then the second half of the trick everyone gets wrong: the one-position label shift.

ATTENDING POSITION (row of the matrix)a
score visible (allowed) −∞ — future, masked LEAK — the cheat, unmasked

ROW’S ATTENTION WEIGHTS — softmax over what it can see

THE LABEL SHIFT — inputs on top, labels one step left, hₜ predicts wₜ₊₁ (hover any input)

T in, T outOne forward pass over T tokens yields T logits — each position trained against the token ONE STEP AHEAD. Nothing is wasted; nothing peeks.
shift by oneIn code it is literally logits[:, :-1] vs ids[:, 1:] — the diagonal you are hovering above.

Mask + shift + the loss you watched fall in §04 — assemble them and “training a GPT” is one sentence of math. Time to total the books, position by position.

07 TRAINING = TEACHER FORCING

T predictions, one pass, one number

Feed the real sentence in (the “teacher”), demand a next-token distribution at every position, score each against the truth that actually followed. The mean of those −log p’s is the training loss — and its exponential, perplexity, is the number the whole field chased for decades. Every value below is computed live from the same corpus counts of §03.

THE TARGETS — click any position (or STEP) to score it

P( NEXT | CONTEXT ) FROM THE COUNTS — truth marked

THE LOSS LEDGER — one bar per position (−ln p)

SELF-SUPERVISION — slide the window; the labels write themselves

No annotators, no task labels: the text is the answer key. Each window position yields one (context → next-token) example the moment it slides past — this is where GPT’s training set comes from.

SLIDE THE TRAINING WINDOW8 tokens

THE WINDOW

EXAMPLES THIS WINDOW YIELDS — context → target

108training examples hidden in this page’s 18 toy sentences — every position is one (context → next) pair, free.
~0.8Bwords of BooksCorpus trained GPT-1 (2018) — ~7,000 books. Each word: one more example.
40 GBof WebText trained GPT-2 (2019) — the objective never changed, only the data.
~300Btokens trained GPT-3 (2020). The scaling story is Lecture 13 §06 — this lecture’s job was the objective.

Training is done — weights frozen forever. Now run the machine the other way: no more teacher, the model eats its own output. Five little stages, forever, until </s>. This loop is every chat box you have ever used.

08 GENERATION

The autoregressive loop, on your marks

The trained §04 model (identical weights, frozen) writes left to right: read context → forward pass → distribution → pick → append → repeat. Greedy always takes the top token; sampling draws from the bars. Flip between them and watch what repetition does — and what variety costs.

CONTEXTlast 2
FORWARDembed → tanh → W₂
SOFTMAX15 probs
PICKgreedy / sample
APPEND↺ loop

CONTEXT

P( NEXT | CONTEXT )

GENERATED STREAM

THE LOOP’S LOG

Count the cost: every generated token costs a full forward pass — at GPT scale, that is the loop’s Achilles heel and the reason the KV cache exists. That speed story (and temperature / top-k / top-p, the full decoding lab) belongs to Lecture 13 · GPT — Generative Pretrained Transformers, which scales everything on this page by 100,000×.

Notice what you never touched in §08: the weights. The only thing you changed was the context — and the output changed completely. Hold that thought; it has a name, and it ate the world.

09 THE INTERFACE

The context is the program

A decoder-only model has no buttons, no modes, no API but one: the prefix. Same frozen weights — different context, different continuation. GPT-2 (2019) showed this could go absurdly far: translate, summarize, answer — all by prompting, zero weight updates. Build a prefix below and steer the model yourself.

YOUR PREFIX (click tokens to extend, ⌫ to remove)

P( NEXT | YOUR PREFIX ) — live

CONTINUATION — greedy, weights frozen

SAME WEIGHTS, THREE FUTURES — the verb distributions the trained model gives each subject

Nothing about the model differs between these three columns — only two tokens of context. Now imagine those two tokens are two thousand, and the “verbs” are “answer in Urdu” / “write Python” / “summarize this document”. That is prompting; that is few-shot learning; that is the entire user interface of every LLM product since.

One objective, half an architecture, a loop, and a prompt — that is the entire mechanism. The last honest question: why did this half of the 2017 machine inherit the earth, and what did it pay?

10 THE HONEST LEDGER

Why decoder-only won

Not because it is the best reader (BERT reads better) or the natural translator (the 2017 original translates better). It won on economics: one objective that never needs labels, one architecture that gets better with nothing but more data and more parameters, and output that IS the product. Pick a task and ask each family for its honest verdict.

PICK A TASK — who should you hire?

The 2018 demonstration

GPT-1: pretrain ONE decoder on books (next-token, §04’s loss), then bolt a tiny head per task and fine-tune — 9 of 12 benchmarks at state of the art. The decoder’s representation, learned from guessing the next word, already understood sentiment, entailment, similarity.

The 2020 escalation

GPT-3: skip the fine-tuning entirely — 175B parameters, tasks solved inside the prompt (§09, scaled a million times over). Few-shot learning with frozen weights proved the objective alone had been the point all along.

The honest bill

Left-context only — never a right neighbor. O(n²) attention over the context. Generation is sequential: one token per pass. Exposure bias: trained on real text, deployed on its own. Every one of those bills is a research field now — none of them overturned the verdict.

11 CODE LAB

The whole lecture, in runnable code

Three programs, era-accurate: the counting LM of §03 in thirty lines of NumPy; GPT-1’s decoder in 2018 PyTorch — the Lecture 11 block plus the causal mask, the weight-tied LM head and the shift; and the generation loop of §08, verbatim.

AUTOREGRESSION · THREE WAYS

12 MASTERY CHECK

Eight questions — one per mechanism.

YOUR SCORE

0 / 8