T5 — every NLP problem as text-in to text-out
BERT (Lecture 14) and RoBERTa (Lecture 16) attached separate linear heads per downstream task and could not naturally generate sequences. GPT-2 (Lecture 15) generated text everywhere, yet restricted its input prefix to causal left-to-right attention. T5 (Text-to-Text Transfer Transformer) unified both worlds: prepend a task prefix string to a bidirectional Transformer encoder and decode the answer token-by-token through one single vocabulary cross-entropy head — systematically ablating architecture, pretraining objectives, the 750 GB C4 corpus, and scaling up to 11 billion parameters.
Why the T5 Paper Became the Scientific Benchmark for Transfer Learning
Rather than tweaking one mechanism in isolation, Google Research held Compute, Optimizer, Tokenizer, and Framework fixed to test every design decision cleanly. Click any card to jump directly to its interactive demo:
POSITION IN THE CAS-NLP EVOLUTION — CLICK ANY ERA
How do you make a text generator solve continuous similarity regression without adding a scalar MSE layer? Raffel et al.’s trick takes 0.2 steps.
02 UNIFIED OBJECTIVE · CLASSIFICATION TO REGRESSION
Even continuous numbers become discrete text strings
Every GLUE task except one is categorical — the outlier is STS-B, which scores sentence-pair semantic similarity as a real number y ∈ [0.0, 5.0]. BERT attaches a single-output regression layer trained with MSE loss. T5 refuses to break the Text-to-Text rule: it rounds every STS-B label to the nearest 0.2 increment, producing 26 discrete string targets ("0.0", "0.2", …, "5.0") trained by plain token Cross-Entropy! Drag any continuous similarity score below:
QUANTIZED STRING TARGETS IN VOCABULARY (STEP = 0.2)
03 T5 ARCHITECTURE BLOCK DIAGRAM & PAPER §3.2 BAKE-OFF
Inside the T5 Stack — Encoder–Decoder Architecture Diagram
Unlike Vaswani (2017) and BERT, T5 places Pre-RMSNorm before every sublayer, drops positional vectors at Layer 0 in favor of 32-bucket scalar relative biases (omitted in Cross-Attention), and removes linear projection biases. Hover or click any block below — or toggle T5 v1.0 vs T5 v1.1 / Flan-T5 — to inspect every tensor shape and equation:
THE COMPLETE T5 ARCHITECTURE BLOCK DIAGRAM — ENCODER × N ↔ CROSS-ATTN BRIDGE ↔ DECODER × N
Click or hover any block in either tower, switch v1.0 / v1.1, or press Trace Forward Pass to walk every tensor step.
PAPER §3.2 MATCHED-FLOP ABLATION — WHY ENCODER–DECODER BEAT CAUSAL LM & PREFIX LM
Wait — why pay double parameters (2P) for an Encoder–Decoder instead of a single GPT-style decoder stack (P)? During generation, input prefix x only runs through the Encoder while target y runs through the Decoder, so an Encoder–Decoder of size 2P performs the exact same FLOPs per token as a decoder-only LM of size P! Click each masking topology to compare Table 2 scores:
ATTENTION VISIBILITY MATRIX (PREFIX x₀..x₃ + TARGET y₀..y₃)
PAPER TABLE 2 SCORES (MATCHED FLOPs PER TOKEN)
04 ARCHITECTURAL SIMPLIFICATIONS · PAPER §2.1
Pre-RMSNorm & 32-Bucket Scalar Relative Position Bias
T5 makes three surgical changes to Vaswani (2017): (1) Pre-RMSNorm drops mean subtraction and LayerNorm bias parameters; (2) Linear projections drop bias vectors; (3) Instead of adding sinusoidal vectors to token embeddings at Layer 0, T5 adds a learned scalar bias b_bucket(j − i) inside pre-softmax attention logits Q·Kᵀ / √d_k + B — using 32 logarithmically spaced buckets up to distance 128! Slide Query position i and toggle Encoder vs Decoder:
Why did BERT’s [MASK] waste compute inside a sequence-to-sequence model — and how do <extra_id_k> sentinels solve it?
05 PRETRAINING OBJECTIVE · PAPER §3.3
Span Corruption — <extra_id_k> sentinels slash decoder cost
If T5 fed a 512-token corrupted sequence to the Encoder and asked the Decoder to emit all 512 reconstructed tokens, the Decoder would burn huge FLOPs copying unmasked words. Instead, T5 drops 15% of tokens in contiguous spans (mean length μ = 3) and replaces each entire span with one unique sentinel token (<extra_id_0>, <extra_id_1>, …). The Decoder only outputs the sentinels followed by the deleted spans — cutting target length to ~25%! Try it live:
1. ORIGINAL SENTENCE (CROSSED OUT = DROPPED SPAN)
2. ENCODER INPUT (ONE SENTINEL PER SPAN)
3. DECODER TARGET (ONLY SENTINELS + DROPPED SPANS + TERMINATOR)
06 4-STEP NUMERICAL TRACE · MACHINE-VERIFIED TO 6 DECIMALS
Follow one sentinel query through T5 by hand
No black boxes: trace the exact arithmetic (d_model = 2, h = 1, γ = [1.2, 0.8]) for predicting missing token "you" inside <extra_id_0> of "Thank <extra_id_0> ." — verifying Pre-RMSNorm, Encoder Self-Attention with Scalar Relative Bias, Bias-Free Cross-Attention, and Token Cross-Entropy Loss:
07 PAPER §2.2 & §3.4 · COLOSSAL CLEAN CRAWLED CORPUS
C4 — distilling 20 TB of noisy Common Crawl into 750 GB
Unfiltered web scrapes are packed with cookie warnings, navigation bars, lorem ipsum boilerplate, and duplicated syndicated text — leaving them in costs −1.82 GLUE points! Inspect all six filtering stages of the C4 pipeline and test your own HTML text snippet against C4’s heuristics live:
INTERACTIVE C4 HEURISTIC LINE INSPECTOR (PICK A PRESET OR TYPE YOUR OWN TEXT)
PAPER TABLE 7 — CONTROLLED CORPUS COMPARISON AT EQUAL COMPUTE
08 PAPER §3.3 · PRETRAINING OBJECTIVE BAKE-OFF
Prefix LM vs BERT MLM vs MASS vs Span Corruption
How did Raffel et al. pick Span Corruption (15%, mean length 3) out of ten candidates? Inspect exact encoder/decoder pairs generated on the paper’s sentence alongside controlled Paper Tables 4 & 5 accuracy scores:
ENCODER INPUT TOKENS
DECODER TARGET TOKENS
BENCHMARK SCORES (TABLES 4 & 5)
Scale 180× from T5-Small (60M) to T5-11B — and discover a hidden 65,536-wide FFN surprise plus the Temperature Mixing trick.
09 PAPER §3.5 & §3.6 · SCALING FAMILY & TEMPERATURE MIXING
60M to 11B + Balancing 156B C4 against 2.5K RTE
Inspect every released T5 size — noticing why T5-11B (v1.0) froze d_model = 1024 while exploding d_ff = 65,536 and h = 128 (later rebalanced in Flan-T5-XXL)! Below it, drag Temperature T in T5’s multi-task sampling rule p_m ∝ min(r_m, K)^(1/T):
10 ADAPTATION & INFERENCE · PROMPT TUNING & BEAM SEARCH
Freezing 99.99% of weights + Beam Search Length Penalty
Storing a separate 11B copy per task is impractical. Later work by Lester et al. (2021) proved that at T5-11B scale, freezing the entire model and learning just 20–100 soft prompt tokens (<0.01% params) matches full fine-tuning! During generation on CNN/DM & WMT, T5 uses Beam Search (B = 4) normalized by LP(Y) = ((5 + |Y|) / 6)^α:
11 SENTENCEPIECE UNIGRAM (32,000 + 100 SENTINELS)
Whitespace becomes ▁ and sentinels count down from 32,099
Unlike BERT’s WordPiece (##) and RoBERTa’s Byte-BPE (Ġ), T5 uses SentencePiece trained jointly on English, French, German, and Romanian — plus 100 reserved <extra_id_k> sentinels and TPU-aligned embedding padding (32,128 rows). Compare them live:
T5 SENTENCEPIECE UNIGRAM (▁ PREFIX + SENTINEL IDs)
BERT WORDPIECE (RISK OF [UNK] ON UNSEEN NAMES)
ROBERTA / GPT-2 BYTE-LEVEL BPE (Ġ PREFIX)
12 2019 → 2026 EVOLUTION · THE T5 FAMILY
From T5 v1.0 to mT5, ByT5, UL2, Flan-T5 & T5Gemma
T5’s unified seq2seq paradigm evolved rapidly across tokenization, nonlinearities, instruction tuning, and modern positional embeddings. Click any model in the lineage:
13 LECTURES 14–16 + 23 · COMPLETE 4-MODEL LEDGER
BERT vs GPT-2 vs RoBERTa vs T5 Side by Side
Standing directly beside RoBERTa on the CAS-NLP Series roadmap, compare every architectural, pretraining, data, and head design choice across the 2018–2020 quartet:
| Dimension | BERT (L14) | GPT-2 (L15) | RoBERTa (L16) | T5 (L23) |
|---|
14 PYTORCH & HUGGING FACE IMPLEMENTATION LAB
Four production modules — inference, span collator, relative buckets, and training
Switch between Hugging Face google/flan-t5-base inference, exact descending-sentinel span corruption, T5LayerNorm + _relative_position_bucket, and -100 label-masked Seq2Seq training:
15 MASTERY CHECK · 8 QUESTIONS