LECTURE 37 · BART DENOISING SEQ2SEQ · Lewis et al. (Facebook AI · ACL 2020 · arXiv:1910.13461) · Companion to T5 (Lecture 23)
BART — Bidirectional Encoder + Autoregressive Denoiser
BERT reads text from both left and right but struggles to write fluent open-ended prose. GPT generates fluent prose left-to-right but cannot peek at future context (dropping to 76.7 F1 on SQuAD). BART fuses both architectures into a Denoising Sequence-to-Sequence Autoencoder: corrupt a document with any arbitrary noising transform, read the damaged version bidirectionally in the Encoder, and reconstruct the clean original word-by-word in the Autoregressive Decoder.
Everything inside BART Denoising Pre-Training
02 THE DENOISING GAP · ARBITRARY CORRUPTION & ELASTIC CROSS-ATTENTION
Why BERT stalls on variable-length holes — and how BART’s Cross-Attention reconstructs 100% of clean text
Plain-English Analogy: Suppose you tear k = 3 consecutive words out of a sentence and patch the hole with a single sticky note [MASK]. An encoder-only model like BERT acts like a rigid grid with 1 output slot per input token — it cannot expand 1 sticky note back into 3 words! BART separates the damaged input room (Encoder, length S_enc) from the clean writing desk (Decoder, length S_dec) across Cross-Attention so 1 [MASK] smoothly expands into 0, 1, 2, 3+ original words.
BERT Masked 15% vs BART Full-Doc Denoising Signal
How 1 [MASK] Expands Into k Original Tokens
Theory explains the elastic bridge — now shred real sentences five ways and watch BART rebuild every clean token in the interactive studio.
03 BART DENOISING VISUAL LEARNING STUDIO · 4 INTERACTIVE LENSES
Corrupt text five ways and watch BART denoise token by token
Pick a two-sentence document (or type your own short prose with two sentences), choose any of Lewis et al.’s 5 noising functions, press Next Token ▶ or ▶ Auto-Play, and switch across 4 visual lenses: (1) Corruption → Reconstruction Battle, (2) Encoder BiDir vs Decoder Causal + Cross-Attn X-Ray, (3) 5-Noise Stress Test vs Table 1, and (4) Downstream Task Switchboard (§3.1–§3.4).
04 INTUITION & ARCHITECTURAL DESIGN PROOFS
Four intuition proofs: Detective + Novelist, Single-[MASK] seams, Final </s> voting & NMT encoder swap
Every major design choice in BART solves a specific blind spot of BERT or GPT. Click through the four plain-English visual stories below before touching any equations.
05 PRETRAINING TEACHER FORCING VS BEAM-5 GENERATION
Systems flow: Parallel Teacher-Forced Reconstruction vs Cached Cross-Attn Beam Search
Plain-English Analogy: Think of restoring a damaged book. During Training, the printer already holds the clean ground-truth manuscript — so all S_dec target positions grade themselves simultaneously in one parallel GPU pass! During Generation (XSum / CNN-DM), the encoder reads the noisy article only once and pins its Cross-Attention keys/values onto a shared chalkboard (H_enc) while 5 parallel beam writers copy phrases word-by-word without ever re-reading the source article.
06 BY-HAND NUMERICAL VERIFICATION (d = 4, 2 HEADS, EXACT GELU)
Six-decimal proof: "purred" turns GPT’s dog (89.3%) into BART’s cat (91.5%)
Encoder receives corrupted input ["The", "[MASK]", "purred"] while the right-shifted Decoder starts ["<s>", "The"] to predict the missing word. Step through all 4 FP64-calibrated stages and watch Cross-Attention carry right-hand feline context directly into the Decoder softmax!
07 BART ENCODER–DECODER ARCHITECTURE SCHEMATIC
Inside every BART block: BiDir GeLU Encoder → Static Cross-KV Vault → Causal Decoder
Switch between BART-Base (139M · 6E+6D), BART-Large (406M · 12E+12D), and mBART (611M · 250K Vocab) and filter by pipeline stage. Spot the upper Residual Bypass Rails, exact GeLU FFNs, and the dashed-teal Static Encoder Cross-KV Vault that freezes once per document.
08 PARAMETER BREAKDOWN (+10% OVER BERT) & POISSON(λ) HISTOGRAM
Interactive budget: Why BART adds +10% weights over BERT & exact Poisson span probabilities
Section 2.1 notes two structural shifts vs equivalent BERT: every decoder layer adds a Cross-Attention module, while BART removes BERT’s extra pre-softmax FFN layer — yielding ~10% extra parameters at matched depth/width. Dial architecture dimensions and Poisson mean λ below to inspect parameter shares alongside exact P(k = 0..8) span counts.
09 EMPIRICAL BENCHMARKS (TABLES 1–6) & 4 PRODUCTION PITFALLS
Headline paper proof & four bugs that break BART
Every single Table 1–6 metric below matches Lewis et al. (arXiv:1910.13461) verbatim — toggle between Base Noise Ablations and Large Generation/NLU SOTA, then inspect four classic PyTorch/HuggingFace traps.
10 ARCHITECTURE COMPARISON LEDGER
BERT/RoBERTa vs GPT Causal LM vs T5 Sentinels vs BART Denoising
Where BART sits directly next to BERT, GPT, and T5 across corruption rules, parameter layout, classification probing, and downstream tasks.
| Engineering Property | BERT / RoBERTa (Encoder) | GPT-2 (Causal Decoder) | T5 (Span Sentinels) | BART (Denoising Seq2Seq) |
|---|
Every equation and trick verified — now run complete PyTorch (.train(False)), Keras 3 (layers.Layer), Poisson Noising & HuggingFace BART recipes.
11 CODE LAB · PYTORCH, KERAS 3 & HUGGINGFACE
BART in clean runnable PyTorch & Keras 3
Inspect five concise scripts across a single tab bar: (1) PyTorch BartDecoderBlock with Cross-Attention & GeLU, (2) backend-agnostic Keras 3 KerasBartDecoderLayer(layers.Layer) using keras.ops, (3) Poisson λ=3 text infilling engine with 0-length insertions, (4) HuggingFace summarization (num_beams=5, no_repeat_ngram_size=3) + final-token classification, and (5) production deployment checklist.
12 MASTERY CHECK