CAS logo CAS NLP LABInteractive Tutorial Series ← Series

LECTURE 37 · BART DENOISING SEQ2SEQ · Lewis et al. (Facebook AI · ACL 2020 · arXiv:1910.13461) · Companion to T5 (Lecture 23)

BART — Bidirectional Encoder + Autoregressive Denoiser

BERT reads text from both left and right but struggles to write fluent open-ended prose. GPT generates fluent prose left-to-right but cannot peek at future context (dropping to 76.7 F1 on SQuAD). BART fuses both architectures into a Denoising Sequence-to-Sequence Autoencoder: corrupt a document with any arbitrary noising transform, read the damaged version bidirectionally in the Encoder, and reconstruct the clean original word-by-word in the Autoregressive Decoder.

BERT + GPTENC-DEC BRIDGEBiDir Encoder + Causal Decoder
5 NoisesPOISSON λ=3 INFILLEven deletes & shuffles text
45.14 R-1XSUM SOTA (+6.3)CNN/DailyMail 44.16 R-1
94.6 / 89.9EQUALS ROBERTASQuAD 1.1 F1 & MNLI Acc
37.96 BLEUWMT16 RO-EN MT2-stage encoder transplant
SIX VISUAL LESSONS

Everything inside BART Denoising Pre-Training

02 THE DENOISING GAP · ARBITRARY CORRUPTION & ELASTIC CROSS-ATTENTION

Why BERT stalls on variable-length holes — and how BART’s Cross-Attention reconstructs 100% of clean text

Plain-English Analogy: Suppose you tear k = 3 consecutive words out of a sentence and patch the hole with a single sticky note [MASK]. An encoder-only model like BERT acts like a rigid grid with 1 output slot per input token — it cannot expand 1 sticky note back into 3 words! BART separates the damaged input room (Encoder, length S_enc) from the clean writing desk (Decoder, length S_dec) across Cross-Attention so 1 [MASK] smoothly expands into 0, 1, 2, 3+ original words.

DOCUMENT CORRUPTION FRACTION (p_mask) 30% tokens (BART-Large default)
POISSON MEAN SPAN LENGTH (λ) λ = 3.0 tokens/span
BERT 1:1 Slot Ambiguity:
Blocked on k > 1 deletions
BART Decoder Full Signal:
100% clean tokens predicted
SPAN COMPRESSION & RECONSTRUCTION CURVE

BERT Masked 15% vs BART Full-Doc Denoising Signal

Click curve to inspect any λ / N
GEOMETRIC VISUAL PROOF TOOL

How 1 [MASK] Expands Into k Original Tokens

Theory explains the elastic bridge — now shred real sentences five ways and watch BART rebuild every clean token in the interactive studio.

03 BART DENOISING VISUAL LEARNING STUDIO · 4 INTERACTIVE LENSES

Corrupt text five ways and watch BART denoise token by token

Pick a two-sentence document (or type your own short prose with two sentences), choose any of Lewis et al.’s 5 noising functions, press Next Token ▶ or ▶ Auto-Play, and switch across 4 visual lenses: (1) Corruption → Reconstruction Battle, (2) Encoder BiDir vs Decoder Causal + Cross-Attn X-Ray, (3) 5-Noise Stress Test vs Table 1, and (4) Downstream Task Switchboard (§3.1–§3.4).

04 INTUITION & ARCHITECTURAL DESIGN PROOFS

Four intuition proofs: Detective + Novelist, Single-[MASK] seams, Final </s> voting & NMT encoder swap

Every major design choice in BART solves a specific blind spot of BERT or GPT. Click through the four plain-English visual stories below before touching any equations.

Verified Paper Source (Lewis et al., ACL 2020): https://arxiv.org/abs/1910.13461 ↗

05 PRETRAINING TEACHER FORCING VS BEAM-5 GENERATION

Systems flow: Parallel Teacher-Forced Reconstruction vs Cached Cross-Attn Beam Search

Plain-English Analogy: Think of restoring a damaged book. During Training, the printer already holds the clean ground-truth manuscript — so all S_dec target positions grade themselves simultaneously in one parallel GPU pass! During Generation (XSum / CNN-DM), the encoder reads the noisy article only once and pins its Cross-Attention keys/values onto a shared chalkboard (H_enc) while 5 parallel beam writers copy phrases word-by-word without ever re-reading the source article.

ENCODER SOURCE DOCUMENT LENGTH (S_enc) 512 source tokens
GENERATED SUMMARY LENGTH (S_tgt, BEAM = 5) 64 summary tokens

06 BY-HAND NUMERICAL VERIFICATION (d = 4, 2 HEADS, EXACT GELU)

Six-decimal proof: "purred" turns GPT’s dog (89.3%) into BART’s cat (91.5%)

Encoder receives corrupted input ["The", "[MASK]", "purred"] while the right-shifted Decoder starts ["<s>", "The"] to predict the missing word. Step through all 4 FP64-calibrated stages and watch Cross-Attention carry right-hand feline context directly into the Decoder softmax!

07 BART ENCODER–DECODER ARCHITECTURE SCHEMATIC

Inside every BART block: BiDir GeLU Encoder → Static Cross-KV Vault → Causal Decoder

Switch between BART-Base (139M · 6E+6D), BART-Large (406M · 12E+12D), and mBART (611M · 250K Vocab) and filter by pipeline stage. Spot the upper Residual Bypass Rails, exact GeLU FFNs, and the dashed-teal Static Encoder Cross-KV Vault that freezes once per document.

08 PARAMETER BREAKDOWN (+10% OVER BERT) & POISSON(λ) HISTOGRAM

Interactive budget: Why BART adds +10% weights over BERT & exact Poisson span probabilities

Section 2.1 notes two structural shifts vs equivalent BERT: every decoder layer adds a Cross-Attention module, while BART removes BERT’s extra pre-softmax FFN layer — yielding ~10% extra parameters at matched depth/width. Dial architecture dimensions and Poisson mean λ below to inspect parameter shares alongside exact P(k = 0..8) span counts.

BART PARAMETER & KV FOOTPRINT: Emb + L×(Enc Self+FFN) + L×(Dec Self+Cross+FFN)
■ Encoder Body + BPE Emb ■ Decoder Self-Attn + GeLU FFN ■ Decoder Cross-Attention Bridge (~10% Delta)
Layers per Stack (L_e = L_d)12 layers (24 total)
Model Hidden Width (d_model)1024 dim
Context Length (S)1,024 tokens
Poisson Infilling Mean (λ)λ = 3.0 tokens

09 EMPIRICAL BENCHMARKS (TABLES 1–6) & 4 PRODUCTION PITFALLS

Headline paper proof & four bugs that break BART

Every single Table 1–6 metric below matches Lewis et al. (arXiv:1910.13461) verbatim — toggle between Base Noise Ablations and Large Generation/NLU SOTA, then inspect four classic PyTorch/HuggingFace traps.

10 ARCHITECTURE COMPARISON LEDGER

BERT/RoBERTa vs GPT Causal LM vs T5 Sentinels vs BART Denoising

Where BART sits directly next to BERT, GPT, and T5 across corruption rules, parameter layout, classification probing, and downstream tasks.

Engineering Property BERT / RoBERTa (Encoder) GPT-2 (Causal Decoder) T5 (Span Sentinels) BART (Denoising Seq2Seq)

Every equation and trick verified — now run complete PyTorch (.train(False)), Keras 3 (layers.Layer), Poisson Noising & HuggingFace BART recipes.

11 CODE LAB · PYTORCH, KERAS 3 & HUGGINGFACE

BART in clean runnable PyTorch & Keras 3

Inspect five concise scripts across a single tab bar: (1) PyTorch BartDecoderBlock with Cross-Attention & GeLU, (2) backend-agnostic Keras 3 KerasBartDecoderLayer(layers.Layer) using keras.ops, (3) Poisson λ=3 text infilling engine with 0-length insertions, (4) HuggingFace summarization (num_beams=5, no_repeat_ngram_size=3) + final-token classification, and (5) production deployment checklist.

BART SEQ2SEQ · PYTORCH, KERAS 3 & TRANSFORMERS

12 MASTERY CHECK

Eight questions — verify your visual, mathematical & engineering mastery of BART.

YOUR SCORE

0 / 8