# LECTURE 25 · ALBERT · Lan et al. (ICLR 2020 · arXiv:1909.11942) · Placed Next to Lecture 24 (XLM)

ALBERT — 18× fewer parameters, stronger representations

Simply making BERT wider hit a wall: BERT-large reached 334M parameters and scaling to H=2048 (1.27B) actually degraded accuracy while exhausting accelerator memory. ALBERT (A Lite BERT) shattered this scaling paradox using two parameter-reduction breakthroughs — Factorized Embedding Parameterization (V×E → E×H) and Cross-Layer Parameter Sharing — paired with a true inter-sentence coherence objective called Sentence-Order Prediction (SOP), reaching GLUE 89.4, RACE 89.4%, and SQuAD 2.0 92.2 F1.

BERT UNFACTORIZED
ALBERT FACTORIZED + TIED
12M · 18MBase & Large9× & 18× Fewer vs BERT
1.7×TPU SpeedupLower Weight Sync Bandwidth
86.5% SOPTrue Coherencevs 52.0% on BERT NSP
89.4 · 92.2GLUE & SQuAD2RACE 86.5% Single / 89.4% Ens
★ SIX CORE BREAKTHROUGHS

How ALBERT Broke the Memory & Coherence Bottleneck

Click any card below to inspect its interactive mathematical simulator, paper ablation table, or verified 6-decimal numerical trace.

02 Why Naive Scaling Fails

The BERT-xlarge (1.27B) Paradox & Parameter Bloat

Increasing unshared BERT hidden size from 1024 → 2048 multiplies parameters to 1.27 Billion — crashing accuracy on RACE and SQuAD 2.0 despite massive compute. Toggle between RACE and SQuAD 2.0 below to compare naive unshared scaling against ALBERT’s parameter-tied trajectory.

03 Complete Model Blueprint

ALBERT Architecture Block Diagram (Factorization Bridge, Tied Stack Loop & SOP)

Inspect every tensor operation inside ALBERT and compare ALBERT-base (12M), ALBERT-xxlarge (235M, H=4096, 64 Heads), and the BERT-large Unshared Baseline (334M). Click any block or press ▶ Trace Forward Pass to watch activations cycle repeatedly through the single weight-tied Transformer block.

04 Decoupling E from H

Factorized Embedding Parameterization: O(V·H) → O(V·E + E·H)

WordPiece and SentencePiece vectors encode context-independent lexical tokens, whereas Transformer hidden vectors store deep context-dependent interactions across 12–24 layers. Why pay for a gigantic V × H table when projecting through a compact rank E = 128 achieves 96.5% parameter reduction at H = 4096?

30,000
128
4096
05 Weight Tying Across Depth (Table 4)

Cross-Layer Parameter Sharing & Smooth L2 Convergence

Toggle between E = 128 and E = 768 and click any sharing regime in Table 4 (Lan et al., 2020) to inspect where parameters live and why all-shared stabilizes layer-to-layer hidden transitions ||H^(l) - H^(l-1)|| rather than oscillating wildly.

Sharing Regime Params SQuAD 1.1 SQuAD 2.0 MNLI SST-2 RACE Avg
06 Exact Algebraic Forward Pass

5-Step Hand-Computed 3D Trace: Factorize (2D → 3D), Tie 2 Passes, & Reverse Softmax

Trace every number (computed live and verified to 6 decimals) on ["[CLS]", "[MASK]", "fox"] with target "quick". Watch the thin E=2D lookup lift via W_proj (2×3) into 3D, iterate twice through the exact same layer weights, reverse-project back to 2D for tied softmax (37.11% "quick"), and classify In-Order SOP (95.04%).

07 True Inter-Sentence Coherence (Table 5)

Why Sentence-Order Prediction (SOP) Outperforms BERT’s NSP

When BERT creates a negative pair by drawing a random paragraph from another Wikipedia article, topic words instantly give the game away (90.5% NSP vs just 52.0% SOP!). Test all three pair regimes below and compare Table 5 empirical scores.

Paper-Verified Table 5: Intrinsic & Downstream Comparison (ALBERT-base)

Pretraining Loss MLM NSP SOP SQuAD 1.1 SQuAD 2.0 MNLI SST-2 RACE Avg
08 Harder Masking Corridors

n-Gram Span Masking (n ≤ 3) Blocks Collocation Shortcuts

Following Joshi et al., ALBERT samples contiguous span lengths n ∈ {1, 2, 3} with probability P(n) ∝ 1/n (giving 54.5% 1-gram, 27.3% 2-gram, and 18.2% 3-gram masks). Toggle between BERT 1-gram masking and ALBERT n-gram spans below.

09 Surprising Optimization Findings

Why 12 Wide Layers Beat 24, Removing Dropout Helps, & LAMB Speeds Training 1.7×

Once embedding bloat and layer duplication are stripped away, scaling laws shift dramatically. Explore all three architectural and optimization discoveries reported in Sections 4.4–4.6 of the paper.

10 Empirical Evidence (Table 8)

GLUE, SQuAD 1.1/2.0 & RACE Reading Comprehension Benchmarks

Compare BERT-large, RoBERTa-large, XLNet-large, and the entire ALBERT family across paper-verified Table 1 and Table 8 scores. Notice how ALBERT-xxlarge uses 30% fewer parameters than BERT-large (235M vs 334M) while surging +14.5% on RACE.

Model Params SQuAD 1.1 F1 SQuAD 2.0 F1 RACE (%) GLUE
11 Input & Output Symmetry

30K SentencePiece Unigram & Reverse Output Softmax Tying

Following XLNet and T5, ALBERT uses lossless 30,000-piece SentencePiece tokenization with ▁ space markers — and pairs input factorization with a reverse linear bottleneck H → E → V so the output softmax never creates a 4096 × 30,000 weight explosion.

12 Conceptual Lineage

From Universal Transformers & ALBERT to Modern LoRA & Recursive LLMs

Select any milestone card to trace how ALBERT’s low-rank factorization and parameter tying shaped efficient NLP architectures from 2018 through modern recurrent-depth reasoning.

13 Side-by-Side Comparison

Full Architectural Tradeoff Ledger: BERT · RoBERTa · XLM-R · T5 · ALBERT

Every parameter budget, sharing structure, pair loss, and speedup tradeoff across our recent CAS-NLP series lectures in one reference table.

Model Vocabulary Emb Factorization Cross-Layer Sharing Pair Objective Masking Params Speed / Role
14 Implementation

PyTorch & HuggingFace ALBERT Code Lab

Minimal, self-contained PyTorch modules implementing V×E → E×H factorized lookup, the single weight-tied nn.TransformerEncoderLayer loop with dropout=0.0, same-document SOP sampling, and a live HuggingFace parameter audit.

15 Verify Understanding

8-Question ALBERT Mastery Quiz

Test your mastery of factorized embeddings, weight-tying FLOP vs parameter memory tradeoffs, L2 stability, SOP vs NSP, n-gram masking, and zero-dropout scaling.

SELF-CHECK SCORE
0 / 8