# LECTURE 24 · XLM & XLM-R · Lample & Conneau (NeurIPS 2019) → Conneau et al. (ACL 2020) · Companion to Lecture 23 (T5)

XLM — one Transformer across every script and border

Monolingual BERT (Lecture 14) and T5 (Lecture 23) excelled inside one vocabulary — yet training separate models for 100+ world languages starves low-resource scripts and prevents cross-lingual transfer. XLM (Cross-lingual Language Model Pretraining) forged a single shared geometric space via shared subword vocabularies, temperature-balanced sampling, continuous-stream MLM, and supervised Translation Language Modeling (TLM) with positional resets — jumping zero-shot NLI to 75.1% and Unsupervised NMT to 34.3 BLEU before XLM-R scaled unsupervised learning to 2.5 TB across 100 languages.

● LIVE CROSS-LINGUAL TRANSFER SWITCHBOARD · CLM · MLM · TLM · XLM-R
Shared Space · Tied Vocab Head
STREAM A / SOURCE INPUT
STREAM B / TARGET ALIGNMENT
15 → 100LanguagesXNLI (XLM) → CC-100 (XLM-R)
95K → 250KShared VocabJoint BPE → SentencePiece
75.1% → 80.9%XNLI Zero-ShotXLM (MLM+TLM) → XLM-R Large
34.3 BLEUUnsupervised MTWMT'16 De→En (+9 BLEU jump)
★ SIX FOUNDATIONAL BREAKTHROUGHS

How XLM & XLM-R Unified 100 Languages in One Geometry

Click any breakthrough card below to jump directly to its interactive mathematical visualizer, paper benchmarks, and PyTorch code lab.

02 Why Representations Must Share One Basis

From Isolated Monolingual Islands to Cross-Lingual Alignment

If an English classifier sits atop a rotated coordinate basis different from French or Urdu embeddings, zero-shot transfer fails completely. XLM unifies all scripts into one shared latent subspace — first weakly via shared BPE anchor tokens (MLM), and then tightly via parallel cross-attention (TLM).

03 Complete Model Blueprint

XLM Architecture Block Diagram (Tri-Embedding, Position Reset & XLM-R)

Inspect every layer inside XLM (1024-d, 8H, GELU, Post-LN) and toggle live between XLM · TLM (parallel streams with target position reset), XLM · MLM (continuous 256-token stream), and XLM-R (250K SentencePiece without Language Embeddings). Click any block or press Trace Forward Pass.

04 Balancing High- & Low-Resource Corpora

Smoothed Language Sampling: qᵢ ∝ pᵢ^α

Wikipedia and CommonCrawl are heavily skewed toward English and major European languages. Training proportionally (α = 1.0) starves low-resource scripts like Urdu and Swahili, whereas uniform sampling (α = 0.0) overfits tiny corpora. Adjust α below to compute live sampling probabilities and amplification multipliers.

0.70
05 Three Pretraining Regimes

CLM vs Continuous-Stream MLM vs Bilingual TLM (with Position Reset)

Lample & Conneau compared three objectives: causal left-to-right CLM, continuous 256-token MLM (removing BERT's NSP while inverse-weighting frequent subwords), and parallel TLM concatenating source and target sentences with restarted positional indices.

06 Exact Algebraic Forward Pass

4-Step Hand-Computed TLM Trace: "my [MASK]" || "chien" → "dog"

Trace every floating-point number (verified to 6 decimals) on a minimal 3D bilingual sequence where English [MASK] sits at pos=1 alongside French "chien" (with Position Reset pos=0 and distinct E_lang("fr")). Watch Cross-Stream Self-Attention pull French semantics into English prediction.

07 Machine Translation Breakthrough

Initializing Encoder & Decoder From XLM Pretraining (34.3 BLEU Unsupervised)

Before XLM, Unsupervised NMT struggled with cold-start bilingual dictionary heuristics. Lample & Conneau initialized both the Seq2Seq Encoder and Decoder directly from a pretrained 6-layer cross-lingual MLM, then applied Denoising Auto-Encoding and Iterative Back-Translation.

Paper-Verified WMT Translation Milestones (BLEU)

08 Empirical Evidence (Table 1)

XNLI Zero-Shot Classification Across All 15 Languages

Inspect the exact paper Table 1 numbers from Lample & Conneau (2019). Select any model row below to render per-language accuracies spanning Latin scripts (en, fr, es, de), non-Latin scripts (el, bg, ru, ar, th, zh), and low-resource tail languages (hi, sw, ur).

Model (Zero-Shot Transfer) enfresdeel bgrutrarvi thzhhiswurAVG
09 Scaling Unsupervised Pretraining

XLM-R (2020) & The Curse of Multilinguality

Conneau et al. (2020) discovered a fundamental trade-off: cramming more languages K into fixed parameters boosts low-resource scripts initially via positive transfer, yet degrades high-resource accuracy via capacity dilution — until you scale to 2.5 TB CC-100, a 250K SentencePiece vocabulary, and 559M parameters.

100 Languages

Illustrative curve of Conneau et al. (2020) Section 4.1: capacity contention hits low-capacity models beyond ~15–30 languages unless width, depth (24L/1024d), and subword allocation (250K) scale together.

Verified XLM-R Benchmarks (ACL 2020)

10 Subword Vocabulary Engineering

Shared 95K Joint BPE (XLM) vs 250K SentencePiece Unigram (XLM-R)

Vocabulary size dictates token fertility across scripts. Compare how XLM's 80K joint BPE merges (~95K tokens) fragment low-resource Arabic/Urdu compared to XLM-R's 250K SentencePiece Unigram — and why dropping Language Embeddings enables seamless code-switching.

11 Fine-Tuning & Deployment Patterns

Zero-Shot vs Translate-Test vs Translate-Train-All

Given a pretrained cross-lingual encoder, practitioners choose among four downstream evaluation regimes trading off offline MT cost, inference latency, and multilingual label accuracy.

12 Lineage 2018 – 2023

The Multilingual Dynasty: mBERT → LASER → XLM → XLM-R → NLLB

Select any milestone model to trace how cross-lingual pretraining evolved from 104 Wikipedias to 200-language sparse translation.

13 Architectural Comparison Matrix

Side-by-Side Specification Ledger

Compare layers, hidden dimensions, vocabulary designs, Language Embedding usage, objectives, and verified XNLI results across the multilingual Transformer canon.

Model Layers & Width Vocabulary Lang Emb Objective Pretraining Corpus Verified XNLI / Role
14 Hands-On Implementation

PyTorch & Hugging Face XLM / XLM-R Code Lab

Inspect exact Hugging Face XLMTokenizer TLM position-reset indexing, qᵢ ∝ pᵢ^α batch sampling, zero-shot XLMRobertaForSequenceClassification inference, and the XLMTriEmbedding layer.

15 Concept Check

XLM & XLM-R Mastery Quiz (8 Questions)

Test your mastery of smoothed sampling α, TLM positional resets, continuous MLM streams, UNMT encoder-decoder initialization, XLM-R Language Embedding removal, and the Curse of Multilinguality.

SCORE: 0 / 8