XLM — one Transformer across every script and border
Monolingual BERT (Lecture 14) and T5 (Lecture 23) excelled inside one vocabulary — yet training separate models for 100+ world languages starves low-resource scripts and prevents cross-lingual transfer. XLM (Cross-lingual Language Model Pretraining) forged a single shared geometric space via shared subword vocabularies, temperature-balanced sampling, continuous-stream MLM, and supervised Translation Language Modeling (TLM) with positional resets — jumping zero-shot NLI to 75.1% and Unsupervised NMT to 34.3 BLEU before XLM-R scaled unsupervised learning to 2.5 TB across 100 languages.
How XLM & XLM-R Unified 100 Languages in One Geometry
Click any breakthrough card below to jump directly to its interactive mathematical visualizer, paper benchmarks, and PyTorch code lab.
From Isolated Monolingual Islands to Cross-Lingual Alignment
If an English classifier sits atop a rotated coordinate basis different from French or Urdu embeddings, zero-shot transfer fails completely. XLM unifies all scripts into one shared latent subspace — first weakly via shared BPE anchor tokens (MLM), and then tightly via parallel cross-attention (TLM).
XLM Architecture Block Diagram (Tri-Embedding, Position Reset & XLM-R)
Inspect every layer inside XLM (1024-d, 8H, GELU, Post-LN) and toggle live between XLM · TLM (parallel streams with target position reset), XLM · MLM (continuous 256-token stream), and XLM-R (250K SentencePiece without Language Embeddings). Click any block or press Trace Forward Pass.
Smoothed Language Sampling: qᵢ ∝ pᵢ^α
Wikipedia and CommonCrawl are heavily skewed toward English and major European languages. Training proportionally (α = 1.0) starves low-resource scripts like Urdu and Swahili, whereas uniform sampling (α = 0.0) overfits tiny corpora. Adjust α below to compute live sampling probabilities and amplification multipliers.
CLM vs Continuous-Stream MLM vs Bilingual TLM (with Position Reset)
Lample & Conneau compared three objectives: causal left-to-right CLM, continuous 256-token MLM (removing BERT's NSP while inverse-weighting frequent subwords), and parallel TLM concatenating source and target sentences with restarted positional indices.
4-Step Hand-Computed TLM Trace: "my [MASK]" || "chien" → "dog"
Trace every floating-point number (verified to 6 decimals) on a minimal 3D bilingual sequence where English [MASK] sits at pos=1 alongside French "chien" (with Position Reset pos=0 and distinct E_lang("fr")). Watch Cross-Stream Self-Attention pull French semantics into English prediction.
Initializing Encoder & Decoder From XLM Pretraining (34.3 BLEU Unsupervised)
Before XLM, Unsupervised NMT struggled with cold-start bilingual dictionary heuristics. Lample & Conneau initialized both the Seq2Seq Encoder and Decoder directly from a pretrained 6-layer cross-lingual MLM, then applied Denoising Auto-Encoding and Iterative Back-Translation.
Paper-Verified WMT Translation Milestones (BLEU)
XNLI Zero-Shot Classification Across All 15 Languages
Inspect the exact paper Table 1 numbers from Lample & Conneau (2019). Select any model row below to render per-language accuracies spanning Latin scripts (en, fr, es, de), non-Latin scripts (el, bg, ru, ar, th, zh), and low-resource tail languages (hi, sw, ur).
| Model (Zero-Shot Transfer) | en | fr | es | de | el | bg | ru | tr | ar | vi | th | zh | hi | sw | ur | AVG |
|---|
XLM-R (2020) & The Curse of Multilinguality
Conneau et al. (2020) discovered a fundamental trade-off: cramming more languages K into fixed parameters boosts low-resource scripts initially via positive transfer, yet degrades high-resource accuracy via capacity dilution — until you scale to 2.5 TB CC-100, a 250K SentencePiece vocabulary, and 559M parameters.
Illustrative curve of Conneau et al. (2020) Section 4.1: capacity contention hits low-capacity models beyond ~15–30 languages unless width, depth (24L/1024d), and subword allocation (250K) scale together.
Verified XLM-R Benchmarks (ACL 2020)
Shared 95K Joint BPE (XLM) vs 250K SentencePiece Unigram (XLM-R)
Vocabulary size dictates token fertility across scripts. Compare how XLM's 80K joint BPE merges (~95K tokens) fragment low-resource Arabic/Urdu compared to XLM-R's 250K SentencePiece Unigram — and why dropping Language Embeddings enables seamless code-switching.
Zero-Shot vs Translate-Test vs Translate-Train-All
Given a pretrained cross-lingual encoder, practitioners choose among four downstream evaluation regimes trading off offline MT cost, inference latency, and multilingual label accuracy.
The Multilingual Dynasty: mBERT → LASER → XLM → XLM-R → NLLB
Select any milestone model to trace how cross-lingual pretraining evolved from 104 Wikipedias to 200-language sparse translation.
Side-by-Side Specification Ledger
Compare layers, hidden dimensions, vocabulary designs, Language Embedding usage, objectives, and verified XNLI results across the multilingual Transformer canon.
| Model | Layers & Width | Vocabulary | Lang Emb | Objective | Pretraining Corpus | Verified XNLI / Role |
|---|
PyTorch & Hugging Face XLM / XLM-R Code Lab
Inspect exact Hugging Face XLMTokenizer TLM position-reset indexing, qᵢ ∝ pᵢ^α batch sampling, zero-shot XLMRobertaForSequenceClassification inference, and the XLMTriEmbedding layer.
XLM & XLM-R Mastery Quiz (8 Questions)
Test your mastery of smoothed sampling α, TLM positional resets, continuous MLM streams, UNMT encoder-decoder initialization, XLM-R Language Embedding removal, and the Curse of Multilinguality.