ELECTRA — Pre-training Text Encoders as Discriminators Rather Than Generators
Instead of masking 15% of tokens and discarding gradients on the remaining 85%, ELECTRA pairs a small Masked LM Generator G with a full-size Discriminator D trained via Replaced Token Detection (RTD) across 100% of input tokens—with zero [MASK] symbols ever reaching D.
Why 15% MLM Throws Away 85% of Every Step
Standard Masked Language Modeling computes cross-entropy loss on roughly 15% of input slots and introduces an artificial [MASK] symbol absent during fine-tuning. RTD learns from every token in every sentence.
From ELMo & BERT to Discriminative Pretraining
Interactive ELECTRA Block Diagram
Click any of the 12 architectural blocks across the Generator, Stop-Gradient Sampling Wall, Discriminator, and Deployment pipeline—or switch between Small (14M), Base (110M), and Large (335M) scales.
Generator Sampling & Accidental Exact Matches
G samples discrete tokens via x_hat ~ p_G(· | x^masked) with a strict stop-gradient barrier. When G happens to sample the true original word, D's target stays y = 1 (Original)!
Why Generator G Must Stay 1/4–1/2 Width
Too weak a generator feeds D obvious nonsense; matching D's width doubles training FLOPs and produces near-perfect replacements that starve D of clean errors.
Complete 6-Decimal G → Sample → D Trace
Inspect every computed number live in the browser: 2D Generator G LayerNorm & MLM softmax on ["the", "[MASK]", "sat"], sampling "dog" across the stop-gradient barrier, projecting Etok into 3D Discriminator D, and joint loss Ltotal = 6.534271.
Decomposing Why ELECTRA Wins
Does ELECTRA gain mostly from removing [MASK] or from supervising all 100% tokens? Compare GLUE Dev scores across five controlled pretraining objectives plus joint vs adversarial training.
ELECTRA-Small (14M): GPT-Beating Single-GPU Pretraining
At just 14M parameters and 1.4e18 training FLOPs (four days on one V100 GPU), ELECTRA-Small lifts GLUE by +4.3 points over BERT-Small and surpasses 117M-parameter GPT.
Large-Scale GLUE & SQuAD 1.1 / 2.0 Results
At 1/4 the pretraining compute of RoBERTa-500K (400K steps, ~7.5e20 FLOPs), ELECTRA matches RoBERTa on GLUE; at compute parity (1.75M steps), it outperforms RoBERTa, XLNet, and ALBERT.
Eliminating the [MASK] Train-Test Mismatch
Downstream tasks (sentiment, NLI, QA) never contain [MASK] tokens. Because G absorbs every mask slot, D trains solely on real vocabulary tokens.
The Discriminative Pretraining Dynasty
ELECTRA sparked MacBERT synonym corruption, DeBERTa-v3's Gradient-Disentangled Embedding Sharing (GDES), and contrastive-discriminative objectives.
ELECTRA vs Masked LM Tradeoff Ledger
When should an engineering team prefer Replaced Token Detection over pure Masked Language Modeling?
PyTorch & HuggingFace ELECTRA Recipes
Inspect the exact pretraining step (remembering accidental matches stay y = 1 and HF checkpoints return logits for Replaced vs Original), tied asymmetric projections, and downstream D deployment.
8-Question ELECTRA Mastery Quiz
Test your grasp of 100% RTD sample density, stop-gradient MLE training, accidental match labels, 1/4–1/2× generator sizing, λ = 50, and paper ablations.