CAS · L26
← ALBERT
ICLR 2020 · arXiv:2003.10555 · Stanford & Google Brain · Lecture 26

ELECTRA — Pre-training Text Encoders as Discriminators Rather Than Generators

Instead of masking 15% of tokens and discarding gradients on the remaining 85%, ELECTRA pairs a small Masked LM Generator G with a full-size Discriminator D trained via Replaced Token Detection (RTD) across 100% of input tokens—with zero [MASK] symbols ever reaching D.

100% vs 15%
Token positions generating loss
79.0 GLUE
ELECTRA-Small (14M) beats GPT-117M
1/4–1/2 ×
Optimal Generator G width vs D
λ = 50
Joint loss: LMLM + 50·LDisc
89.5 / 90.6
GLUE Dev Avg · SQuAD 2.0 F1
Interactive Headline Switcher
02 The Sample-Efficiency Bottleneck

Why 15% MLM Throws Away 85% of Every Step

Standard Masked Language Modeling computes cross-entropy loss on roughly 15% of input slots and introduces an artificial [MASK] symbol absent during fine-tuning. RTD learns from every token in every sentence.

Sequence Length 14 tokens

From ELMo & BERT to Discriminative Pretraining

03 Dual-Network Architecture

Interactive ELECTRA Block Diagram

Click any of the 12 architectural blocks across the Generator, Stop-Gradient Sampling Wall, Discriminator, and Deployment pipeline—or switch between Small (14M), Base (110M), and Large (335M) scales.

Model Scale Preset
Stage Focus Filter
04 Plausible Corrupted Inputs

Generator Sampling & Accidental Exact Matches

G samples discrete tokens via x_hat ~ p_G(· | x^masked) with a strict stop-gradient barrier. When G happens to sample the true original word, D's target stays y = 1 (Original)!

Step A · Choose Sentence Context
Step B · Simulate Multinomial Draw from G
05 Asymmetric Sizing Rule

Why Generator G Must Stay 1/4–1/2 Width

Too weak a generator feeds D obvious nonsense; matching D's width doubles training FLOPs and produces near-perfect replacements that starve D of clean errors.

06 Live Pure-JS Tensor Verification

Complete 6-Decimal G → Sample → D Trace

Inspect every computed number live in the browser: 2D Generator G LayerNorm & MLM softmax on ["the", "[MASK]", "sat"], sampling "dog" across the stop-gradient barrier, projecting Etok into 3D Discriminator D, and joint loss Ltotal = 6.534271.

STEP 1 / 7
07 Paper Ablations (Section 3.3)

Decomposing Why ELECTRA Wins

Does ELECTRA gain mostly from removing [MASK] or from supervising all 100% tokens? Compare GLUE Dev scores across five controlled pretraining objectives plus joint vs adversarial training.

Controlled Pretraining Objectives (GLUE Dev)
Joint MLE vs Two-Stage vs Adversarial RL/GAN
Joint Objective: min LMLM(θG) + 50 · LDisc(θD) | Stop-Grad: ∂LDisc/∂θG = 0
08 Coupling & Gradient Scale

Tied Token Embeddings & λ = 50 Loss Weighting

Sharing Etok transfers D's dense 100% gradients without forcing dG = dD, while λ = 50 balances binary BCE (~0.1–0.3) against 30,522-way MLM loss (~2–6).

Interactive Loss Scale Slider (from verified trace: LMLM = 0.6331, LDisc = 0.1180)
09 Paper Table 1 · Small Scale

ELECTRA-Small (14M): GPT-Beating Single-GPU Pretraining

At just 14M parameters and 1.4e18 training FLOPs (four days on one V100 GPU), ELECTRA-Small lifts GLUE by +4.3 points over BERT-Small and surpasses 117M-parameter GPT.

Table 1 Comparison · Small & Base Models
10 Tables 2, 3 & 4 Benchmarks

Large-Scale GLUE & SQuAD 1.1 / 2.0 Results

At 1/4 the pretraining compute of RoBERTa-500K (400K steps, ~7.5e20 FLOPs), ELECTRA matches RoBERTa on GLUE; at compute parity (1.75M steps), it outperforms RoBERTa, XLNet, and ALBERT.

11 Distribution Alignment

Eliminating the [MASK] Train-Test Mismatch

Downstream tasks (sentiment, NLI, QA) never contain [MASK] tokens. Because G absorbs every mask slot, D trains solely on real vocabulary tokens.

12 Lineage & Evolution

The Discriminative Pretraining Dynasty

ELECTRA sparked MacBERT synonym corruption, DeBERTa-v3's Gradient-Disentangled Embedding Sharing (GDES), and contrastive-discriminative objectives.

13 Engineering Judgment

ELECTRA vs Masked LM Tradeoff Ledger

When should an engineering team prefer Replaced Token Detection over pure Masked Language Modeling?

14 Implementation

PyTorch & HuggingFace ELECTRA Recipes

Inspect the exact pretraining step (remembering accidental matches stay y = 1 and HF checkpoints return logits for Replaced vs Original), tied asymmetric projections, and downstream D deployment.

15 Self-Check

8-Question ELECTRA Mastery Quiz

Test your grasp of 100% RTD sample density, stop-gradient MLE training, accidental match labels, 1/4–1/2× generator sizing, λ = 50, and paper ablations.

Current Score
0 / 8