ModernBERT — Unpadding, GeGLU & Alternating Local/Global Attention at 8,192 Context
Six years after BERT, most production RAG, routing, and guardrail pipelines still run 2018-era 512-token encoders. ModernBERT (149M Base / 395M Large) replaces BERT end-to-end with Sequence Unpadding (cu_seqlens), Alternating Local (128) & Global (every 3rd layer) Dual-RoPE Attention, Bias-Free Pre-LN + GeGLU, a % 64 hardware-aligned 50,368 OLMo BPE vocabulary, and 2 Trillion tokens of web + code pretraining — beating DeBERTaV3 on GLUE Base (88.4 vs 88.1), crushing CodeSearchNet (56.4 / 59.5), dominating MLDR ColBERT (80.2 / 80.4), and running up to 4.2x faster.
Interactive Overview: Four Pillars of ModernBERT
Why 2018-Era Bidirectional Encoders Aged Out
While causal LLMs adopted RoPE, SwiGLU, FlashAttention, Pre-LN, and multi-trillion-token code mixes, bidirectional encoders remained stuck on Post-LN, 512-token position tables, unaligned Tensor Core dims, and quadratic padding waste.
Encoder Evolution Timeline (2018 → Dec 2024)
Interactive 12-Block ModernBERT Architecture Diagram
Toggle between Base (149M, 22L, d=768, GeGLU=1152) and Large (395M, 28L, d=1024, GeGLU=2624), filter by stage, and click any block to inspect shapes, equations, and hardware design choices.
Whole-Model Sequence Unpadding (cu_seqlens) & Packing
Unlike wrappers that only pack inside attention kernels, ModernBERT strips [PAD] tokens before Layer 0 embeddings enter the Transformer stack — saving Linear, LayerNorm, RoPE, GeGLU, and FlashAttention compute across all 22 / 28 layers.
Alternating Local (w = 128) & Global (Every 3rd Layer) Dual-RoPE
Click any layer in Base (22L) or Large (28L) below. Layers with L % 3 == 0 run full 8,192 global attention with theta = 160,000; all other layers use a fast 128-token bidirectional sliding window with theta = 10,000.
Exact 6-Decimal Pure-JS Forward Trace (Token 1: "runs")
Step through every exact tensor operation inside ModernBERT (d=4 pedagogical trace verified against Python FP64): cu_seqlens unpadding, bias-free Pre-LN, dual-theta RoPE (160K vs 10K), FlashAttention context, GeGLU gate product, and tied MLM cross-entropy loss.
Bias-Free Pre-LN, GeGLU & % 64 Hardware-Aware Sizing
Every ModernBERT dimension was co-designed for NVIDIA Tensor Core warps: dropping unnecessary linear/LayerNorm biases, merging gate + value GeGLU matrices, and snapping hidden/FFN/vocab dimensions cleanly onto multiples of 64.
2 Trillion Token WSD Curriculum & 30% Masked LM
Trained across three Warmup-Stable-Decay (WSD) stages using web documents, GitHub code, and scientific corpora at 30% masking (double BERT 15%) with zero Next Sentence Prediction.
Table 1: GLUE NLU, CodeSearchNet, BEIR & MLDR Long-Doc Benchmarks
ModernBERT bridges the historical tradeoff between short-context NLU accuracy and long-context code/retrieval generalization.
Table 2: RTX 4090 Inference Throughput & Max Batch Size
Compare fixed-length and variable-length BF16 throughput (k tok/s) plus maximum batch sizes measured on a consumer RTX 4090.