CAS · L30
← NOMIC/JINA-V3
Warner et al. · Answer.AI & LightOn · Dec 2024 · arXiv:2412.13663 · Lecture 30

ModernBERT — Unpadding, GeGLU & Alternating Local/Global Attention at 8,192 Context

Six years after BERT, most production RAG, routing, and guardrail pipelines still run 2018-era 512-token encoders. ModernBERT (149M Base / 395M Large) replaces BERT end-to-end with Sequence Unpadding (cu_seqlens), Alternating Local (128) & Global (every 3rd layer) Dual-RoPE Attention, Bias-Free Pre-LN + GeGLU, a % 64 hardware-aligned 50,368 OLMo BPE vocabulary, and 2 Trillion tokens of web + code pretraining — beating DeBERTaV3 on GLUE Base (88.4 vs 88.1), crushing CodeSearchNet (56.4 / 59.5), dominating MLDR ColBERT (80.2 / 80.4), and running up to 4.2x faster.

149M / 395M
Base (22L) & Large (28L)
8,192
Native RoPE Context (16x BERT)
147.3k / 133.8k
RTX 4090 tok/s (Short / Long Var)
59.5 / 80.4
Large CSN Code / MLDR ColBERT
2.0T
Pretraining Tokens (30% MLM)

Interactive Overview: Four Pillars of ModernBERT

02

Why 2018-Era Bidirectional Encoders Aged Out

While causal LLMs adopted RoPE, SwiGLU, FlashAttention, Pre-LN, and multi-trillion-token code mixes, bidirectional encoders remained stuck on Post-LN, 512-token position tables, unaligned Tensor Core dims, and quadratic padding waste.

Encoder Evolution Timeline (2018 → Dec 2024)

03

Interactive 12-Block ModernBERT Architecture Diagram

Toggle between Base (149M, 22L, d=768, GeGLU=1152) and Large (395M, 28L, d=1024, GeGLU=2624), filter by stage, and click any block to inspect shapes, equations, and hardware design choices.

04

Whole-Model Sequence Unpadding (cu_seqlens) & Packing

Unlike wrappers that only pack inside attention kernels, ModernBERT strips [PAD] tokens before Layer 0 embeddings enter the Transformer stack — saving Linear, LayerNorm, RoPE, GeGLU, and FlashAttention compute across all 22 / 28 layers.

05

Alternating Local (w = 128) & Global (Every 3rd Layer) Dual-RoPE

Click any layer in Base (22L) or Large (28L) below. Layers with L % 3 == 0 run full 8,192 global attention with theta = 160,000; all other layers use a fast 128-token bidirectional sliding window with theta = 10,000.

06

Exact 6-Decimal Pure-JS Forward Trace (Token 1: "runs")

Step through every exact tensor operation inside ModernBERT (d=4 pedagogical trace verified against Python FP64): cu_seqlens unpadding, bias-free Pre-LN, dual-theta RoPE (160K vs 10K), FlashAttention context, GeGLU gate product, and tied MLM cross-entropy loss.

STEP 1 / 6
07

Bias-Free Pre-LN, GeGLU & % 64 Hardware-Aware Sizing

Every ModernBERT dimension was co-designed for NVIDIA Tensor Core warps: dropping unnecessary linear/LayerNorm biases, merging gate + value GeGLU matrices, and snapping hidden/FFN/vocab dimensions cleanly onto multiples of 64.

08

2 Trillion Token WSD Curriculum & 30% Masked LM

Trained across three Warmup-Stable-Decay (WSD) stages using web documents, GitHub code, and scientific corpora at 30% masking (double BERT 15%) with zero Next Sentence Prediction.

09

Table 1: GLUE NLU, CodeSearchNet, BEIR & MLDR Long-Doc Benchmarks

ModernBERT bridges the historical tradeoff between short-context NLU accuracy and long-context code/retrieval generalization.

10

Table 2: RTX 4090 Inference Throughput & Max Batch Size

Compare fixed-length and variable-length BF16 throughput (k tok/s) plus maximum batch sizes measured on a consumer RTX 4090.

11

Production Engineering: Choosing Your Encoder Stack

12

Bidirectional Encoder Dynasty (BERT → ModernBERT)

13

Architecture & Trace Verification Ledger

14

4 Production PyTorch & Transformers Recipes

15

8-Question ModernBERT Mastery Check

MASTERY SCORE 0 / 8