NeoBERT & EuroBERT — Deep-Narrow 250M, Multilingual 2.1B GQA & Modern Encoder Scaling
Following ModernBERT, two landmark 2025 encoder families complete the modernization of bidirectional transformers using Pre-RMSNorm, SwiGLU, FlashAttention, and high-base RoPE. NeoBERT (250M) proves an optimal 28L × 768 deep-and-narrow shape reaches 89.0 GLUE and 250M MTEB SOTA as a plug-and-play WordPiece drop-in up to 4,096 tokens, while EuroBERT (210M / 610M / 2.1B) pairs 3:1 Grouped-Query Attention (18Q:6KV), RoPE theta = 250,000 at 8,192 context, LLaMA-3 128K BPE, and 5T multilingual + code + math tokens.
Interactive Architecture Spotlight: NeoBERT vs EuroBERT
Why Deep-and-Narrow (28L × 768) Beats Wide-Shallow Encoders
Le Breton et al. swept depth-to-width allocations at a fixed 250M budget and discovered that stacking 28 narrower layers outperforms wider 12L or 16L encoders on semantic GLUE and MTEB tasks while preserving full BERT-Base 768-dim checkpoint compatibility.
Interactive 12-Block Architecture Diagram (NeoBERT 250M & EuroBERT 210M–2.1B)
Select any scale (NeoBERT 250M, EuroBERT 210M, 610M, or 2.1B), filter by stage, and click any block inside the SVG pipeline to examine shapes, equations, and GQA/RoPE parameters.
Grouped-Query Attention (18Q : 6KV) in EuroBERT-610M & 2.1B
Click any KV Group below to inspect how EuroBERT ties 3 distinct Query heads onto each shared Key-Value head — cutting KV weights and memory bandwidth by 3x compared to standard Multi-Head Attention.
Pre-RMSNorm & Bias-Free SwiGLU Activation Lab
Both 2025 encoders adopt the LLaMA-style Pre-RMSNorm + SwiGLU pairing: normalizing purely by root-mean-square energy prior to each sub-layer and gating MLP channels via SiLU(x * W_gate) * (x * W_up).
Exact 6-Decimal Pure-JS Forward Trace (Token 1: "code" at pos=13)
Walk through every exact numerical step verified against Python FP64: Pre-RMSNorm, Dual-RoPE frequency divergence (10K vs 250K), 2:1 GQA shared Key-Value attention, SwiGLU gating, and MLM cross-entropy loss (L_MLM = 0.144502).
RoPE 4K / 8K Extension & Annealed Masking Rate Tradeoffs
Compare masking-rate regimes reported by NeoBERT (fixed 20%) and EuroBERT (15%–50% annealing ablations): higher masking biases encoders toward holistic retrieval (BEIR/MIRACL), whereas lower masking preserves sharp local NLI discrimination.
Multi-Trillion-Token Pretraining Curricula (2.1T & 5.0T)
Inspect how NeoBERT’s 2.1T RefinedWeb two-stage schedule and EuroBERT’s 4.8T + 200B annealing curriculum expand bidirectional capacity by over 600x relative to BERT’s original 3.3B words.
Complete Architecture & Scale Comparison Matrix
Full side-by-side comparison across 2018–2019 legacy encoders, ModernBERT, NeoBERT, and all three EuroBERT parameter tiers.
GLUE, MTEB Contrastive Embeddings & Multilingual Retrieval
Switch between NeoBERT’s English GLUE & MTEB SOTA leaderboards and EuroBERT’s multilingual + retrieval scaling across XLM-R baselines.
15+ Multilingual Coverage, Code & Math in 128K BPE
Explore EuroBERT’s cross-lingual and STEM domains enabled by adopting the 128,256-token LLaMA-3 vocabulary alongside 8,192 RoPE context.
Selecting Between NeoBERT, EuroBERT & ModernBERT
Pure-JS vs Python FP64 Verification Ledger
Every vector displayed in Section 06 is computed live in Pure JavaScript inside your browser and checked continuously against calibrated IEEE-754 Python FP64 reference tensors.