FlashAttention (FA-1 → FA-2 → FA-3)
Standard self-attention does not crawl because GPU math is slow — it crawls because it repeatedly writes and reads massive N × N matrices in slow off-chip HBM (1.5–2.0 TB/s) while ignoring blazing-fast 192 KB on-chip SRAM (~19 TB/s). By slicing Q, K, V into tiles and tracking running online-softmax statistics (m, l), FlashAttention computes 100% exact attention in linear O(N) activation memory — training GPT-2 3.5× faster, unlocking 64K long context, and scaling to 840 TFLOPs/s BF16 / 1.2 PFLOPs/s FP8 in FA-3.
What Makes FlashAttention Exact, Linear-Memory, and 3.5× Faster
How fast does the N × N attention matrix blow past an 80 GB GPU when sequence length grows from 512 to 65,536?
↓ 02 · O(N²) → O(N) SCALING & GEOMETRIC TILING PROOF02 THE QUADRATIC MEMORY & IO WALL
From O(N²) HBM OOM to O(N) SRAM Tiling
KV Cache (Lectures 21, 33, 34) shrinks persistent autoregressive K/V history during decode. Yet during prefill and training across N tokens, Standard Attention still materializes huge [B, H, N, N] score and probability matrices inside slow off-chip HBM. Explore the scaling cliff below.
O(N²) Standard vs O(N) Flash
How One Tiny SRAM Tile Replaces N×N
Watch actual HBM blocks stream into on-chip SRAM frame by frame across Standard, FlashAttention-1, and FlashAttention-2 — then explore the 4-Lens Studio.
↓ 03 · PURPLE KERNEL GPU FRAME ANIMATOR & 4-LENS STUDIO03 FRAME-BY-FRAME GPU ANIMATOR & STUDIO
Watch HBM ↔ SRAM Blocks Move Frame-by-Frame
Recreated frame-by-frame from Purple Kernel's GPU animation series: watch every single block leave slow off-chip HBM, fill tiny on-chip SRAM cells, execute exact tiled matmuls & online softmax, and see why swapping loops in FlashAttention-2 cuts HBM accesses from 70 (Std) & 80 (FA-1) down to 35 on N=5 — and from 240 & 160 down to 70 on N=10.
HBM (Emerald) ↔ SRAM Grid (Amber) · Live Block Access Simulator
Before diving into CUDA kernels, ground the core intuition with everyday analogies and the exact algebra of 1-pass softmax.
↓ 04 · ANALOGY, ONLINE SOFTMAX & RECOMPUTE04 INTUITION & MATHEMATICAL PROOF
Tiny Desk vs Warehouse, Safe Online Softmax & Recompute Paradox
Why can a single fast pass over tiles match a full global softmax bit-for-bit — and why does doing MORE FLOPs during backpropagation finish training 3.5× FASTER?
Now inspect the actual physical lanes between GPU HBM and on-chip SRAM — where every 10× speedup originates.
↓ 05 · GPU MEMORY HIERARCHY BRIDGE05 IO-AWARE GPU MEMORY HIERARCHY
Why Kernel Fusion Beats Fast Matmuls
An NVIDIA A100 delivers 312 TFLOPs/s of Tensor Core compute, yet its global HBM bandwidth is only 1.5–2.0 TB/s. Fusing QK^T, Mask, Softmax, Dropout, and PV inside ~19 TB/s on-chip SRAM turns a memory-bound bottleneck into pure compute saturation.
Prove the math for yourself by hand: trace 4 real tokens to 6 decimal places in both Full and Causal modes.
↓ 06 · 6-DECIMAL STEP-BY-STEP HAND TRACE06 CALIBRATED FP64 HAND TRACE
Prove Exact Parity: "The flash tiles fast" (N=4, B_r=2, B_c=2)
Every number below is calibrated in Python FP64 to match full standard 4×4 attention within 1.11e-16. Toggle Causal Masking to watch Step 2 (Q_0 × KV_1) skip cleanly with zero compute!
Watch how Tri Dao & collaborators evolved the kernel design across three hardware generations (Ampere → Hopper).
↓ 07 · FA-1 → FA-2 → FA-3 KERNEL ARCHITECTURE07 KERNEL EVOLUTION SCHEMATIC
Inside FA-1 → FA-2 → FA-3 Hardware Execution
Select a FlashAttention generation (or click any hardware block inside the schematic) to see how changing outer loops, Split-Q warps, and Hopper async TMA + WGMMA lifted MFU from 25% to 85%.
Calculate exact Algorithm 1 SRAM block dimensions (B_r, B_c), activation VRAM savings, and HBM IO cuts for your cluster.
↓ 08 · PRODUCTION VRAM & SRAM BLOCK CALCULATOR08 PRODUCTION VRAM & IO SIZER
Algorithm 1 SRAM Block Sizing & VRAM Calculator
In Algorithm 1 (Dao et al., 2022), SRAM capacity M (48–192 KB) directly determines column tile width B_c = ceil(M / 4d) and row tile height B_r = min(ceil(M / 4d), d). Test any model config below.
B_c = ceil(M_elems / 4d) | B_r = min(B_c, d) | Flash VRAM = 2BHNd + 4BHN
Verify every published training & long-context benchmark from the FA-1/FA-2/FA-3 papers, plus the 4 silent fallback traps engineers hit.
↓ 09 · VERIFIED BENCHMARKS & CODE PITFALLS09 BENCHMARKS & PRODUCTION PITFALLS
Published Speedups & 4 Silent Fallback Bugs
All metrics below come directly from Dao et al. (2022), Dao (2024), and Shah et al. (2024). Even with FlashAttention installed, four common tensor bugs silently drop back to slow O(N²) execution.
| Benchmark / System | Baseline | FlashAttention Result | Verified Gain |
|---|
Compare Standard Attention, FlashAttention-1, FlashAttention-2, and FlashAttention-3 side-by-side in one architectural ledger.
↓ 10 · ARCHITECTURAL EVOLUTION LEDGER10 SIDE-BY-SIDE LEDGER
Standard vs FA-1 vs FA-2 vs FA-3
A concise engineering comparison across HBM IO bounds, outer loop axes, warp partitioning, and peak A100/H100 FLOP utilization.
| Property | Standard Attention | FlashAttention-1 | FlashAttention-2 | FlashAttention-3 |
|---|
Inspect clean runnable PyTorch (.train(False)) & Keras 3 (layers.Layer, training=False) tiled FlashAttention plus production SDPA.
↓ 11 · PYTORCH & KERAS 3 CODE LAB11 CODE LAB · PYTORCH & KERAS 3
Clean PyTorch & Keras 3 Implementations
Five production modules covering pure-PyTorch tiled online softmax (.train(False)), backend-agnostic Keras 3 layers.Layer (training=False), 1D Milakov-Gimelshein proof, enforced PyTorch SDPA / flash_attn_func, and Algorithm 1 SRAM sizing.
Check your mastery across SRAM/HBM bandwidth, online softmax rescaling, backward recomputation, and silent SDPA fallback rules.
↓ 12 · MASTERY QUIZ12 MASTERY CHECK
FlashAttention Engineering Quiz
Eight scenario questions covering IO-awareness, O(N) activation storage, online softmax, FA-2 Split-Q warps, FA-3 Hopper TMA/WGMMA, and SDPA flags.