CAS Logo CAS-NLP · L35 FlashAttention ← Hub
CAS-NLP SERIES · LECTURE 35 · DAO ET AL. (NEURIPS 2022) · ARXIV:2205.14135

FlashAttention (FA-1 → FA-2 → FA-3)

Standard self-attention does not crawl because GPU math is slow — it crawls because it repeatedly writes and reads massive N × N matrices in slow off-chip HBM (1.5–2.0 TB/s) while ignoring blazing-fast 192 KB on-chip SRAM (~19 TB/s). By slicing Q, K, V into tiles and tracking running online-softmax statistics (m, l), FlashAttention computes 100% exact attention in linear O(N) activation memory — training GPT-2 3.5× faster, unlocking 64K long context, and scaling to 840 TFLOPs/s BF16 / 1.2 PFLOPs/s FP8 in FA-3.

O(N²) → O(N)Activation MemoryZero N×N S or P in HBM
1.11e-16FP64 Math Error100% Exact vs Std Attn
3.5× GPT-2Wall-Clock Training9.5 days → 2.7 days (8×A100)
225 → 840 TFFA-2 → FA-3 BF16 Peak72% A100 → 85% H100 MFU
61.4% / 63.1%Path-X / Path-256First Transformer at 16K–64K
SIX BREAKTHROUGHS · CLICK ANY CARD TO JUMP STRAIGHT IN

What Makes FlashAttention Exact, Linear-Memory, and 3.5× Faster

How fast does the N × N attention matrix blow past an 80 GB GPU when sequence length grows from 512 to 65,536?

↓ 02 · O(N²) → O(N) SCALING & GEOMETRIC TILING PROOF

02 THE QUADRATIC MEMORY & IO WALL

From O(N²) HBM OOM to O(N) SRAM Tiling

KV Cache (Lectures 21, 33, 34) shrinks persistent autoregressive K/V history during decode. Yet during prefill and training across N tokens, Standard Attention still materializes huge [B, H, N, N] score and probability matrices inside slow off-chip HBM. Explore the scaling cliff below.

SELECT PRODUCTION MODEL PROFILE OR ADJUST SEQUENCE LENGTH (B=4, H=32, FP16)
Sequence Length N 8,192 tokens
Head Dimension d_head 128 d_head
INTERACTIVE MEMORY CLIFF

O(N²) Standard vs O(N) Flash

GEOMETRIC TILING PROOF

How One Tiny SRAM Tile Replaces N×N

Watch actual HBM blocks stream into on-chip SRAM frame by frame across Standard, FlashAttention-1, and FlashAttention-2 — then explore the 4-Lens Studio.

↓ 03 · PURPLE KERNEL GPU FRAME ANIMATOR & 4-LENS STUDIO

03 FRAME-BY-FRAME GPU ANIMATOR & STUDIO

Watch HBM ↔ SRAM Blocks Move Frame-by-Frame

Recreated frame-by-frame from Purple Kernel's GPU animation series: watch every single block leave slow off-chip HBM, fill tiny on-chip SRAM cells, execute exact tiled matmuls & online softmax, and see why swapping loops in FlashAttention-2 cuts HBM accesses from 70 (Std) & 80 (FA-1) down to 35 on N=5 — and from 240 & 160 down to 70 on N=10.

PART A · PURPLE KERNEL FRAME-BY-FRAME GPU MEMORY ANIMATOR

HBM (Emerald) ↔ SRAM Grid (Amber) · Live Block Access Simulator

Frame 1 / 1
PHASE Params

PART B · 4-LENS X-RAY STUDIO · SELECT MODEL & TILE SCALE

Before diving into CUDA kernels, ground the core intuition with everyday analogies and the exact algebra of 1-pass softmax.

↓ 04 · ANALOGY, ONLINE SOFTMAX & RECOMPUTE

04 INTUITION & MATHEMATICAL PROOF

Tiny Desk vs Warehouse, Safe Online Softmax & Recompute Paradox

Why can a single fast pass over tiles match a full global softmax bit-for-bit — and why does doing MORE FLOPs during backpropagation finish training 3.5× FASTER?

Now inspect the actual physical lanes between GPU HBM and on-chip SRAM — where every 10× speedup originates.

↓ 05 · GPU MEMORY HIERARCHY BRIDGE

05 IO-AWARE GPU MEMORY HIERARCHY

Why Kernel Fusion Beats Fast Matmuls

An NVIDIA A100 delivers 312 TFLOPs/s of Tensor Core compute, yet its global HBM bandwidth is only 1.5–2.0 TB/s. Fusing QK^T, Mask, Softmax, Dropout, and PV inside ~19 TB/s on-chip SRAM turns a memory-bound bottleneck into pure compute saturation.

Prove the math for yourself by hand: trace 4 real tokens to 6 decimal places in both Full and Causal modes.

↓ 06 · 6-DECIMAL STEP-BY-STEP HAND TRACE

06 CALIBRATED FP64 HAND TRACE

Prove Exact Parity: "The flash tiles fast" (N=4, B_r=2, B_c=2)

Every number below is calibrated in Python FP64 to match full standard 4×4 attention within 1.11e-16. Toggle Causal Masking to watch Step 2 (Q_0 × KV_1) skip cleanly with zero compute!

Watch how Tri Dao & collaborators evolved the kernel design across three hardware generations (Ampere → Hopper).

↓ 07 · FA-1 → FA-2 → FA-3 KERNEL ARCHITECTURE

07 KERNEL EVOLUTION SCHEMATIC

Inside FA-1 → FA-2 → FA-3 Hardware Execution

Select a FlashAttention generation (or click any hardware block inside the schematic) to see how changing outer loops, Split-Q warps, and Hopper async TMA + WGMMA lifted MFU from 25% to 85%.

SELECT KERNEL GENERATION
HIGHLIGHT KERNEL STAGE (CLICK ANY BOX BELOW TO INSPECT)

Calculate exact Algorithm 1 SRAM block dimensions (B_r, B_c), activation VRAM savings, and HBM IO cuts for your cluster.

↓ 08 · PRODUCTION VRAM & SRAM BLOCK CALCULATOR

08 PRODUCTION VRAM & IO SIZER

Algorithm 1 SRAM Block Sizing & VRAM Calculator

In Algorithm 1 (Dao et al., 2022), SRAM capacity M (48–192 KB) directly determines column tile width B_c = ceil(M / 4d) and row tile height B_r = min(ceil(M / 4d), d). Test any model config below.

1 · FRONTIER MODEL PROFILES
Algorithm 1 Equations: B_c = ceil(M_elems / 4d) | B_r = min(B_c, d) | Flash VRAM = 2BHNd + 4BHN
Standard O(N²) S + P Activation VRAM100% Baseline
FlashAttention O(N) O + L VRAMLinear Cut
Sequence Length (N)8,192 tokens
Attention Heads (H)32 heads
Head Dimension (d_head)128 d_head
Batch Size (B)4 batch
On-Chip SRAM per SM (M)192 KB SRAM

Verify every published training & long-context benchmark from the FA-1/FA-2/FA-3 papers, plus the 4 silent fallback traps engineers hit.

↓ 09 · VERIFIED BENCHMARKS & CODE PITFALLS

09 BENCHMARKS & PRODUCTION PITFALLS

Published Speedups & 4 Silent Fallback Bugs

All metrics below come directly from Dao et al. (2022), Dao (2024), and Shah et al. (2024). Even with FlashAttention installed, four common tensor bugs silently drop back to slow O(N²) execution.

Benchmark / System Baseline FlashAttention Result Verified Gain
DEBUG 4 PRODUCTION PITFALLS:

Compare Standard Attention, FlashAttention-1, FlashAttention-2, and FlashAttention-3 side-by-side in one architectural ledger.

↓ 10 · ARCHITECTURAL EVOLUTION LEDGER

10 SIDE-BY-SIDE LEDGER

Standard vs FA-1 vs FA-2 vs FA-3

A concise engineering comparison across HBM IO bounds, outer loop axes, warp partitioning, and peak A100/H100 FLOP utilization.

Property Standard Attention FlashAttention-1 FlashAttention-2 FlashAttention-3

Inspect clean runnable PyTorch (.train(False)) & Keras 3 (layers.Layer, training=False) tiled FlashAttention plus production SDPA.

↓ 11 · PYTORCH & KERAS 3 CODE LAB

11 CODE LAB · PYTORCH & KERAS 3

Clean PyTorch & Keras 3 Implementations

Five production modules covering pure-PyTorch tiled online softmax (.train(False)), backend-agnostic Keras 3 layers.Layer (training=False), 1D Milakov-Gimelshein proof, enforced PyTorch SDPA / flash_attn_func, and Algorithm 1 SRAM sizing.

Check your mastery across SRAM/HBM bandwidth, online softmax rescaling, backward recomputation, and silent SDPA fallback rules.

↓ 12 · MASTERY QUIZ

12 MASTERY CHECK

FlashAttention Engineering Quiz

Eight scenario questions covering IO-awareness, O(N) activation storage, online softmax, FA-2 Split-Q warps, FA-3 Hopper TMA/WGMMA, and SDPA flags.

CURRENT SCORE
0 / 8