Reading a Transformer: A Method for Engineers
A repeatable method for analyzing transformer architectures: tokens, attention, FFNs, residuals — and the questions to ask at each layer of the stack.
Start from the tensor
Before reading prose about a model, write down what flows through it: token IDs become embeddings, embeddings become contextualized vectors, vectors become logits. Most architectural confusion is shape confusion. Track dimensions and the architecture usually explains itself.
Ask what attention is routing
Attention is content-based information routing. For any model, ask: what can each position see (masking), how many independent routing heads exist, and what does the FFN do with the routed information? Causal masking, cross-attention and expert routing are all variations of this single question.
Residuals are the architecture's spine
Deep transformers are ensembles of incremental refinements along a residual stream. Once you see the residual path as the main object and attention/FFN as writers to it, variants from ResNet to MoE stop looking unrelated. This is the lens ModLens applies to every diagram we publish.
Continue