The Transformer — Attention Is All You Need

The Transformer — Attention Is All You Need An architecture diagram generated by Archify. Source Sentence · subword IDs · Architecture component Source Sentence subword IDs Embedding + Position · E[t] ⊕ PE · Encoder — reads everything at once Embedding + Position E[t] ⊕ PE Encoder Block · self-attn · FFN · Encoder — reads everything at once · × 6 Encoder Block self-attn · FFN × 6 K · V Memory · encoder output · Architecture component K · V Memory encoder output Partial Output · shifted right · Decoder — writes one token at a time Partial Output shifted right Output Embedding · same table ⊕ position · Decoder — writes one token at a time Output Embedding same table ⊕ position Decoder Block · self · cross · FFN · Decoder — writes one token at a time · × 6 Decoder Block self · cross · FFN × 6 Causal Mask · future = −∞ · Decoder — writes one token at a time Causal Mask future = −∞ Linear + Softmax · vocab probs · Architecture component Linear + Softmax vocab probs Next Token · append & rerun · Architecture component Next Token append & rerun token IDs K · V → cross-attn P(next) shifted right blocks the future Encoder — reads everything at once Decoder — writes one token at a time Legend Backend Security Message bus External

The 2017 recipe

  • • No recurrence — attention is the only mixer
  • • Positions injected once, at the very bottom
  • • Residual + LayerNorm rails around every sublayer

Why it won

  • • Every word attends to every word in one step (O(1) path)
  • • The whole sequence trains in parallel
  • • Scaling this exact shape produced BERT, GPT and T5

The honest bill

  • • O(n²) compute and memory per attention layer
  • • Generation stays one token per decoder pass
  • • Later research (FlashAttention, KV-cache) negotiates that bill