Foundation · 2018– · autoregressive scaling
GPT-style Decoder
A stack of causal transformer blocks trained to predict the next token. At scale, this single objective yields general-purpose language capability.
ModLensFoundation
Interactive Diagram
Focus the lens
Click any component to read what it does. Signal direction follows the edges.
inputTokens. The sequence so far — the only input.
Core idea
Compression is comprehension: a model forced to predict the next token must internalize grammar, facts, reasoning patterns and style to minimize loss.
Why it exists
One simple objective, one architecture, scaled — turned out to be a path to broad capability rather than a narrow language model.
Mathematics
\mathcal{L} = -\sum_t \log P(x_t \mid x_{<t})
Data Flow
What moves through the system
- 01Tokens embed with causal masking.
- 02Each block attends only to earlier positions.
- 03The final hidden state scores the vocabulary.
- 04Sampling or decoding strategies produce text.
Strengths
- + Single unified objective
- + Natural interface for generation and tool use
- + Scaling behavior is well characterized
Limitations
- − Left-to-right only during training
- − Capability is emergent and hard to inspect
Applications
- · Assistants
- · Code generation
- · Reasoning loops
- · Agent cores