TinyStories ARM
Explore

Side by side: left to right, or all at once

The same prompt written by the two kinds of model. The autoregressive GPT writes one token per pass, left to right. The masked diffusion model starts from a continuation that is all [MASK] (█) and, at every pass, predicts every masked position at once and reveals a few. Both have 124M parameters and were trained on the same 2.5B tokens with the same tokenizer: only the training objective differs. These runs were recorded on an Apple M5 (2026-11-17 20:10); the app writes new ones live.

One prompt, two models

Autoregressive: one token per pass

Which position each pass writes

rows: passes · columns: positions

Diffusion: every position, every pass

When each position is revealed

rows: passes · columns: positions

Both grids use the same height per pass, so the taller one took more passes. Left to right draws a staircase; the diffusion model fills its positions block by block, the most confident first. The colour is the model's probability for the token it chose. Lined up by clock time, each lane moves at its recorded speed.

The numbers

the better of the two in bold where lower or higher is better

What differs, and what doesn't

same tokenizer, same kind of transformer blocks