TinyStories ARM
Built from scratch, trained on a Mac

A GPT and a diffusion language model, from scratch

A small GPT-style language model written in PyTorch and trained on Apple Silicon (and on Colab when it gets big), with a local web app that shows every stage: data, tokens, attention, training, generation and evaluation. Next to it is a masked diffusion language model of the same size, trained on the same 2.5B tokens. The two are compared on the same footing.

Try itChat with the 124M model, or let the base model continue your text, running in your browser: nothing is sent anywhere.JourneyEvery decision from the first commit to today: what changed, why, what it measured and what it cost.ExplorerOne prompt through the real 124M model: every block's attention, the feed-forward layer and the logit lens.Side by sideThe same prompt written left to right by the GPT and all at once by the diffusion model, replayed in your browser.The storyFrom a 356-line training script to a 124M model that beats GPT-2 small on held-out web text.How to use itInstall, run the local web app, train on your Mac's GPU or on Colab, evaluate and compare.Understanding the componentsData, tokenization, batching, the transformer, attention, training and inference, one at a time.The planThe final goals and the roadmap: what's done, what's next and who does it.

Where the models stand

zero-shot accuracy, %; from the results store
HellaSwagLAMBADAARC-EasyPIQAWinograndeAverage
GPT-2 small (OpenAI, published)
the yardstick
31.132.643.862.951.644.4
Autoregressive 124M (the base model)32.3
+1.1
20.3
−12.2
46.8
+3.0
62.4
≈ level
49.9
−1.7
42.3
−2.1
Diffusion 124M, 2.5B tokens28.0
−3.1
14.4
−18.2
30.1
−13.7
55.4
−7.4
49.1
−2.5
35.4
−9.0
Diffusion 124M, 1B tokens27.6
−3.6
11.8
−20.8
30.7
−13.1
54.2
−8.6
49.6
−2.1
34.8
−9.6

Against GPT-2 small

  • Autoregressive 124M (the base model): 2.1 points behind GPT-2 small on average. Ahead on HellaSwag (+1.1), ARC-Easy (+3.0); level on PIQA; behind on LAMBADA (−12.2), Winogrande (−1.7).
  • Diffusion 124M, 2.5B tokens: 9.0 points behind GPT-2 small on average. Behind on HellaSwag (−3.1), LAMBADA (−18.2), ARC-Easy (−13.7), PIQA (−7.4), Winogrande (−2.5).
  • Diffusion 124M, 1B tokens: 9.6 points behind GPT-2 small on average. Behind on HellaSwag (−3.6), LAMBADA (−20.8), ARC-Easy (−13.1), PIQA (−8.6), Winogrande (−2.1).

Under each score, the difference from GPT-2 small in points: green ahead, red behind, ≈ level within 1 point, about each benchmark's margin of error. GPT-2 small is OpenAI's 124M model from 2019, trained on 40 GB of web text; our models have the same size. Scored the way EleutherAI's lm-evaluation-harness scores them; the diffusion models score each answer by its likelihood bound. All of it: Results.