A GPT and a diffusion language model, from scratch
A small GPT-style language model written in PyTorch and trained on Apple Silicon (and on Colab when it gets big), with a local web app that shows every stage: data, tokens, attention, training, generation and evaluation. Next to it is a masked diffusion language model of the same size, trained on the same 2.5B tokens. The two are compared on the same footing.
Where the models stand
zero-shot accuracy, %; from the results store| HellaSwag | LAMBADA | ARC-Easy | PIQA | Winogrande | Average | |
|---|---|---|---|---|---|---|
| GPT-2 small (OpenAI, published) the yardstick | 31.1 | 32.6 | 43.8 | 62.9 | 51.6 | 44.4 |
| Autoregressive 124M (the base model) | 32.3 +1.1 | 20.3 −12.2 | 46.8 +3.0 | 62.4 ≈ level | 49.9 −1.7 | 42.3 −2.1 |
| Diffusion 124M, 2.5B tokens | 28.0 −3.1 | 14.4 −18.2 | 30.1 −13.7 | 55.4 −7.4 | 49.1 −2.5 | 35.4 −9.0 |
| Diffusion 124M, 1B tokens | 27.6 −3.6 | 11.8 −20.8 | 30.7 −13.1 | 54.2 −8.6 | 49.6 −2.1 | 34.8 −9.6 |
Against GPT-2 small
- Autoregressive 124M (the base model): 2.1 points behind GPT-2 small on average. Ahead on HellaSwag (+1.1), ARC-Easy (+3.0); level on PIQA; behind on LAMBADA (−12.2), Winogrande (−1.7).
- Diffusion 124M, 2.5B tokens: 9.0 points behind GPT-2 small on average. Behind on HellaSwag (−3.1), LAMBADA (−18.2), ARC-Easy (−13.7), PIQA (−7.4), Winogrande (−2.5).
- Diffusion 124M, 1B tokens: 9.6 points behind GPT-2 small on average. Behind on HellaSwag (−3.6), LAMBADA (−20.8), ARC-Easy (−13.1), PIQA (−8.6), Winogrande (−2.1).
Under each score, the difference from GPT-2 small in points: green ahead, red behind, ≈ level within 1 point, about each benchmark's margin of error. GPT-2 small is OpenAI's 124M model from 2019, trained on 40 GB of web text; our models have the same size. Scored the way EleutherAI's lm-evaluation-harness scores them; the diffusion models score each answer by its likelihood bound. All of it: Results.