TinyStories ARM

From docs/results.md

Results

Generated by python3 -m infra.results render from results/results.jsonl, the one results store: every evaluation adds its numbers there. Don't edit this page by hand. The latest measurement of each model is shown; the store keeps the earlier ones, with when and at which commit each was measured.

Benchmarks

Zero-shot, lm-evaluation-harness style, in percent: HellaSwag acc_norm, LAMBADA acc, ARC-Easy acc, PIQA acc, Winogrande acc. Diffusion models score each choice by its likelihood bound. Average over the five tasks when all were run.

Model HellaSwag LAMBADA ARC-Easy PIQA Winogrande Average Measured
Chance 25.0 0.0 25.0 50.0 50.0 30.0
OpenAI GPT-2 small (published) 31.1 32.6 43.8 62.9 51.6 44.4
Medium 51M · TinyStories · 20k steps 26.5 8.7 25.9 54.1 51.6 33.4 2026-11-14
OpenAI GPT-2 small (124M, 2019) 31.1 32.6 43.8 62.9 51.6 44.4 2026-11-12
Large 124M RoPE SwiGLU no-bias QK-norm · Mini FineWeb-Edu · 30k steps 27.9 9.6 42.1 58.4 50.6 37.7 2026-11-12
Large 124M best design · FineWeb-Edu + Cosmopedia · 500M tokens · context 512 29.7 15.7 44.0 62.6 50.0 40.4 2026-11-13
Diffusion 51M · TinyStories · 164M tokens 25.7 1.7 25.7 51.9 48.4 30.7 2026-11-14
Diffusion 51M · Mini FineWeb-Edu · 164M tokens 25.3 0.2 31.2 52.1 50.5 31.9 2026-11-14
Diffusion 51M · Mini FineWeb-Edu · 492M tokens 26.6 2.2 29.2 52.2 51.9 32.4 2026-11-14
Chat Large 124M RoPE SwiGLU no-bias QK-norm · instruction-tuned on smol-smoltalk · 11.284k steps 28.9 10.6 38.1 58.5 50.4 37.3 2026-11-14
Chat Large 124M RoPE SwiGLU no-bias QK-norm · instruction-tuned on smol-smoltalk + DPO against repetition · 11.284k steps 29.3 9.5 38.5 58.5 50.4 37.2 2026-11-14
Large 124M best design · stage 2 · FineWeb-Edu full + Cosmopedia · 2.5B tokens · context 512 32.3 20.3 46.8 62.4 49.9 42.3 2026-11-15
Chat Large 124M · stage 2 base · instruction-tuned on smol-smoltalk · 11.284k steps 30.1 13.5 41.8 59.1 47.9 38.5 2026-11-15
Chat Large 124M · stage 2 base · instruction-tuned on smol-smoltalk + DPO against repetition 30.1 12.9 42.0 59.0 47.9 38.4 2026-11-15
Diffusion 124M · FineWeb-Edu full + Cosmopedia · 999M tokens 27.6 11.8 30.7 54.2 49.6 34.8 2026-11-16
Diffusion 124M · FineWeb-Edu full + Cosmopedia · 2,500M tokens 28.0 14.4 30.1 55.4 49.1 35.4 2026-11-17
Chat Large 124M · stage 2 base · instruction-tuned on smol-smoltalk, lr 5e-5 + replay 0.2 · 13.888k steps 32.5 16.0 47.5 62.2 48.9 41.4 2026-11-18
Chat Large 124M · stage 2 base · SFT (lr 5e-5 + replay 0.2) + DPO · 948 steps 32.7 15.0 47.9 62.1 49.4 41.4 2026-11-18
Chat Large 124M · stage 2 base · SFT (lr 5e-5 + replay 0.2) + DPO · 948 steps · RM pairs DPO 33.0 14.4 48.5 61.9 48.9 41.3 2026-11-19

Held-out loss

Cross-entropy (nats per token) on held-out text from evaluate.py; lower is better. One table per set of data sets the models were measured on (the suite grew as new data was added), newest first.

Model Cosmopedia khanacademy Cosmopedia openstax Cosmopedia stanford-first100000 Cosmopedia stories-first300000 Cosmopedia wikihow Mini FineWeb-Edu Mini FineWeb-Edu tiny Tiny Shakespeare TinyStories WikiText-2 Measured
Large 124M best design · stage 2 · FineWeb-Edu full + Cosmopedia · 2.5B tokens · context 512 2.336 2.717 2.641 2.785 2.342 3.296 3.310 5.446 2.751 5.047 2026-11-15
Model Mini FineWeb-Edu Tiny Shakespeare TinyStories WikiText-2 Measured
OpenAI GPT-2 small (124M, 2019) 3.401 4.807 2.553 4.180 2026-11-11
GPT-2 style · Mini FineWeb-Edu · 5,000 updates · NVIDIA L4 4.580 6.296 4.682 6.528 2026-11-11
Our best, dense · Mini FineWeb-Edu · 5,000 updates · NVIDIA L4 4.301 6.138 4.465 6.276 2026-11-11
Large 124M · Mini FineWeb-Edu · 30k steps 3.735 5.823 3.872 5.582 2026-11-11
Our best + experts · Mini FineWeb-Edu · 5,000 updates · NVIDIA L4 4.245 6.121 4.398 6.240 2026-11-11
Large 124M RoPE SwiGLU no-bias QK-norm · Mini FineWeb-Edu · 30k steps 3.691 5.753 3.856 5.605 2026-11-12
Large 124M best design · FineWeb-Edu + Cosmopedia · 500M tokens · context 512 3.549 5.614 2.790 5.370 2026-11-13
Model TinyStories Measured
I5 drafter: Tiny best design · tiny-best · tiny rope layernorm swiglu no-bias qk-norm lr=0.0018 · 2,000 updates 2.063 2026-11-12
Model Mini FineWeb-Edu Tiny Shakespeare WikiText-2 Measured
Large 124M, best design · lr 6e-4 · Mini FineWeb-Edu · 1,000 updates · NVIDIA L4 5.172 6.458 7.136 2026-11-12
Large 124M, best design · lr 1.2e-3 · Mini FineWeb-Edu · 1,000 updates · NVIDIA L4 5.155 6.482 7.138 2026-11-12
Large 124M, best design · lr 1.8e-3 · Mini FineWeb-Edu · 1,000 updates · NVIDIA L4 5.148 6.467 7.140 2026-11-12
Model Tiny Shakespeare TinyStories WikiText-2 Measured
Medium 51M · TinyStories · 20k steps 7.632 1.561 8.693 2026-11-10
Tiny 16M · WikiText · 40 steps 10.783 10.701 10.547 2026-11-10
Small 30M · TinyStories · 2k steps 8.100 4.620 9.765 2026-11-10
Small 30M · TinyStories + Tiny Shakespeare · 1k steps 7.047 6.382 9.375 2026-11-10
Large 124M · TinyStories · 2k steps 8.495 4.510 9.842 2026-11-10
A1 baseline (learned) · Small 30M · TinyStories · 1k steps 8.149 2.602 9.569 2026-11-10
A1 RoPE · Small 30M · TinyStories · 1k steps 8.040 2.440 9.461 2026-11-10
A5 QK-norm · baseline · small rope layernorm swiglu no-bias lr=0.0006 · 1,000 updates · NVIDIA L4 8.216 2.329 9.456 2026-11-11
A5 QK-norm · qk-norm · small rope layernorm swiglu no-bias qk-norm lr=0.0006 · 1,000 updates · NVIDIA L4 8.016 2.300 9.461 2026-11-11
A5 QK-norm · baseline-3x · small rope layernorm swiglu no-bias lr=0.0018 · 1,000 updates · NVIDIA L4 8.365 2.230 9.900 2026-11-11
LR sweep · base-3e-3 · small rope layernorm swiglu no-bias lr=0.003 · 1,000 updates · NVIDIA L4 8.611 2.621 10.399 2026-11-11
LR sweep · qk-3e-3 · small rope layernorm swiglu no-bias qk-norm lr=0.003 · 1,000 updates · NVIDIA L4 8.510 2.339 9.794 2026-11-11
LR sweep · base-6e-3 · small rope layernorm swiglu no-bias lr=0.006 · 1,000 updates · NVIDIA L4 9.029 2.956 10.692 2026-11-11
LR sweep · qk-6e-3 · small rope layernorm swiglu no-bias qk-norm lr=0.006 · 1,000 updates · NVIDIA L4 8.503 2.512 9.889 2026-11-11
LR sweep · base-1.2e-2 · small rope layernorm swiglu no-bias lr=0.012 · 1,000 updates · NVIDIA L4 9.560 3.281 10.665 2026-11-11
LR sweep · qk-1.2e-2 · small rope layernorm swiglu no-bias qk-norm lr=0.012 · 1,000 updates · NVIDIA L4 8.713 2.517 10.043 2026-11-11
A6 GQA · kv3 · small rope layernorm swiglu no-bias qk-norm kv=3 lr=0.0018 · 1,000 updates · NVIDIA L4 8.379 2.212 9.863 2026-11-11
A6 GQA · kv2 · small rope layernorm swiglu no-bias qk-norm kv=2 lr=0.0018 · 1,000 updates · NVIDIA L4 8.484 2.226 9.973 2026-11-11
A6 GQA · kv1 · small rope layernorm swiglu no-bias qk-norm kv=1 lr=0.0018 · 1,000 updates · NVIDIA L4 8.395 2.248 10.077 2026-11-11
A7 depth vs width · wide-3x448 · 3×448 (7 heads) rope layernorm swiglu no-bias qk-norm lr=0.0018 · 1,000 updates · NVIDIA L4 8.403 2.235 9.896 2026-11-11
A7 depth vs width · deep-11x320 · 11×320 (5 heads) rope layernorm swiglu no-bias qk-norm lr=0.0018 · 1,000 updates · NVIDIA L4 8.391 2.233 10.061 2026-11-11
A7 depth vs width · deeper-22x256 · 22×256 (4 heads) rope layernorm swiglu no-bias qk-norm lr=0.0018 · 1,000 updates · NVIDIA L4 8.380 2.360 9.716 2026-11-11
A8 layer sharing · share2 · small rope layernorm swiglu no-bias qk-norm repeat=2 lr=0.0018 · 1,000 updates · NVIDIA L4 8.316 2.311 9.924 2026-11-11
A8 layer sharing · share3 · small rope layernorm swiglu no-bias qk-norm repeat=3 lr=0.0018 · 1,000 updates · NVIDIA L4 8.480 2.481 9.802 2026-11-11
A8 layer sharing · wide-share2 · 3×448 (7 heads) rope layernorm swiglu no-bias qk-norm repeat=2 lr=0.0018 · 1,000 updates · NVIDIA L4 8.400 2.311 9.918 2026-11-11
A9 multi-token prediction · mtp2 · small rope layernorm swiglu no-bias qk-norm mtp=2 lr=0.0018 · 1,000 updates · NVIDIA L4 8.412 2.302 10.034 2026-11-11
A9 multi-token prediction · mtp3 · small rope layernorm swiglu no-bias qk-norm mtp=3 lr=0.0018 · 1,000 updates · NVIDIA L4 8.456 2.454 10.285 2026-11-11
A9 multi-token prediction · mtp4 · small rope layernorm swiglu no-bias qk-norm mtp=4 lr=0.0018 · 1,000 updates · NVIDIA L4 8.610 2.494 10.258 2026-11-11
A5 QK-norm · qk-norm-3x · small rope layernorm swiglu no-bias qk-norm lr=0.0018 · 1,000 updates · NVIDIA L4 8.360 2.219 9.938 2026-11-11
A10 mixture of experts · moe4-top1 · small rope layernorm swiglu no-bias qk-norm experts=4 topk=1 lr=0.0018 · 1,000 updates · NVIDIA L4 8.320 2.361 9.996 2026-11-11
A10 mixture of experts · moe8-top1 · small rope layernorm swiglu no-bias qk-norm experts=8 topk=1 lr=0.0018 · 1,000 updates · NVIDIA L4 8.390 2.435 10.027 2026-11-11
A10 mixture of experts · moe4-top2 · small rope layernorm swiglu no-bias qk-norm experts=4 topk=2 lr=0.0018 · 1,000 updates · NVIDIA L4 8.409 2.171 9.899 2026-11-11
Final: GPT-2 style vs all the wins · gpt2-style · small learned layernorm gelu lr=0.0006 · 5,000 updates · NVIDIA L4 8.342 1.878 9.508 2026-11-11
Final: GPT-2 style vs all the wins · gpt2-style-lr · small learned layernorm gelu lr=0.0018 · 5,000 updates · NVIDIA L4 8.344 1.820 9.792 2026-11-11
Final: GPT-2 style vs all the wins · best-dense · small rope layernorm swiglu no-bias qk-norm lr=0.0018 · 5,000 updates · NVIDIA L4 8.300 1.754 9.777 2026-11-11
A2 RoPE + RMSNorm · Small 30M · TinyStories · 1k steps 8.037 2.451 9.454 2026-11-11
Final: GPT-2 style vs all the wins · best-moe · small rope layernorm swiglu no-bias qk-norm experts=4 topk=2 lr=0.0018 · 5,000 updates · NVIDIA L4 8.194 1.700 9.597 2026-11-11
A3 RoPE + SwiGLU · Small 30M · TinyStories · 1k steps 8.019 2.338 9.501 2026-11-11
A4 RoPE + SwiGLU, no biases · Small 30M · TinyStories · 1k steps 8.093 2.324 9.482 2026-11-11
A5 QK-norm · RoPE SwiGLU no-bias · Small 30M · TinyStories · 1k steps 7.980 2.299 9.441 2026-11-11

Chat

sft.py compare: loss on held-out smol-smoltalk answers, and greedy answers to the fixed questions (repeated 4-grams, share that end by themselves). dpo.py eval: the same on 100 held-out questions, with the loss on their reference answers.

Model Answer loss Repetition Ended Held-out repetition Held-out ended Reference loss Measured
Large 124M best design · FineWeb-Edu + Cosmopedia · 500M tokens · context 512 2.437 29.4 0.0 2026-11-14
Chat Large 124M RoPE SwiGLU no-bias QK-norm · instruction-tuned on smol-smoltalk · 11.284k steps 1.548 27.5 25.0 20.0 59.0 1.312 2026-11-15
outputs/dpo/smoltalk-2000 1.617 16.1 50.0 11.2 84.0 1.399 2026-11-14
Chat Large 124M RoPE SwiGLU no-bias QK-norm · instruction-tuned on smol-smoltalk + DPO against repetition · 11.284k steps 11.2 84.0 1.399 2026-11-15
Large 124M best design · stage 2 · FineWeb-Edu full + Cosmopedia · 2.5B tokens · context 512 2.329 27.8 0.0 2026-11-15
Chat Large 124M · stage 2 base · instruction-tuned on smol-smoltalk · 11.284k steps 1.406 22.3 16.7 12.4 70.0 1.156 2026-11-18
Chat Large 124M · stage 2 base · instruction-tuned on smol-smoltalk + DPO against repetition 1.459 9.9 50.0 7.5 85.0 1.163 2026-11-19
Chat Large 124M · stage 2 base · instruction-tuned on smol-smoltalk, lr 5e-5 + replay 0.2 · 13.888k steps 1.456 20.9 8.3 16.0 69.0 1.230 2026-11-19
Chat Large 124M · stage 2 base · SFT (lr 5e-5 + replay 0.2) + DPO · 948 steps 8.5 87.0 1.333 2026-11-19
Chat Large 124M · stage 2 base · SFT (lr 5e-5 + replay 0.2) + DPO · 948 steps · RM pairs DPO 2.3 99.0 1.425 2026-11-19
Chat Large 124M · stage 2 base · SFT (lr 5e-5 + replay 0.2) + DPO · 948 steps · RM pairs SimPO 2.2 99.0 1.474 2026-11-19

Diffusion models

Best likelihood bound on held-out text during training (nats per token, an upper bound on the loss), and diffusion_models/quality.py: 64-token continuations judged by an autoregressive model (its loss on the text; lower reads as more natural) and the share of repeated 4-grams, per sampler.

Model Bound Tokens Sampler Judge loss Repetition Distinct 2-grams
Diffusion 51M · TinyStories · 164M tokens 2.525 164M base AR model, top-k 50 (judge's own text) 2.001 0.6 97.0
real continuation 2.675 0.8 93.9
keep (confident first), top-k 50 2.47 6.2 78.9
keep, blocks of 4 3.057 2.5 86.8
keep, blocks of 8, rep 1.2 3.27 0.2 94.9
keep, distinct, 16 steps 3.168 4.3 81.8
keep, rep 1.2, blocks of 16 2.995 0.6 94.8
keep, repetition 1.3 2.797 0.4 94.8
Diffusion 51M · Mini FineWeb-Edu · 164M tokens 5.269 164M base AR model, top-k 50 (judge's own text) 2.34 0.6 94.6
real continuation 3.793 0.0 97.2
keep (confident first), top-k 50 2.326 42.8 35.7
keep, blocks of 16 2.8 23.9 46.6
keep, blocks of 8, rep 1.3 4.131 0.1 95.9
keep, noise level from masks 2.614 38.4 37.1
keep, remask 0.1 3.433 28.9 44.6
keep, repetition 1.3 3.622 3.8 85.1
random order, temp 0.8 5.269 0.0 95.1
random order, temp 1.0 6.933 0.0 99.4
Diffusion 51M · Mini FineWeb-Edu · 492M tokens 4.729 492M base AR model, top-k 50 (judge's own text) 2.34 0.6 94.6
real continuation 3.793 0.0 97.2
keep (confident first), top-k 50 2.235 37.5 42.8
keep, 16 steps 2.85 32.9 41.3
keep, blocks of 16 2.697 20.0 55.7
keep, blocks of 4 3.277 7.6 75.8
keep, blocks of 4, rep 1.2 3.797 0.9 92.2
keep, blocks of 8, rep 1.2 3.444 1.1 90.1
keep, blocks of 8, rep 1.3 3.692 0.5 93.5
keep, distinct 2.235 37.5 42.8
keep, distinct, 16 steps 3.031 9.6 63.8
keep, distinct, blocks of 16 2.697 20.0 55.7
keep, distinct, rep 1.1 2.3 27.4 53.1
keep, distinct, rep 1.2 2.731 11.4 74.0
keep, distinct, rep 1.2, blocks of 16 3.016 8.4 81.8
keep, noise level from masks 2.132 44.0 37.8
keep, remask 0.1 3.72 21.2 56.8
keep, rep 1.15 2.384 24.9 57.8
keep, rep 1.2 2.731 11.4 74.0
keep, rep 1.2, 128 steps 2.838 9.1 73.6
keep, rep 1.2, blocks of 16 3.016 8.4 81.8
keep, rep 1.2, blocks of 32 2.889 8.8 77.6
keep, rep 1.2, noise from masks 2.704 12.7 72.0
keep, rep 1.2, remask 0.1 4.199 8.2 77.6
keep, rep 1.2, temp 0.8 2.557 14.9 68.8
keep, rep 1.2, top-p 0.9 2.531 19.5 64.7
keep, rep 1.5 3.503 6.2 89.8
keep, repetition 1.3 3.014 6.1 84.4
left to right (blocks of 1) 4.17 0.9 90.0
left to right, rep 1.1 4.455 0.0 97.1
left to right, rep 1.2 4.815 0.1 98.5
random order, temp 0.8 3.616 2.1 85.9
random order, temp 1.0 4.207 0.2 92.8
Diffusion 124M · FineWeb-Edu full + Cosmopedia · 999M tokens 3.923 999M base AR model, top-k 50 (judge's own text) 2.073 1.2 93.4
real continuation 3.063 0.4 95.5
keep (confident first), top-k 50 2.155 24.5 59.7
keep, 16 steps 2.776 30.2 53.2
keep, blocks of 16 2.246 21.0 64.9
keep, blocks of 4 2.559 22.2 64.2
keep, blocks of 4, rep 1.2 3.419 3.2 90.6
keep, blocks of 8, rep 1.2 3.16 2.2 91.8
keep, blocks of 8, rep 1.3 3.233 2.8 93.7
keep, distinct 2.155 24.5 59.7
keep, distinct, 16 steps 2.687 13.8 67.8
keep, distinct, blocks of 16 2.246 21.0 64.9
keep, distinct, rep 1.1 2.293 15.5 72.8
keep, distinct, rep 1.2 2.473 10.6 79.6
keep, distinct, rep 1.2, blocks of 16 2.745 4.6 87.3
keep, noise level from masks 2.054 28.1 55.6
keep, remask 0.1 2.583 26.6 57.9
keep, rep 1.15 2.424 11.7 76.1
keep, rep 1.2 2.473 10.6 79.6
keep, rep 1.2, 128 steps 2.488 8.0 80.6
keep, rep 1.2, blocks of 16 2.745 4.6 87.3
keep, rep 1.2, blocks of 32 2.717 4.9 86.6
keep, rep 1.2, noise from masks 2.447 15.0 74.5
keep, rep 1.2, remask 0.1 2.98 6.7 82.9
keep, rep 1.2, temp 0.8 2.425 13.3 76.2
keep, rep 1.2, top-p 0.9 2.514 7.7 81.3
keep, rep 1.5 3.236 1.9 95.0
keep, repetition 1.3 2.622 7.4 86.0
left to right (blocks of 1) 3.673 4.2 88.7
left to right, rep 1.1 4.137 1.6 93.9
left to right, rep 1.2 4.354 0.4 97.2
random order, temp 0.8 3.418 3.7 83.9
random order, temp 1.0 3.83 0.6 91.4
Diffusion 124M · FineWeb-Edu full + Cosmopedia · 2,500M tokens 3.830 2.5B base AR model, top-k 50 (judge's own text) 2.073 1.2 93.4
real continuation 3.063 0.4 95.5
keep (confident first), top-k 50 1.604 47.1 41.9
keep, 16 steps 2.184 53.1 32.0
keep, blocks of 16 1.964 34.4 51.7
keep, blocks of 4 2.411 24.4 63.1
keep, blocks of 4, rep 1.2 3.109 3.0 90.9
keep, blocks of 8, rep 1.2 2.878 6.7 84.2
keep, blocks of 8, rep 1.3 3.012 5.3 88.1
keep, distinct 1.604 47.1 41.9
keep, distinct, 16 steps 2.525 17.7 62.8
keep, distinct, blocks of 16 1.964 34.4 51.7
keep, distinct, rep 1.1 2.12 27.4 59.4
keep, distinct, rep 1.2 2.269 18.7 71.3
keep, distinct, rep 1.2, blocks of 16 2.584 11.5 77.5
keep, noise level from masks 1.625 45.2 43.3
keep, remask 0.1 1.846 45.1 42.8
keep, rep 1.15 2.232 25.3 62.5
keep, rep 1.2 2.269 18.7 71.3
keep, rep 1.2, 128 steps 2.298 19.1 70.3
keep, rep 1.2, blocks of 16 2.584 11.5 77.5
keep, rep 1.2, blocks of 32 2.298 19.2 69.7
keep, rep 1.2, noise from masks 2.252 21.1 65.8
keep, rep 1.2, remask 0.1 2.868 20.4 70.8
keep, rep 1.2, temp 0.8 2.302 15.6 73.9
keep, rep 1.2, top-p 0.9 2.245 20.5 69.3
keep, rep 1.5 3.106 2.7 92.6
keep, repetition 1.3 2.551 10.1 82.8
left to right (blocks of 1) 3.419 7.0 85.6
left to right, rep 1.1 3.834 2.5 91.8
left to right, rep 1.2 4.08 1.2 96.0
random order, temp 0.8 3.393 2.6 84.2
random order, temp 1.0 3.882 1.6 90.3