From docs/results.md
Results
Generated by python3 -m infra.results render from results/results.jsonl, the one results store: every evaluation adds its numbers there. Don't edit this page by hand. The latest measurement of each model is shown; the store keeps the earlier ones, with when and at which commit each was measured.
Benchmarks
Zero-shot, lm-evaluation-harness style, in percent: HellaSwag acc_norm, LAMBADA acc, ARC-Easy acc, PIQA acc, Winogrande acc. Diffusion models score each choice by its likelihood bound. Average over the five tasks when all were run.
| Model | HellaSwag | LAMBADA | ARC-Easy | PIQA | Winogrande | Average | Measured |
|---|---|---|---|---|---|---|---|
| Chance | 25.0 | 0.0 | 25.0 | 50.0 | 50.0 | 30.0 | |
| OpenAI GPT-2 small (published) | 31.1 | 32.6 | 43.8 | 62.9 | 51.6 | 44.4 | |
| Medium 51M · TinyStories · 20k steps | 26.5 | 8.7 | 25.9 | 54.1 | 51.6 | 33.4 | 2026-11-14 |
| OpenAI GPT-2 small (124M, 2019) | 31.1 | 32.6 | 43.8 | 62.9 | 51.6 | 44.4 | 2026-11-12 |
| Large 124M RoPE SwiGLU no-bias QK-norm · Mini FineWeb-Edu · 30k steps | 27.9 | 9.6 | 42.1 | 58.4 | 50.6 | 37.7 | 2026-11-12 |
| Large 124M best design · FineWeb-Edu + Cosmopedia · 500M tokens · context 512 | 29.7 | 15.7 | 44.0 | 62.6 | 50.0 | 40.4 | 2026-11-13 |
| Diffusion 51M · TinyStories · 164M tokens | 25.7 | 1.7 | 25.7 | 51.9 | 48.4 | 30.7 | 2026-11-14 |
| Diffusion 51M · Mini FineWeb-Edu · 164M tokens | 25.3 | 0.2 | 31.2 | 52.1 | 50.5 | 31.9 | 2026-11-14 |
| Diffusion 51M · Mini FineWeb-Edu · 492M tokens | 26.6 | 2.2 | 29.2 | 52.2 | 51.9 | 32.4 | 2026-11-14 |
| Chat Large 124M RoPE SwiGLU no-bias QK-norm · instruction-tuned on smol-smoltalk · 11.284k steps | 28.9 | 10.6 | 38.1 | 58.5 | 50.4 | 37.3 | 2026-11-14 |
| Chat Large 124M RoPE SwiGLU no-bias QK-norm · instruction-tuned on smol-smoltalk + DPO against repetition · 11.284k steps | 29.3 | 9.5 | 38.5 | 58.5 | 50.4 | 37.2 | 2026-11-14 |
| Large 124M best design · stage 2 · FineWeb-Edu full + Cosmopedia · 2.5B tokens · context 512 | 32.3 | 20.3 | 46.8 | 62.4 | 49.9 | 42.3 | 2026-11-15 |
| Chat Large 124M · stage 2 base · instruction-tuned on smol-smoltalk · 11.284k steps | 30.1 | 13.5 | 41.8 | 59.1 | 47.9 | 38.5 | 2026-11-15 |
| Chat Large 124M · stage 2 base · instruction-tuned on smol-smoltalk + DPO against repetition | 30.1 | 12.9 | 42.0 | 59.0 | 47.9 | 38.4 | 2026-11-15 |
| Diffusion 124M · FineWeb-Edu full + Cosmopedia · 999M tokens | 27.6 | 11.8 | 30.7 | 54.2 | 49.6 | 34.8 | 2026-11-16 |
| Diffusion 124M · FineWeb-Edu full + Cosmopedia · 2,500M tokens | 28.0 | 14.4 | 30.1 | 55.4 | 49.1 | 35.4 | 2026-11-17 |
| Chat Large 124M · stage 2 base · instruction-tuned on smol-smoltalk, lr 5e-5 + replay 0.2 · 13.888k steps | 32.5 | 16.0 | 47.5 | 62.2 | 48.9 | 41.4 | 2026-11-18 |
| Chat Large 124M · stage 2 base · SFT (lr 5e-5 + replay 0.2) + DPO · 948 steps | 32.7 | 15.0 | 47.9 | 62.1 | 49.4 | 41.4 | 2026-11-18 |
| Chat Large 124M · stage 2 base · SFT (lr 5e-5 + replay 0.2) + DPO · 948 steps · RM pairs DPO | 33.0 | 14.4 | 48.5 | 61.9 | 48.9 | 41.3 | 2026-11-19 |
Held-out loss
Cross-entropy (nats per token) on held-out text from evaluate.py; lower is better. One table per set of data sets the models were measured on (the suite grew as new data was added), newest first.
| Model | Cosmopedia khanacademy | Cosmopedia openstax | Cosmopedia stanford-first100000 | Cosmopedia stories-first300000 | Cosmopedia wikihow | Mini FineWeb-Edu | Mini FineWeb-Edu tiny | Tiny Shakespeare | TinyStories | WikiText-2 | Measured |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Large 124M best design · stage 2 · FineWeb-Edu full + Cosmopedia · 2.5B tokens · context 512 | 2.336 | 2.717 | 2.641 | 2.785 | 2.342 | 3.296 | 3.310 | 5.446 | 2.751 | 5.047 | 2026-11-15 |
| Model | Mini FineWeb-Edu | Tiny Shakespeare | TinyStories | WikiText-2 | Measured |
|---|---|---|---|---|---|
| OpenAI GPT-2 small (124M, 2019) | 3.401 | 4.807 | 2.553 | 4.180 | 2026-11-11 |
| GPT-2 style · Mini FineWeb-Edu · 5,000 updates · NVIDIA L4 | 4.580 | 6.296 | 4.682 | 6.528 | 2026-11-11 |
| Our best, dense · Mini FineWeb-Edu · 5,000 updates · NVIDIA L4 | 4.301 | 6.138 | 4.465 | 6.276 | 2026-11-11 |
| Large 124M · Mini FineWeb-Edu · 30k steps | 3.735 | 5.823 | 3.872 | 5.582 | 2026-11-11 |
| Our best + experts · Mini FineWeb-Edu · 5,000 updates · NVIDIA L4 | 4.245 | 6.121 | 4.398 | 6.240 | 2026-11-11 |
| Large 124M RoPE SwiGLU no-bias QK-norm · Mini FineWeb-Edu · 30k steps | 3.691 | 5.753 | 3.856 | 5.605 | 2026-11-12 |
| Large 124M best design · FineWeb-Edu + Cosmopedia · 500M tokens · context 512 | 3.549 | 5.614 | 2.790 | 5.370 | 2026-11-13 |
| Model | TinyStories | Measured |
|---|---|---|
| I5 drafter: Tiny best design · tiny-best · tiny rope layernorm swiglu no-bias qk-norm lr=0.0018 · 2,000 updates | 2.063 | 2026-11-12 |
| Model | Mini FineWeb-Edu | Tiny Shakespeare | WikiText-2 | Measured |
|---|---|---|---|---|
| Large 124M, best design · lr 6e-4 · Mini FineWeb-Edu · 1,000 updates · NVIDIA L4 | 5.172 | 6.458 | 7.136 | 2026-11-12 |
| Large 124M, best design · lr 1.2e-3 · Mini FineWeb-Edu · 1,000 updates · NVIDIA L4 | 5.155 | 6.482 | 7.138 | 2026-11-12 |
| Large 124M, best design · lr 1.8e-3 · Mini FineWeb-Edu · 1,000 updates · NVIDIA L4 | 5.148 | 6.467 | 7.140 | 2026-11-12 |
| Model | Tiny Shakespeare | TinyStories | WikiText-2 | Measured |
|---|---|---|---|---|
| Medium 51M · TinyStories · 20k steps | 7.632 | 1.561 | 8.693 | 2026-11-10 |
| Tiny 16M · WikiText · 40 steps | 10.783 | 10.701 | 10.547 | 2026-11-10 |
| Small 30M · TinyStories · 2k steps | 8.100 | 4.620 | 9.765 | 2026-11-10 |
| Small 30M · TinyStories + Tiny Shakespeare · 1k steps | 7.047 | 6.382 | 9.375 | 2026-11-10 |
| Large 124M · TinyStories · 2k steps | 8.495 | 4.510 | 9.842 | 2026-11-10 |
| A1 baseline (learned) · Small 30M · TinyStories · 1k steps | 8.149 | 2.602 | 9.569 | 2026-11-10 |
| A1 RoPE · Small 30M · TinyStories · 1k steps | 8.040 | 2.440 | 9.461 | 2026-11-10 |
| A5 QK-norm · baseline · small rope layernorm swiglu no-bias lr=0.0006 · 1,000 updates · NVIDIA L4 | 8.216 | 2.329 | 9.456 | 2026-11-11 |
| A5 QK-norm · qk-norm · small rope layernorm swiglu no-bias qk-norm lr=0.0006 · 1,000 updates · NVIDIA L4 | 8.016 | 2.300 | 9.461 | 2026-11-11 |
| A5 QK-norm · baseline-3x · small rope layernorm swiglu no-bias lr=0.0018 · 1,000 updates · NVIDIA L4 | 8.365 | 2.230 | 9.900 | 2026-11-11 |
| LR sweep · base-3e-3 · small rope layernorm swiglu no-bias lr=0.003 · 1,000 updates · NVIDIA L4 | 8.611 | 2.621 | 10.399 | 2026-11-11 |
| LR sweep · qk-3e-3 · small rope layernorm swiglu no-bias qk-norm lr=0.003 · 1,000 updates · NVIDIA L4 | 8.510 | 2.339 | 9.794 | 2026-11-11 |
| LR sweep · base-6e-3 · small rope layernorm swiglu no-bias lr=0.006 · 1,000 updates · NVIDIA L4 | 9.029 | 2.956 | 10.692 | 2026-11-11 |
| LR sweep · qk-6e-3 · small rope layernorm swiglu no-bias qk-norm lr=0.006 · 1,000 updates · NVIDIA L4 | 8.503 | 2.512 | 9.889 | 2026-11-11 |
| LR sweep · base-1.2e-2 · small rope layernorm swiglu no-bias lr=0.012 · 1,000 updates · NVIDIA L4 | 9.560 | 3.281 | 10.665 | 2026-11-11 |
| LR sweep · qk-1.2e-2 · small rope layernorm swiglu no-bias qk-norm lr=0.012 · 1,000 updates · NVIDIA L4 | 8.713 | 2.517 | 10.043 | 2026-11-11 |
| A6 GQA · kv3 · small rope layernorm swiglu no-bias qk-norm kv=3 lr=0.0018 · 1,000 updates · NVIDIA L4 | 8.379 | 2.212 | 9.863 | 2026-11-11 |
| A6 GQA · kv2 · small rope layernorm swiglu no-bias qk-norm kv=2 lr=0.0018 · 1,000 updates · NVIDIA L4 | 8.484 | 2.226 | 9.973 | 2026-11-11 |
| A6 GQA · kv1 · small rope layernorm swiglu no-bias qk-norm kv=1 lr=0.0018 · 1,000 updates · NVIDIA L4 | 8.395 | 2.248 | 10.077 | 2026-11-11 |
| A7 depth vs width · wide-3x448 · 3×448 (7 heads) rope layernorm swiglu no-bias qk-norm lr=0.0018 · 1,000 updates · NVIDIA L4 | 8.403 | 2.235 | 9.896 | 2026-11-11 |
| A7 depth vs width · deep-11x320 · 11×320 (5 heads) rope layernorm swiglu no-bias qk-norm lr=0.0018 · 1,000 updates · NVIDIA L4 | 8.391 | 2.233 | 10.061 | 2026-11-11 |
| A7 depth vs width · deeper-22x256 · 22×256 (4 heads) rope layernorm swiglu no-bias qk-norm lr=0.0018 · 1,000 updates · NVIDIA L4 | 8.380 | 2.360 | 9.716 | 2026-11-11 |
| A8 layer sharing · share2 · small rope layernorm swiglu no-bias qk-norm repeat=2 lr=0.0018 · 1,000 updates · NVIDIA L4 | 8.316 | 2.311 | 9.924 | 2026-11-11 |
| A8 layer sharing · share3 · small rope layernorm swiglu no-bias qk-norm repeat=3 lr=0.0018 · 1,000 updates · NVIDIA L4 | 8.480 | 2.481 | 9.802 | 2026-11-11 |
| A8 layer sharing · wide-share2 · 3×448 (7 heads) rope layernorm swiglu no-bias qk-norm repeat=2 lr=0.0018 · 1,000 updates · NVIDIA L4 | 8.400 | 2.311 | 9.918 | 2026-11-11 |
| A9 multi-token prediction · mtp2 · small rope layernorm swiglu no-bias qk-norm mtp=2 lr=0.0018 · 1,000 updates · NVIDIA L4 | 8.412 | 2.302 | 10.034 | 2026-11-11 |
| A9 multi-token prediction · mtp3 · small rope layernorm swiglu no-bias qk-norm mtp=3 lr=0.0018 · 1,000 updates · NVIDIA L4 | 8.456 | 2.454 | 10.285 | 2026-11-11 |
| A9 multi-token prediction · mtp4 · small rope layernorm swiglu no-bias qk-norm mtp=4 lr=0.0018 · 1,000 updates · NVIDIA L4 | 8.610 | 2.494 | 10.258 | 2026-11-11 |
| A5 QK-norm · qk-norm-3x · small rope layernorm swiglu no-bias qk-norm lr=0.0018 · 1,000 updates · NVIDIA L4 | 8.360 | 2.219 | 9.938 | 2026-11-11 |
| A10 mixture of experts · moe4-top1 · small rope layernorm swiglu no-bias qk-norm experts=4 topk=1 lr=0.0018 · 1,000 updates · NVIDIA L4 | 8.320 | 2.361 | 9.996 | 2026-11-11 |
| A10 mixture of experts · moe8-top1 · small rope layernorm swiglu no-bias qk-norm experts=8 topk=1 lr=0.0018 · 1,000 updates · NVIDIA L4 | 8.390 | 2.435 | 10.027 | 2026-11-11 |
| A10 mixture of experts · moe4-top2 · small rope layernorm swiglu no-bias qk-norm experts=4 topk=2 lr=0.0018 · 1,000 updates · NVIDIA L4 | 8.409 | 2.171 | 9.899 | 2026-11-11 |
| Final: GPT-2 style vs all the wins · gpt2-style · small learned layernorm gelu lr=0.0006 · 5,000 updates · NVIDIA L4 | 8.342 | 1.878 | 9.508 | 2026-11-11 |
| Final: GPT-2 style vs all the wins · gpt2-style-lr · small learned layernorm gelu lr=0.0018 · 5,000 updates · NVIDIA L4 | 8.344 | 1.820 | 9.792 | 2026-11-11 |
| Final: GPT-2 style vs all the wins · best-dense · small rope layernorm swiglu no-bias qk-norm lr=0.0018 · 5,000 updates · NVIDIA L4 | 8.300 | 1.754 | 9.777 | 2026-11-11 |
| A2 RoPE + RMSNorm · Small 30M · TinyStories · 1k steps | 8.037 | 2.451 | 9.454 | 2026-11-11 |
| Final: GPT-2 style vs all the wins · best-moe · small rope layernorm swiglu no-bias qk-norm experts=4 topk=2 lr=0.0018 · 5,000 updates · NVIDIA L4 | 8.194 | 1.700 | 9.597 | 2026-11-11 |
| A3 RoPE + SwiGLU · Small 30M · TinyStories · 1k steps | 8.019 | 2.338 | 9.501 | 2026-11-11 |
| A4 RoPE + SwiGLU, no biases · Small 30M · TinyStories · 1k steps | 8.093 | 2.324 | 9.482 | 2026-11-11 |
| A5 QK-norm · RoPE SwiGLU no-bias · Small 30M · TinyStories · 1k steps | 7.980 | 2.299 | 9.441 | 2026-11-11 |
Chat
sft.py compare: loss on held-out smol-smoltalk answers, and greedy answers to the fixed questions (repeated 4-grams, share that end by themselves). dpo.py eval: the same on 100 held-out questions, with the loss on their reference answers.
| Model | Answer loss | Repetition | Ended | Held-out repetition | Held-out ended | Reference loss | Measured |
|---|---|---|---|---|---|---|---|
| Large 124M best design · FineWeb-Edu + Cosmopedia · 500M tokens · context 512 | 2.437 | 29.4 | 0.0 | 2026-11-14 | |||
| Chat Large 124M RoPE SwiGLU no-bias QK-norm · instruction-tuned on smol-smoltalk · 11.284k steps | 1.548 | 27.5 | 25.0 | 20.0 | 59.0 | 1.312 | 2026-11-15 |
| outputs/dpo/smoltalk-2000 | 1.617 | 16.1 | 50.0 | 11.2 | 84.0 | 1.399 | 2026-11-14 |
| Chat Large 124M RoPE SwiGLU no-bias QK-norm · instruction-tuned on smol-smoltalk + DPO against repetition · 11.284k steps | 11.2 | 84.0 | 1.399 | 2026-11-15 | |||
| Large 124M best design · stage 2 · FineWeb-Edu full + Cosmopedia · 2.5B tokens · context 512 | 2.329 | 27.8 | 0.0 | 2026-11-15 | |||
| Chat Large 124M · stage 2 base · instruction-tuned on smol-smoltalk · 11.284k steps | 1.406 | 22.3 | 16.7 | 12.4 | 70.0 | 1.156 | 2026-11-18 |
| Chat Large 124M · stage 2 base · instruction-tuned on smol-smoltalk + DPO against repetition | 1.459 | 9.9 | 50.0 | 7.5 | 85.0 | 1.163 | 2026-11-19 |
| Chat Large 124M · stage 2 base · instruction-tuned on smol-smoltalk, lr 5e-5 + replay 0.2 · 13.888k steps | 1.456 | 20.9 | 8.3 | 16.0 | 69.0 | 1.230 | 2026-11-19 |
| Chat Large 124M · stage 2 base · SFT (lr 5e-5 + replay 0.2) + DPO · 948 steps | 8.5 | 87.0 | 1.333 | 2026-11-19 | |||
| Chat Large 124M · stage 2 base · SFT (lr 5e-5 + replay 0.2) + DPO · 948 steps · RM pairs DPO | 2.3 | 99.0 | 1.425 | 2026-11-19 | |||
| Chat Large 124M · stage 2 base · SFT (lr 5e-5 + replay 0.2) + DPO · 948 steps · RM pairs SimPO | 2.2 | 99.0 | 1.474 | 2026-11-19 |
Diffusion models
Best likelihood bound on held-out text during training (nats per token, an upper bound on the loss), and diffusion_models/quality.py: 64-token continuations judged by an autoregressive model (its loss on the text; lower reads as more natural) and the share of repeated 4-grams, per sampler.
| Model | Bound | Tokens | Sampler | Judge loss | Repetition | Distinct 2-grams |
|---|---|---|---|---|---|---|
| Diffusion 51M · TinyStories · 164M tokens | 2.525 | 164M | base AR model, top-k 50 (judge's own text) | 2.001 | 0.6 | 97.0 |
| real continuation | 2.675 | 0.8 | 93.9 | |||
| keep (confident first), top-k 50 | 2.47 | 6.2 | 78.9 | |||
| keep, blocks of 4 | 3.057 | 2.5 | 86.8 | |||
| keep, blocks of 8, rep 1.2 | 3.27 | 0.2 | 94.9 | |||
| keep, distinct, 16 steps | 3.168 | 4.3 | 81.8 | |||
| keep, rep 1.2, blocks of 16 | 2.995 | 0.6 | 94.8 | |||
| keep, repetition 1.3 | 2.797 | 0.4 | 94.8 | |||
| Diffusion 51M · Mini FineWeb-Edu · 164M tokens | 5.269 | 164M | base AR model, top-k 50 (judge's own text) | 2.34 | 0.6 | 94.6 |
| real continuation | 3.793 | 0.0 | 97.2 | |||
| keep (confident first), top-k 50 | 2.326 | 42.8 | 35.7 | |||
| keep, blocks of 16 | 2.8 | 23.9 | 46.6 | |||
| keep, blocks of 8, rep 1.3 | 4.131 | 0.1 | 95.9 | |||
| keep, noise level from masks | 2.614 | 38.4 | 37.1 | |||
| keep, remask 0.1 | 3.433 | 28.9 | 44.6 | |||
| keep, repetition 1.3 | 3.622 | 3.8 | 85.1 | |||
| random order, temp 0.8 | 5.269 | 0.0 | 95.1 | |||
| random order, temp 1.0 | 6.933 | 0.0 | 99.4 | |||
| Diffusion 51M · Mini FineWeb-Edu · 492M tokens | 4.729 | 492M | base AR model, top-k 50 (judge's own text) | 2.34 | 0.6 | 94.6 |
| real continuation | 3.793 | 0.0 | 97.2 | |||
| keep (confident first), top-k 50 | 2.235 | 37.5 | 42.8 | |||
| keep, 16 steps | 2.85 | 32.9 | 41.3 | |||
| keep, blocks of 16 | 2.697 | 20.0 | 55.7 | |||
| keep, blocks of 4 | 3.277 | 7.6 | 75.8 | |||
| keep, blocks of 4, rep 1.2 | 3.797 | 0.9 | 92.2 | |||
| keep, blocks of 8, rep 1.2 | 3.444 | 1.1 | 90.1 | |||
| keep, blocks of 8, rep 1.3 | 3.692 | 0.5 | 93.5 | |||
| keep, distinct | 2.235 | 37.5 | 42.8 | |||
| keep, distinct, 16 steps | 3.031 | 9.6 | 63.8 | |||
| keep, distinct, blocks of 16 | 2.697 | 20.0 | 55.7 | |||
| keep, distinct, rep 1.1 | 2.3 | 27.4 | 53.1 | |||
| keep, distinct, rep 1.2 | 2.731 | 11.4 | 74.0 | |||
| keep, distinct, rep 1.2, blocks of 16 | 3.016 | 8.4 | 81.8 | |||
| keep, noise level from masks | 2.132 | 44.0 | 37.8 | |||
| keep, remask 0.1 | 3.72 | 21.2 | 56.8 | |||
| keep, rep 1.15 | 2.384 | 24.9 | 57.8 | |||
| keep, rep 1.2 | 2.731 | 11.4 | 74.0 | |||
| keep, rep 1.2, 128 steps | 2.838 | 9.1 | 73.6 | |||
| keep, rep 1.2, blocks of 16 | 3.016 | 8.4 | 81.8 | |||
| keep, rep 1.2, blocks of 32 | 2.889 | 8.8 | 77.6 | |||
| keep, rep 1.2, noise from masks | 2.704 | 12.7 | 72.0 | |||
| keep, rep 1.2, remask 0.1 | 4.199 | 8.2 | 77.6 | |||
| keep, rep 1.2, temp 0.8 | 2.557 | 14.9 | 68.8 | |||
| keep, rep 1.2, top-p 0.9 | 2.531 | 19.5 | 64.7 | |||
| keep, rep 1.5 | 3.503 | 6.2 | 89.8 | |||
| keep, repetition 1.3 | 3.014 | 6.1 | 84.4 | |||
| left to right (blocks of 1) | 4.17 | 0.9 | 90.0 | |||
| left to right, rep 1.1 | 4.455 | 0.0 | 97.1 | |||
| left to right, rep 1.2 | 4.815 | 0.1 | 98.5 | |||
| random order, temp 0.8 | 3.616 | 2.1 | 85.9 | |||
| random order, temp 1.0 | 4.207 | 0.2 | 92.8 | |||
| Diffusion 124M · FineWeb-Edu full + Cosmopedia · 999M tokens | 3.923 | 999M | base AR model, top-k 50 (judge's own text) | 2.073 | 1.2 | 93.4 |
| real continuation | 3.063 | 0.4 | 95.5 | |||
| keep (confident first), top-k 50 | 2.155 | 24.5 | 59.7 | |||
| keep, 16 steps | 2.776 | 30.2 | 53.2 | |||
| keep, blocks of 16 | 2.246 | 21.0 | 64.9 | |||
| keep, blocks of 4 | 2.559 | 22.2 | 64.2 | |||
| keep, blocks of 4, rep 1.2 | 3.419 | 3.2 | 90.6 | |||
| keep, blocks of 8, rep 1.2 | 3.16 | 2.2 | 91.8 | |||
| keep, blocks of 8, rep 1.3 | 3.233 | 2.8 | 93.7 | |||
| keep, distinct | 2.155 | 24.5 | 59.7 | |||
| keep, distinct, 16 steps | 2.687 | 13.8 | 67.8 | |||
| keep, distinct, blocks of 16 | 2.246 | 21.0 | 64.9 | |||
| keep, distinct, rep 1.1 | 2.293 | 15.5 | 72.8 | |||
| keep, distinct, rep 1.2 | 2.473 | 10.6 | 79.6 | |||
| keep, distinct, rep 1.2, blocks of 16 | 2.745 | 4.6 | 87.3 | |||
| keep, noise level from masks | 2.054 | 28.1 | 55.6 | |||
| keep, remask 0.1 | 2.583 | 26.6 | 57.9 | |||
| keep, rep 1.15 | 2.424 | 11.7 | 76.1 | |||
| keep, rep 1.2 | 2.473 | 10.6 | 79.6 | |||
| keep, rep 1.2, 128 steps | 2.488 | 8.0 | 80.6 | |||
| keep, rep 1.2, blocks of 16 | 2.745 | 4.6 | 87.3 | |||
| keep, rep 1.2, blocks of 32 | 2.717 | 4.9 | 86.6 | |||
| keep, rep 1.2, noise from masks | 2.447 | 15.0 | 74.5 | |||
| keep, rep 1.2, remask 0.1 | 2.98 | 6.7 | 82.9 | |||
| keep, rep 1.2, temp 0.8 | 2.425 | 13.3 | 76.2 | |||
| keep, rep 1.2, top-p 0.9 | 2.514 | 7.7 | 81.3 | |||
| keep, rep 1.5 | 3.236 | 1.9 | 95.0 | |||
| keep, repetition 1.3 | 2.622 | 7.4 | 86.0 | |||
| left to right (blocks of 1) | 3.673 | 4.2 | 88.7 | |||
| left to right, rep 1.1 | 4.137 | 1.6 | 93.9 | |||
| left to right, rep 1.2 | 4.354 | 0.4 | 97.2 | |||
| random order, temp 0.8 | 3.418 | 3.7 | 83.9 | |||
| random order, temp 1.0 | 3.83 | 0.6 | 91.4 | |||
| Diffusion 124M · FineWeb-Edu full + Cosmopedia · 2,500M tokens | 3.830 | 2.5B | base AR model, top-k 50 (judge's own text) | 2.073 | 1.2 | 93.4 |
| real continuation | 3.063 | 0.4 | 95.5 | |||
| keep (confident first), top-k 50 | 1.604 | 47.1 | 41.9 | |||
| keep, 16 steps | 2.184 | 53.1 | 32.0 | |||
| keep, blocks of 16 | 1.964 | 34.4 | 51.7 | |||
| keep, blocks of 4 | 2.411 | 24.4 | 63.1 | |||
| keep, blocks of 4, rep 1.2 | 3.109 | 3.0 | 90.9 | |||
| keep, blocks of 8, rep 1.2 | 2.878 | 6.7 | 84.2 | |||
| keep, blocks of 8, rep 1.3 | 3.012 | 5.3 | 88.1 | |||
| keep, distinct | 1.604 | 47.1 | 41.9 | |||
| keep, distinct, 16 steps | 2.525 | 17.7 | 62.8 | |||
| keep, distinct, blocks of 16 | 1.964 | 34.4 | 51.7 | |||
| keep, distinct, rep 1.1 | 2.12 | 27.4 | 59.4 | |||
| keep, distinct, rep 1.2 | 2.269 | 18.7 | 71.3 | |||
| keep, distinct, rep 1.2, blocks of 16 | 2.584 | 11.5 | 77.5 | |||
| keep, noise level from masks | 1.625 | 45.2 | 43.3 | |||
| keep, remask 0.1 | 1.846 | 45.1 | 42.8 | |||
| keep, rep 1.15 | 2.232 | 25.3 | 62.5 | |||
| keep, rep 1.2 | 2.269 | 18.7 | 71.3 | |||
| keep, rep 1.2, 128 steps | 2.298 | 19.1 | 70.3 | |||
| keep, rep 1.2, blocks of 16 | 2.584 | 11.5 | 77.5 | |||
| keep, rep 1.2, blocks of 32 | 2.298 | 19.2 | 69.7 | |||
| keep, rep 1.2, noise from masks | 2.252 | 21.1 | 65.8 | |||
| keep, rep 1.2, remask 0.1 | 2.868 | 20.4 | 70.8 | |||
| keep, rep 1.2, temp 0.8 | 2.302 | 15.6 | 73.9 | |||
| keep, rep 1.2, top-p 0.9 | 2.245 | 20.5 | 69.3 | |||
| keep, rep 1.5 | 3.106 | 2.7 | 92.6 | |||
| keep, repetition 1.3 | 2.551 | 10.1 | 82.8 | |||
| left to right (blocks of 1) | 3.419 | 7.0 | 85.6 | |||
| left to right, rep 1.1 | 3.834 | 2.5 | 91.8 | |||
| left to right, rep 1.2 | 4.08 | 1.2 | 96.0 | |||
| random order, temp 0.8 | 3.393 | 2.6 | 84.2 | |||
| random order, temp 1.0 | 3.882 | 1.6 | 90.3 |