TinyStories ARM

From docs/plan.md

The plan

The project's roadmap: what's done, what's next and who does it. It's kept up to date as work lands; the results of each step are in Analysis and the story in the journey.

Base model (what new work builds on, since 15 Nov): Large 124M best design · stage 2 · FineWeb-Edu full + Cosmopedia · 2.5B tokens · context 512 (mini-fineweb-edu-ful-202611151103-bb160c): held-out FineWeb-Edu loss 3.296 (GPT-2 small 3.401); HellaSwag 32.3%, LAMBADA 20.3%, ARC-Easy 46.8%, PIQA 62.4%, Winogrande 49.9%, average 42.3 (GPT-2 small 44.4). Its chat model is chat-mini-fineweb-ed-202611152150-2746ba (SFT + DPO).

Before 15 Nov the base was Large 124M best design · FineWeb-Edu + Cosmopedia · 500M tokens · context 512 (mini-fineweb-edu-202611130718-14ed26): held-out FineWeb-Edu loss 3.549; ARC-Easy 44.0%, PIQA 62.6%, HellaSwag 29.7%, LAMBADA 15.7%. The yardstick is OpenAI's GPT-2 small (3.401; 43.8%, 62.9%, 31.1%, 32.6%). The earlier TinyStories and WikiText models are kept as the history of how we got here.

Last updated: 18 November 2026, 00:15 IST.

Final goals (adopted by the author, 17 Nov)

What "done" looks like for the project; every step below should serve one of these. On 17 Nov the author confirmed all four and paused new work until F6 reaches 2.5B tokens: then the project has two 124M models, autoregressive and diffusion, on the same data, and goal 2's comparison (F4) comes next.

  1. A 124M model that matches GPT-2 small across the board, and a chat model built on it. Already past GPT-2 on held-out web text (3.296 vs 3.401) and on HellaSwag and ARC-Easy; still behind on the benchmark average (42.3 vs 44.4), LAMBADA and WikiText. Done when the average reaches 44.4 and the chat model answers short questions with under 5% greedy repetition and ends 90%+ of its answers. The chat half is met (19 Nov, D4): the served chat model repeats 2.3% and ends 99% on the 100 held-out questions. The benchmark half (42.3 vs 44.4) is open: it needs more or better pre-training.
  2. A fair, published head-to-head of autoregressive and masked diffusion models at 124M: the same size, data, tokens and tokenizer (F6 extended to 2.5B tokens), compared on loss, benchmarks, text quality, speed and checkable reasoning (F4, I7), with the side-by-side visualizations (H1) on a public website (G).
  3. A small model that visibly reasons: with thinking (IA, I0–I3), the 124M model solves short checkable tasks (multi-digit arithmetic, countdown) clearly better than without it, and the same for the diffusion model (I4–I6).
  4. Infrastructure: any number in the README regenerates from a clean clone with one command, tested in CI, with training jobs that survive a closed browser tab and models that move between Colab and the Mac without manual downloads (J below).

Where things stand (19 November)

The chat model is cleaner (D4): DPO on pairs judged by a small open reward model (the best and worst of the model's own answers) cuts greedy repetition to 2.3% and lets 99% of answers end, with the knowledge and benchmarks unchanged; it is served on Hugging Face and the website's Try it page. D3 (GRPO) and the reasoning work (IA, I) remain the author's by hand.

Next, per the goals: publish the 124M AR vs diffusion comparison on the website (goal 2, the last open part on Claude's side); then goal 1's benchmark gap (a pre-training plan, the author's call on compute); then goal 4's one-command reproduction of the README numbers, J7 and J8. Paused until Sunday 22 Nov.

18 November, morning

F and G are done, G3 included (18 Nov: the website's Try it page runs the 124M models in the browser): diffusion is a choice inside Generate (F3), and the website has an Explorer that replays the base model's real numbers (G2). D3 (GRPO) and the reasoning work (IA, I) are the author's by hand from here.

18 November, just after midnight

F6 is evaluated at 2.5B tokens (bound 3.830, benchmark average 35.4) and F4 is done: like for like at 124M, the masked diffusion model trails the autoregressive one by about 0.85 nats per token on held-out web text (the same gap as at 51M) and 7 benchmark points, and at the same writing speed its text is much worse. Goal 2's comparison now exists; publishing it on the website (G) and the reasoning comparison (I7) remain. J6, H1 and G were done on 17 Nov.

17 November, evening

F6 reached 2.5B tokens (best bound 3.830) and is on the Mac; its evaluations run tonight with one command (J6, python3 -m infra.eval_queue f6-124m-stage2-2p5b), then F4 compares the two 124M models. The author's order for new building work: J6 ✅, then H1 ✅ (the Side by side page), then G ✅ (the website is live; the Explorer replay followed on 18 Nov; G3 optional). C1 and C2 (GPTQ) are done, with a Quantize page and the GPTQ base model on Hugging Face.

16 November, night

Everything below the stage 2 base is done: E2 (stage 2, the base since 15 Nov), D1 and D2 redone on it, H2, the prefill vs decode card on Generate, and F6 at 1B tokens (bound 3.923, benchmark average 34.8 against the AR model's 42.3).

Pending, in suggested order:

  1. F6 extension to 2.5B tokens (~3 h on an A100, --steps 152600 --resume; ask first), then re-measure the bound, quality and benchmarks (~3 h on the Mac).
  2. F4 the diffusion vs autoregressive comparison, and H1's diffusion half (with diffusion in Generate, F3's last piece).
  3. J1–J3 (tests and CI, a pinned environment, a resilient Colab job runner) before the next long runs.
  4. D3 GRPO with checkable rewards (needed by I3 and I6).
  5. I0 the reasoning eval set and longer context; then the author's IA (I1–I3), Claude integrates; I4–I6 for diffusion; I7 the comparison.
  6. G GitHub Pages once F4/H1 are in.
  7. Any time: C1 the author's GPTQ, then C2; E1 an A100 speed note (now measured: 10.3–10.7 steps/s for stage 2), E3 as needed.

Earlier status (15 November, morning)

Stage 2 is training again (from step 61,000, done about 21:30 on 15 Nov); everything else waits for it, as the author asked.

Lane What Status
Colab L4 F2c the long diffusion run on FineWeb-Edu (492M tokens) ✅ done 14:25: likelihood bound 4.729 (short run 5.269), 145 min; downloaded to diffusion_models/runs/fineweb-edu-small-colab-long
Colab L4 D1 instruction tuning ✅ 16:10: held-out answer loss 2.437 → 1.548 in 90 min; in the library as chat-mini-fineweb-ed-202611141041-96c673, evaluated (see D1)
Mac D2 DPO against repetition, on the chat model ✅ 19:30: held-out repetition 20.0% → 11.2%, answers that end 59% → 84%; in the library as chat-mini-fineweb-ed-202611141900-a8f8a7 (see D2)
Colab L4, now E2 stage 2: stage2.py, ~2.5B tokens, continued from the base training again since 15 Nov 10:00 from step 61,000, done about 21:30 on 15 Nov; snapshots every 10,000 steps. History: tokenizing took 25 min (FineWeb-Edu full = 3.8B tokens, all 2,974 held-out documents removed); trained from 16:35 on 14 Nov, step 30,500 → 152,600 at ~34,000 tokens/s (step 45,000 at 18:40); the runtime disconnected after 20:59 at step 61,000 (it had been due about 08:50 on 15 Nov); checkpoints on Drive (MyDrive/tinystories-runs/stage2-fineweb-full-cosmopedia), so a disconnect only means running the cell again
Mac F5 benchmarks for the two diffusion models ✅ done: Benchmarks page, README, journey

What comes next, in order (suggested):

  1. ~~Evaluate D1~~ ✅ done (see the D1 row).
  2. ~~Close F2c and F5~~ ✅ done: the long run's bound, quality and benchmarks are in the README, journey and on the Benchmarks page.
  3. While E2 trains (~1 day), work that needs no Colab GPU (Claude): F4 the diffusion vs autoregressive comparison, then the visualizations H1 (diffusion half of the side-by-side page) and ~~H2~~ ✅ (noising slider, denoising player). F3's last piece, diffusion in Generate, folds into H1.
  4. After E2: ~~evaluate stage 2 against the base and GPT-2~~ ✅ better than the base on every held-out set and the benchmark average (42.3 vs 40.4); ~~if better, it becomes the base~~ ✅ the base since 15 Nov (mini-fineweb-edu-ful-202611151103-bb160c, also the app default and the judge of generated text); ~~D1 is re-run on it~~ ✅ (15 Nov, A100), and ~~D2~~ ✅ (15 Nov, Mac) (~1 h). Then F6: the 124M diffusion model on the stage 2 data, 1B tokens (~8 h on the L4), extended to 2.5B if it keeps improving, so F4, H1 and I7 compare like with like.
  5. ~~D2 (DPO against repetition) on today's chat model~~ ✅ done on the Mac; redone on the stage 2 chat model later. Then D3 (GRPO).
  6. Then reasoning (I): the author writes the autoregressive reasoning model by hand on the best model (IA, after stage 2), Claude integrates it; the eval set and longer context (I0), thinking by fine-tuning (I1), test-time compute (I2), RL (I3); for diffusion, instruction tuning (I4), thinking (I5), RL (I6); then compare (I7).
  7. Any time: C1, the author's GPTQ (Claude adapts it in C2 when it's ready); G, GitHub Pages, once there's more to show.
# Phase / step Who Status
A Foundations ✅ done
A1 Fix the training set-up, move to Mini FineWeb-Edu, evaluation suite, model library, Lab, Journey page Claude ✅
A2 Architecture track A1–A10 (RoPE, SwiGLU, no biases, QK-norm, learning rate, GQA, experts, …), each A/B tested Claude ✅
A3 Inference track I0–I8 (1.8–3.5× faster generation, DRY, contrastive and beam search, 8/4-bit weights, speculative decoding, batching) Claude ✅
A4 Stage 1: the best design at 124M (held-out 3.691 vs 3.735 for GPT-2 style) Claude ✅
A5 Explorer (Flow, Diagram, Play the data flow); Benchmarks page (GPT-2 reproduces its published scores) Claude ✅
B Faster training and more kinds of data (13 Nov) ✅ done
B1 L1, faster training: torch.compile gives 1.7× on an L4, on by default Claude ✅
B2 Cosmopedia subsets, dataset mixing, streaming the first N rows, 512-token context Claude ✅
B3 Data run: 500M tokens, FineWeb-Edu + Cosmopedia, 4 h 20 min on a Colab L4 Colab ✅
B4 Evaluation suite and the five benchmarks Claude ✅
B5 Compare with the first base and GPT-2; it became the new base model Claude, then the author ✅
C Inference C1 and C2 done; the author's next inference ideas (C3)
C1 GPTQ quantization with llm-compressor's oneshot, written by hand for a standard Hugging Face model (e.g. GPT-2 small) the author; Claude helps when asked ✅ 17 Nov: quantization/llm_compressor.py (the author's, made consistent by Claude) on Qwen3-0.6B, W4A16, own environment quantization/requirements.txt: 1.11 GB → 514 MB, WikiText-2 perplexity 22.48 → 24.51 (+9%), 147 s on the Mac CPU; journey section 37
C2 Adapt C1 to our architecture (a local Hugging Face wrapper; RoPE, QK-norm, SwiGLU, tied output layer, 512 context); load the compressed weights back; compare with round-to-nearest on loss, benchmarks, size and speed; add to the app Claude ✅ 17 Nov (benchmarks still to run): everything in quantization/ (rtn, checkpoint, hf_wrapper, gptq, compare, llm_compressor). Our GPT in oneshot via TinyGPTForCausalLM; GPTQ integers read back into our QuantLinear, saved as a library model. Stage 2 base at 4 bits: RTN 87 MB, loss +0.034 (mean of 10 held-out sets); GPTQ 84 MB, +0.030, better on 8 of 10 sets, 44 s on the CPU. New Quantize page: model, recipe, sizes, perplexity, generate on both. The GPTQ copy is on Hugging Face: aramkris/fineweb-edu-124m-base-gptq-4bit (84 MB)
C3 The author's other inference ideas the author decides after C2
C4 Prefill vs decode, measured on our models (added 17 Nov by the author; reference: tryrag.io/prefill-vs-decode). Prefill reads the prompt in one pass and sets the time to first token; decode writes one token per pass and is limited by reading the weights from memory. Build on the existing prefill vs decode card on Generate: a roofline view (arithmetic intensity against the Mac GPU's memory bandwidth and compute ceiling) placing our prefill and decode, time to first token against prompt length, tokens/s against batch size (sharing one weight read across sequences), and float vs 8/4-bit weights (quantization should help decode, which is memory-bound, but our dequantize-every-token layers are slower today) Claude when the inference work starts
D Post-training (on the base model)
D1 Supervised fine-tuning on instructions (L7) on the base model: smol-smoltalk conversations in a plain User: / Assistant: format, loss on the assistant's words only, about 1 h on a Colab L4; a Chat mode in Generate Claude ✅ 14 Nov: sft.py (prepare / train / chat / compare), sft_colab.ipynb, Chat mode on Generate. 409K conversations (smol-smoltalk is long for 512 tokens: ~11% dropped, over half cut to the turns that fit), 11,284 updates, 90 min on an L4. Held-out answer loss 2.437 → 1.548; it answers in the right shape instead of writing blog posts, but facts are as weak as the base's ("the sky is blue because it is a blue color"); greedy repetition 29.4% → 27.5% (D2's target); 3 of 12 answers end by themselves within 160 tokens (answers run long); benchmarks HellaSwag 28.9, LAMBADA 10.6, ARC-Easy 38.1, PIQA 58.5, Winogrande 50.4 (base 29.7, 15.7, 44.0, 62.6, 50.0): the alignment tax. Redo on the stage 2 model (~1.5 h) Redone 15 Nov on the stage 2 base (chat-mini-fineweb-ed-202611151211-947df2, 19.4 min on a Colab A100): held-out answer loss 2.316 → 1.390 (was 1.548); 100 held-out questions: greedy repetition 12.4% (old SFT 20.0%, old DPO 11.2%), answers that end 70% (59%, 84%), reference-answer loss 1.156 (1.312, 1.399); benchmarks 30.1 · 13.5 · 41.8 · 59.1 · 47.9, average 38.5 (old chat 37.3; its base 42.3)
D2 DPO against repetition (greedy repetition is 25% today; the chat model's greedy answers: 27.5%) Claude ✅ 14 Nov, on the Mac: dpo.py (pairs / train / eval). Pairs from 2,000 smol-smoltalk single-turn questions with a short reference answer (≤ 200 tokens): chosen = the reference answer (it ends), rejected = the chat model's own greedy answer (20% repeated 4-grams, 38% hit the 200-token limit). DPO loss (β 0.1) against the frozen chat model plus a likelihood term (0.2) on the chosen answer; 1,900 pairs, 2 epochs, 948 updates, lr 3e-6, 21 min on the M5 (one pair at a time with gradients accumulated, fixed padded lengths: 16 GB). Held-out preference accuracy 5% → 99%. On 100 held-out questions: repetition 20.0% → 11.2%, answers that end within 200 tokens 59% → 84%, length 148 → 124 tokens; the twelve questions: 27.5% → 16.1%, 3 → 6 of 12 end. Cost: held-out answer loss 1.548 → 1.617 (DPO lowers the chosen answers' probability a little too). Benchmarks unchanged: HellaSwag 29.3, LAMBADA 9.5, ARC-Easy 38.5, PIQA 58.5, Winogrande 50.4 (average 37.2; the chat model 37.3). Some loops remain (poems). Library: chat-mini-fineweb-ed-202611141900-a8f8a7. Redo on the stage 2 chat model Redone 15 Nov on the stage 2 chat model (chat-mini-fineweb-ed-202611152150-2746ba, same 2,000 questions, 19 min on the Mac): 100 held-out questions: greedy repetition 12.4% → 6.5%, answers that end 70% → 86%, reference-answer loss 1.156 → 1.196; 12 questions: repetition 22.3% → 9.9%, 6/12 end; benchmarks 30.1 · 12.9 · 42.0 · 59.0 · 47.9 (average 38.4, unchanged). Now the default chat model in Generate
D1b Instruction tuning without forgetting (18 Nov: the chat model called India's capital "Vrindindjana"). facts.py, 200 closed-book questions (evals/facts.jsonl), showed SFT erased facts the base knew: choice 57.5% → 44%, the right answer about 100 times less likely (log prob −5.66 → −9.58), DPO no worse. Fix in sft.py: a gentler learning rate and --replay (pre-training text in each batch), with the facts checked at every evaluation; then compare base / old / new chat on facts, held-out answer loss, repetition and benchmarks, and redo DPO on the winner Claude ✅ 18 Nov: the run (lr 5e-5, replay 0.2, 13,888 updates, 25 min on an A100) is chat-mini-fineweb-ed-202611181029-abf820. Against the old SFT: facts 44.0% → 50.0% (log prob −9.58 → −7.17; answered right 21% → 25.5%, base 24%), benchmark average 37.7 → 40.3 (base 41.5; HellaSwag 32.5, ARC-Easy 42.5, LAMBADA 16.0), held-out answer loss 1.390 → 1.448, greedy repetition on the 100 held-out questions 12.4% → 15.6%, answers that end 70% → 73%. DPO on it (2,000 fresh pairs from its own greedy answers, same settings, 36 min on the Mac): chat-mini-fineweb-ed-202611182008-4c0ef4, facts 50.5% (answered right 30.0%, base 24%), benchmark average 40.5 (old SFT+DPO 37.7, base 41.5); on the D1b held-out 100 repetition 16.0% → 8.5%, answers that end 69% → 87% (corrected 19 Nov: the first figures, 15.6% → 8.5% and 73% → 82%, used dpo.py eval's default question set, 80 of whose questions were in these DPO pairs). Against the old chat model: far more knowledge, about the same repetition and endings. Next, if wanted: on-policy pairs scored by a reward model, with SimPO
D4 A reward model as the judge (best-of-N, then on-policy pairs, DPO vs SimPO) Claude ✅ 19 Nov, on the Mac: bestofn.py, judge Skywork-Reward-V2-Qwen3-0.6B (Apache-2.0, local). Pilot on 300 questions: at temperature 0.8 the judge's best of 6 loses to greedy (facts answered right 26.5% vs 30%); at 0.5 its best of greedy + 6 wins (37% vs 30%, judge score −2.87 vs −5.43, half the repetition). 1,999 on-policy pairs at 0.5 (--pairs, chosen = best, rejected = worst); dpo.py --method simpo added. On the D1b held-out 100: DPO on them (chat-mini-fineweb-ed-202611191448-e45919) repetition 8.5% → 2.3%, answers that end 87% → 99%, length 127 → 99, facts 50.5% (answered right 31.0%), benchmark average 40.5 (unchanged); SimPO (…-0881b0) 2.2%, 99%, but answered right 27%. e45919 is the best chat model and served since 19 Nov. Found on the way: the 18 Nov held-out numbers of 4c0ef4 used a question set 80% inside its DPO pairs (corrected in D1b). Self-consistency on facts (selfconsistency.py, greedy + 10 samples at 0.5 on e45919): a majority vote doesn't beat greedy (30–31.5% vs 31%; where greedy is wrong the right answer is rare, a median 2 of 11, and wrong ones repeat), nor does the model's own confidence; the judge's pick reaches 37% (oracle 48.5%), but that's the judge's knowledge. The facts it misses need pre-training or retrieval
D3 GRPO with automatically checkable rewards Claude after D2
E Scaling the data (the data run showed more data helps)
E1 Measure training speed on an A100 (none was free on 13 Nov) Claude when one is free
E2 Stage 2: a long, resumable run on FineWeb-Edu full + Cosmopedia (~2.5B tokens, ~20 tokens per parameter for 124M), continued from the base model; about 20 h on an L4 or ~8 h on an A100, checkpoints on Drive Colab ✅ 15 Nov 16:35: 2.5B tokens, library id mini-fineweb-edu-ful-202611151103-bb160c. Held-out Mini FineWeb-Edu 3.549 → 3.296 (GPT-2 small 3.401), TinyStories 2.790 → 2.751, WikiText-2 5.370 → 5.047; benchmarks HellaSwag 32.3, LAMBADA 20.3, ARC-Easy 46.8, PIQA 62.4, Winogrande 49.9, average 40.4 → 42.3 (GPT-2 44.4); greedy repetition 25% → 55%. Validation on the stage 2 mix 2.963. Four Colab sessions (the runtime died three times when the browser tab closed): L4 2.18 steps/s, A100 10.3–10.7. Journey section 36. Earlier status: training on an A100 since 15 Nov 15:35 (paused 14 Nov 20:59 at step 61,000, 15 Nov ~12:30 at step 79,000 and ~14:01 at step 121,000, each time when the browser tab holding the runtime closed; each time resumed from the Drive checkpoint): 2.18 steps/s on the L4, 10.3–10.7 steps/s on an A100, so done about 16:30 on 15 Nov; step 121,000 val 3.064 (best 3.061); a full snapshot of the training state is now kept every 10,000 steps (snapshots/, --from-snapshot). stage2.py extends the base run's own training state (step 30,500 → ~152,600, the cosine schedule stretched, the learning rate re-warmed) on Mini FineWeb-Edu full (3.5M documents) + the same Cosmopedia subsets. Small's held-out documents are removed from full first (the tiny subset alone held 594 of the 2,974), so the held-out comparison stays fair
E3 L2 (low-memory tokenizing on the Mac), L4 (quality filter) Claude as needed
F A masked diffusion language model in progress
F1 The diffusion model (L8), written and first trained on TinyStories the author ✅ first version generates text (diffusion_models/v1_original.py)
F1b Make it modular in diffusion_models/ (GPT-2 tokenizer, the project's token files, plain text, a likelihood bound to compare with), plus a Colab notebook; the colab run mode matches the 51M autoregressive TinyStories model (164M tokens) Claude ✅
F1c Train it on TinyStories on Colab and check it generates Colab ✅ 164M tokens, 49 min on an L4 (59,000 tokens/s): likelihood bound 2.525 nats/token (the 51M autoregressive model's loss: 1.583). It learned only word frequencies for ~2,500 updates, then used context (6e-4 beat 2e-3 in a 2,000-update check). Coherent stories once revealed tokens are kept fixed while sampling (v1's re-sampling of every position broke the final text; kept as sampler="v1")
F2 Train the same diffusion model on Mini FineWeb-Edu (small) on Colab Claude ✅ 164M tokens, 50 min on an L4: likelihood bound 5.269 nats/token, still falling (the plateau lasted ~3,000 updates). Confident-first sampling collapses into one repeated word ("fish, fish, fish"); revealing in random order at temperature 0.8 (sampler="random") gives varied, loosely fluent text without much coherence: 164M tokens is little for FineWeb-Edu at 51M parameters (our base AR model saw 500M tokens at 124M)
F2b Fix the diffusion models' generation Claude ✅ diffusion_models/quality.py scores the text on held-out prompts (repetition, the base model's loss on it). Confident-first sampling copied neighbouring words (43% of 4-token runs repeated on FineWeb-Edu); a repetition penalty of 1.3 (the new default) brings it to 4% on FineWeb-Edu and 0.4% on TinyStories at about real text's judge loss. Blocks, left-to-right order, re-masking, random order, top-p and noise-level choices were measured too (README). FineWeb-Edu text is now on topic but loosely connected: the model's limit; a longer run would be the next lever
F2c A longer FineWeb-Edu run: colab-long, 60,000 updates = 492M tokens (3×), about 2 h 30 min on an L4 Colab ✅ 14 Nov: likelihood bound 4.729 (5.269 for the 164M-token run), 145 min, 59,300 tokens/s; bound 6.06 at 6,000 updates, 5.59 at 12,000, 5.32 at 20,000, 4.73 at 60,000: still falling slowly at the end
F3 Integrate it into the app (Lab, Generate, Explorer) Claude ✅ 18 Nov: the Diffusion page (replay the denoising of a held-out prompt, a step × position grid, the trained runs' table); the Side by side page (H1) next to an autoregressive model; and Generate itself: the model list has the diffusion runs, and picking one swaps the autoregressive-only controls for passes, which tokens stay and blocks, starts the sliders at F4's settings (temperature 1, top-k 50, repetition penalty 1.2) and replays the passes in the output, coloured by confidence, with the time per pass
F4 Compare on both datasets: the TinyStories diffusion model against the autoregressive Medium 51M · TinyStories · 20k steps (the same size, tokens and schedule as the colab run mode), and the FineWeb-Edu one against the base model; on the prompt set's TinyStories and FineWeb-Edu prompts: the outputs, loss (the diffusion loss is an upper bound, ours is exact), steps and speed Claude ✅ 17 Nov: diffusion_models/compare.py (outputs/f4/), journey section 38. 124M pair (base model vs F6 2.5B, the same size, data, tokens and tokenizer): on the same 64 × 256-token held-out windows, AR loss vs diffusion bound 3.296 vs 4.148 on FineWeb-Edu (+0.85), +1.13 to +1.19 on Cosmopedia, +0.52 to +0.77 on TinyStories, WikiText and Shakespeare. Benchmarks: average 42.3 vs 35.4. Writing (8 prompts × 2 seeds, 64 tokens, judged by GPT-2 small; real text 3.118): AR sampled 2.982 at 101 tokens/s with 1.4% repetition; diffusion 4.480 / 3.622 / 2.828 at 16 / 32 / 64 passes (113 / 55 / 28 tokens/s, 3.2 / 4.8 / 9.7% repetition). 51M TinyStories pair: gap +0.95, averages 33.4 vs 30.7. Each diffusion pass takes 36 ms against 10 ms for an AR step with its KV cache
F5 Benchmark the diffusion models like the autoregressive ones: the held-out losses (the likelihood bound on the same held-out sets), the five benchmarks (ARC-Easy, PIQA, HellaSwag, Winogrande scored by each choice's likelihood bound; LAMBADA by masking the last word), and generation quality (repetition, the AR base model's loss on the text, speed), shown on the Benchmarks page next to the AR models and GPT-2 Claude ✅ diffusion_models/benchmarks.py (each choice scored by a Monte Carlo likelihood bound, 8 maskings); the Benchmarks page shows diffusion runs; the full runs on both models are going on the Mac (14 Nov). FineWeb-Edu (164M-token run): HellaSwag 25.3% (length-normalized, as for the others), LAMBADA 0.2%, ARC-Easy 31.2%, PIQA 52.1%, Winogrande 50.5% (the 124M AR base: 29.7, 15.7, 39.4, 61.1, 50.0); TinyStories (164M tokens): 25.7, 1.8, 25.7, 51.9, 48.4, against 26.5, 8.7, 25.9, 54.1, 51.6 for the autoregressive model of the same size and tokens (benchmarked for this): both at chance except LAMBADA. The long run (492M tokens): 26.6, 2.2, 29.2, 52.2, 51.9: better on LAMBADA and HellaSwag. Generation quality ✅ (F2b); the Benchmarks page shows all three diffusion runs next to the AR models and GPT-2 ✅ (checked 15 Nov; the two chat models' columns now say how each was tuned)
F6 A diffusion model the size of the base, on the stage 2 data (option B, chosen 15 Nov): a 124M masked diffusion model with the base's design (12 layers × 768, RoPE, SwiGLU, no biases, QK-norm, 512 context; bidirectional attention, the masking objective) trained from scratch on the stage 2 mix (Mini FineWeb-Edu full + the same Cosmopedia subsets, small's held-out documents removed; the token files already cached on Drive). 1B tokens first (~8 h on an L4, snapshots every 10,000 steps), then extended to 2.5B tokens (~12 h more) if the likelihood bound is still falling, so the comparison with the stage 2 AR model is like for like: same size, data, tokens and tokenizer, only the objective differs. A new run mode in diffusion_models/config.py (the author's v1_original.py untouched); measured like F2–F5 (bound, quality, five benchmarks). Later, as a separate experiment: converting the AR model into a diffusion model (option C, Dream-7B / DiffuLLaMA style, ~0.5B tokens) as a quick strong base for I4–I6 Claude, Colab code ready (15 Nov): run mode f6-124m (123.6M parameters, 61,000 updates × 32 × 512 = 1.0B tokens, lr 6e-4, snapshots every 10,000 updates), denoiser arch="gpt" (the base's blocks with causal=False in model.py, plus the noise-level embedding), train.py --mix stage2 --data-cache …, --from-snapshot, and --steps 152600 --resume re-warms onto the 2.5B schedule; smoke-tested on the Mac (f6-smoke on the stage 2 mix with FineWeb-Edu tiny; padding, bidirectionality, extension; old diffusion runs and the AR models unchanged). Runs after E2 and after D1/D2 are redone on the stage 2 model (it needs the L4; ask before starting). ✅ 1B tokens done (16 Nov): 2 h 8 min on a Colab A100 (140K tokens/s); diffusion_models/runs/f6-124m-stage2. Likelihood bound 3.923 (the 51M long run: 4.729; the stage 2 AR model's loss on the same validation text: 2.963), still improving slowly at the learning-rate floor. Benchmarks (HellaSwag, LAMBADA, ARC-Easy, PIQA, Winogrande): 27.6, 11.8, 30.7, 54.2, 49.6, average 34.8 (51M: 32.4; AR stage 2: 42.3). Repetition with the recommended sampler 8.4% → 4.6%. Journey section 33, README. Extension to 2.5B tokens running (17 Nov, from 10:00 IST) on a Colab A100 through infra/colab_job.py (J3): resumed at update 61,000 of 152,600 on the stretched schedule (learning rate re-warmed to 4.2e-4), 142.7K tokens/s, done about 13:00 IST; progress in MyDrive/diffusion-runs/f6-124m-stage2/status.json. ✅ 2.5B tokens done (17 Nov, 16:30 IST): the runtime was restarted at update 103,500, resumed from the save at 102,000 on an A100-80GB (153K tokens/s), 98.5 min for the rest. Best likelihood bound 3.830 at update 120,000 (1B: 3.923); it drifted back up to ~3.9 over the last 30,000 updates at the learning-rate floor (not yet explained). Pushed to aramkris/fineweb-edu-124m-diffusion and pulled to diffusion_models/runs/f6-124m-stage2-2p5b (the 1B model stays in f6-124m-stage2). ✅ Measured 17 Nov (infra.eval_queue, 2 h 40 min on the Mac): benchmarks 28.0 · 14.4 · 30.1 · 55.4 · 49.1, average 35.4 (1B: 34.8); repetition with the recommended sampler 4.6% → 11.5% (more training, more loops, as for the AR model after stage 2). Then F4 ✅
G Publish on GitHub Pages
G1 The documentation as a website Claude ✅ 17 Nov (live): pages/build.py turns the README, usage, components, the story (journey.md), analysis, results, the plan, the diffusion README and the five model cards into pages with a side menu and a contents column. Links between docs become links between pages (GitHub's heading anchors kept), images are copied, and links to code go to GitHub. All internal links and anchors check out; it works at phone width. The landing page takes its benchmark table from the results store, so F6 2.5B's numbers appear once tonight's evaluation runs
G2 Browser-only versions of the Journey timeline, the Benchmarks results, the model comparison, and the Explorer replaying the base model's real numbers for a few saved prompts Claude ✅ 18 Nov: the Journey timeline and Side by side (17 Nov), and the Explorer (explorer.html): the app's explorer.js, flow.js and flowplay.js on the base model's real numbers for three prompts, every block in detail, recorded by pages/record.py --only explorer into pages/recorded/explorer/ (7 MB, one file per block so the page loads only the blocks you open). Benchmarks and the model comparison are the generated Results page
G3 Live generation in the browser Claude ✅ 18 Nov: the website's Try it page runs the published Q8_0 GGUF (174 MB, from the S6 Hugging Face repos) in wllama 3.8.1 (llama.cpp in WebAssembly, WebGPU when present); chat and base models, streaming, temperature and length; about 77 tokens/s in the app's browser pane; nothing leaves the browser. Same day: inference metrics from llama.cpp's own timings (time to first token, prefill and decode tokens/s, GFLOP/s, memory read GB/s, context used, tokens reused from the KV cache), a per-token time chart, where the model runs (from the load log: layers on the GPU, weight/KV/scratch memory) and a GPU/CPU switch with a table of runs. On an M-series Mac all 13 layers run on WebGPU: decode 117 vs 38 tokens/s on one CPU thread, prefill 303 vs 67
G4 A GitHub Actions workflow to build and deploy the site, and links from the README Claude ✅ 17 Nov: .github/workflows/pages.yml builds on pushes that touch the docs, results or pages, then deploys. GitHub Pages is switched on (Source: GitHub Actions; the author's go-ahead). Live at a-ramakrishnan.github.io/tinystories-ARM-model, linked from the top of the README
H Visualizations (some ideas borrowed from Vizuara's diffusion explainer, built in our own style)
H1 One page, side by side: what happens in an autoregressive model and in a diffusion model. The same prompt in two lanes (left to right one token per step vs all positions refined in parallel over k steps), a step × position grid for each, causal vs bidirectional attention, the training objective (next token vs masked tokens), and the numbers (steps, speed, loss, repetition) Claude ✅ 17 Nov: the Side by side page (Explore). One prompt goes to an autoregressive model (the base by default) and a diffusion run (F6 2.5B by default: the same size, data, tokens and tokenizer). Both are replayed together, lined up by model pass or by clock time, each with its pass × position grid at the same row height: a staircase of 64 passes against 32 passes filled block by block. Both are scored the same way: passes, time, tokens/s, ms per pass, the base model's loss on each text, repetition, and the loss on the real continuation (exact vs the likelihood bound from 32 maskings). A table compares attention (causal vs bidirectional), the training objective, the cost of a pass and the models. GET/POST /api/sidebyside, app/static/sidebyside.js. By default both sample with top-k 50, temperature 1; greedy is optional. First prompt (FineWeb-Edu, 64 tokens, 32 passes): AR 64 passes in 0.9 s, diffusion 32 passes in 2.7 s (83 ms per pass against 14 ms); real-continuation loss 4.45 against a bound of 5.40. Earlier: autoregressive half (the Writing text page)
H2 A separate set of illustrations for diffusion models: a noising slider (a sentence going from clean to fully masked as t grows, with the mask schedule), a denoising player (from all [MASK], positions revealed by confidence, with a steps-vs-quality slider), the loss on masked positions only, and the architecture flow with bidirectional attention and the time embedding Claude ✅ 15 Nov: the Diffusion page has three new cards. Adding noise: one real text masked at every level t (one random number per position, so raising t only adds masks), the model's guesses (right/wrong) at the hidden positions, the loss on the hidden tokens only and the share guessed right, with the loss-by-t curve and the t/T schedule (POST /api/diffusion/noise, all T levels in one batch). Fewer steps or more?: the same prompt and seed with 4, 8, 16, 32 and 64 steps, each scored by the base model (judge loss) with time and repetition (POST /api/diffusion/steps; 51M long run, one prompt: judge loss 5.5 at 4 steps → 3.2–3.4 at 32–64, real text 3.6). Inside the denoiser: the architecture flow from the run's own config (masked input, time embedding, bidirectional blocks, loss on hidden positions only) and causal vs bidirectional attention grids. The existing denoising player stays as it was
H3 Numbered stage cards in the Flow player (autoregressive): what happens at each stage, its formula and the example's real numbers Claude ✅ 15 Nov: "Play the data flow" shows a strip of the ten numbered stages (embedding, query/key/value, into the head, scores, mask, softmax, weighted values, residual, feed-forward, prediction; click one to jump there) and each step as a card: the stage number, its formula with the model's own sizes (√64, 12 heads, SwiGLU or GELU, RoPE, QK-norm when the model has them) and the real numbers; later blocks fold stages 2–8 into one Attention card
I Reasoning models (added 14 Nov at the author's request): make both kinds of model think before answering. At 124M / 51M the realistic goal is reasoning on small, automatically checkable tasks (multi-digit arithmetic, short word problems, countdown, sudoku-like puzzles), not GSM8K-level maths; the point is to see the mechanics work and compare the two kinds of model after D1
IA The autoregressive reasoning model, written by hand by the author (15 Nov, the author's decision, as with the diffusion model in F1): built on the best model this project produces (the stage 2 model if it beats today's base, else the base mini-fineweb-edu-202611130718-14ed26). The author writes the core (I1–I3 below are the outline: thinking format, test-time compute, RL); Claude then integrates it: app (Chat mode with visible thinking), notebook, evaluation, comparison with the diffusion reasoning model (I7) the author open: after stage 2 picks the best model
I0 Prerequisites. A reasoning eval set with checkable answers: synthetic tasks we generate (arithmetic with carries, countdown to a target, small sudoku/logic grids, unit conversions), plus GSM8K's test set as a stretch goal; exact-match scoring. Longer context for thinking: extend the AR model from 512 to 1024–2048 tokens (RoPE scaling plus a short continued training run), and the diffusion model from 256 to 512+ Claude after D1
I1 AR: thinking format by fine-tuning. Extend D1's format with a visible thought: User: … / Assistant: <think> step by step … </think> answer, in plain text like D1. Data: worked solutions with steps (GSM8K's train solutions, the synthetic tasks' step-by-step traces we generate exactly, short public reasoning traces that fit the context). Loss on the thought and the answer the author (IA); Claude integrates afterwards after I0
I2 AR: test-time compute, no training. Self-consistency (sample N thoughts, majority vote on the answer), best-of-N with a checker, and budget forcing (append "Wait," when it stops thinking too early); measure accuracy against the number of samples and tokens the author (IA); Claude integrates afterwards after I1
I3 AR: reinforcement learning with checkable rewards (D3's GRPO): reward 1 for a correct answer, a little for the right format; groups of samples per question, no value model. Watch whether thoughts get longer and accuracy climbs, as in DeepSeek-R1, on tasks the model can just about solve (curriculum from easy to hard) the author (IA); Claude integrates afterwards after I1 and D3
I4 Diffusion: instruction tuning first. The diffusion version of D1: mask only the answer (the question stays visible), as LLaDA's fine-tuning does; Chat mode on the Diffusion page Claude after D1
I5 Diffusion: thinking. The same <think> traces as I1; the model fills the thought and the answer together and can revise early steps after later ones (diffusion-of-thought), with a semi-autoregressive block order as an option Claude after I4
I6 Diffusion: reinforcement learning (d1 / diffu-GRPO style): the answer's log probability estimated from one masked pass (the likelihood bound used in F5), the same rewards as I3 Claude after I5 and I3
I7 Compare the two kinds on reasoning: accuracy, tokens and time per answer, and how accuracy grows with test-time compute (more samples for AR, more denoising steps for diffusion); planning-style puzzles (sudoku, countdown) are where diffusion models have been reported to do relatively well Claude after I3 and I6
J Infrastructure (added 16 Nov at the author's request; proposed, for the author to pick from) open
J1 Tests and CI. A tests/ folder with fast smoke tests on CPU: a tiny model trains 20 steps and its loss falls, generation with and without the KV cache gives the same tokens, a diffusion model trains and samples (f6-smoke), the benchmark harness runs with --limit, the app's API answers; a GitHub Actions workflow runs them on every push Claude ✅ 17 Nov: tests/test_smoke.py, ten tests in 3.5 s on the CPU (a tiny GPT learns, in the base design and GPT-2's; greedy generation identical with and without the KV cache, RoPE and learned positions; bidirectional attention sees later tokens; the diffusion model trains and samples, both architectures; the job runner). .github/workflows/tests.yml runs them on every push (39 s, green). Not yet: the benchmark harness with --limit and the app's API (they need downloads or a loaded model)
J2 A reproducible environment. Today the Mac uses the system Python 3.9 and requirements.txt has no versions: a pyproject.toml with pinned versions (uv lock file), Python 3.11+ in a virtual environment, the same pins in the notebooks Claude ✅ 17 Nov: pyproject.toml + uv.lock (Python 3.11, 94 packages, PyTorch 2.15): uv sync makes .venv, the smoke tests pass in it. Colab keeps requirements.txt on top of its own PyTorch (pinning it there would download a second copy every runtime). Not yet: switching the app and scripts on the Mac over to .venv (they still run on the system Python 3.9 with PyTorch 2.8)
J3 A resilient Colab job runner. One launcher for every long job (pretraining, stage 2, SFT, diffusion) that resumes from Drive by itself and writes a small status.json heartbeat (step, loss, tokens/s, ETA) to Drive, so progress can be checked, and a dead runtime noticed, without the browser tab; optionally a cloud GPU with a command-line API (Modal, RunPod, Lambda) so jobs don't depend on a browser at all Claude ✅ first version 17 Nov: infra/colab_job.py wraps any training command: output also to job.log on Drive, every progress line (update …/… tokens/s, tqdm bars, eval @, Step N: … val loss) into status.json (step, total, loss, speed, ETA, last evaluation, time written), and --retries restarts a failed command (the trainers' --resume continues). python3 -m infra.colab_job show <dir> summarizes and says stale after 5 minutes without an update. First used for the F6 extension. It can't keep a runtime alive (Colab needs the tab open, or Pro+ background execution); a command-line GPU provider is still open
J4 Model storage and transfer. Models pushed to a private Hugging Face Hub repo (or synced with rclone) from Colab and pulled by the Mac, with checksums in the model library: no browser downloads, zips or temp files Claude ✅ 17 Nov: one public repo per model, the author's choice. infra/model_store.py: push <kind> <folder> <repo> [--card], pull <repo> (into models/, diffusion_models/runs/ or outputs/, from the manifest), list (repos tagged tinystories-arm), verify; a manifest.json with each file's size and SHA-256 at the top of the repo, checked on pull; never overwrites a different local folder without --force; training state stays behind unless --with-state; new repos public (--private otherwise). Model cards kept in docs/model_cards/. Live: fineweb-edu-124m-base, -chat-sft, -chat-dpo (a real pull of the base matched the local copy); F6 2.5B queued on Colab for fineweb-edu-124m-diffusion when training finishes. Push cells at the end of diffusion_colab.ipynb and sft_colab.ipynb. Seven tests against a fake Hub. The earlier shared repos tinystories-arm-models and tinystories-arm-diffusion were deleted on 17 Nov (only duplicates). A collection needs a token with collection permission Renamed 17 Nov after their data: aramkris/fineweb-edu-124m-{base, chat-sft, chat-dpo, diffusion, base-gptq-4bit} (the old tinystories-arm-124m-* names redirect)
J5 One results store. Every evaluation (held-out losses, benchmarks, quality, chat evals) appends a record (model, commit, settings, numbers, time) to one file the app, README tables and docs read from, so the numbers are generated rather than copied by hand Claude ✅ 17 Nov: infra/results.py, the store results/results.jsonl (391 records scanned from the existing result files: benchmarks, held-out losses, generation, chat, DPO held-out, diffusion bounds and quality) and the generated page docs/results.md. The evaluation suite, benchmarks.py, sft.py compare, dpo.py eval, and the diffusion benchmarks.py / quality.py add records as they save; model_store pull scans what it brings down; /api/results in the app; six tests, one fails when the page is out of date. The README and analysis prose still quote numbers by hand
J6 Evaluation queue on the Mac. One command evaluates a new model end to end (losses, quality, the five benchmarks), resumable per task, with memory guards (batch size chosen from free GPU memory; today's out-of-memory crashes) and caffeinate built in Claude ✅ 17 Nov: infra/eval_queue.py. python3 -m infra.eval_queue <model id or diffusion run>: autoregressive models get the suite then the five benchmarks; diffusion runs get the bound (into the results store), quality (every sampler setting, as before) and the five benchmarks, quick ones first and HellaSwag last. Each task's state is kept in <folder>/eval_queue.json, so the same command picks up after a crash (--redo to rerun, --plan to see it, --only to choose). The scoring batch is chosen from free GPU memory (Metal's working-set size minus the model; 32 on the 16 GB M5 for a 512-token 124M model), and halved and retried on an out-of-memory error. The queue runs under caffeinate. --quick runs 20 questions per benchmark and saves nothing; it was checked on F6 2.5B and the base. Six tests

| J7 | Repository layout. The training, evaluation and post-training scripts move from the top level into a package (lm/, eval/, post/), with a single command-line entry point; notebooks import from it | Claude | open | | J8 | Serving. An OpenAI-compatible /v1/chat/completions endpoint on the app's server (streaming), so standard clients and evaluation tools can talk to our models; latency numbers feed the prefill vs decode card | Claude | Claude | open: grown into S below (S2 is J8) |

S. Serving the model (added 18 Nov by the author)

From a model in the app to a model that standard clients can call. Serve the DPO chat model for chat and the stage 2 base for plain completions; the diffusion model can come later through the same API, though standard serving tools don't support it. Suggested order: S1 (it decides how much of S2–S4 we build), S2, S7, S3, S4, then S5 and S6.

Step What Who Status
S1 Check whether our model maps onto Qwen3's format Claude ✅ 18 Nov: it maps exactly. serving/export_hf.py <model id> writes a standard Hugging Face Qwen3 folder (safetensors, the GPT-2 tokenizer, and for chat models a chat template giving exactly sft.py's tokens) into exports/. Two tricks: our block norms are LayerNorm, not RMSNorm, so the residual stream is centered (input embeddings and every matrix writing into it), which makes the two the same function, and the output layer keeps the uncentered embeddings (untied, 329 MB in bfloat16); Qwen3 has one epsilon, so the query and key projections are scaled by √10 to keep QK-norm's 1e-6 (RMSNorm(s·x, ε) = RMSNorm(x, ε/s²)). Base and DPO chat models: largest logit difference 9e-5 / 2e-4 in float32, top token identical on 384/384 held-out tokens, greedy text identical, also loaded from disk with AutoModelForCausalLM from the bfloat16 file. Five tests. Served by llama.cpp: serving/gguf.sh fetches llama.cpp's converter (pinned to the Homebrew build's commit) and writes bfloat16 and 8-bit GGUF files (310 / 164 MB); llama-server --jinja serves them on its OpenAI-compatible API with the exact prompt tokens, streaming and answers that stop at <|endoftext|>, greedy text identical to ours for chat and completions, at 256 tokens/s in bfloat16 and 377 in 8-bit on the M5 (our own generation: ~101). So S2–S4 (our own server, batching, speed) are largely covered by llama.cpp; LM Studio and Ollama load the same GGUF
S2 A standalone server (serve.py), apart from the app's training state (this is J8): load once; /v1/completions, /v1/chat/completions, /v1/models; streaming, stop sequences, max_tokens, temperature, top-p; D1's chat format; finish_reason: "length" at the 512-token context Claude open
S3 Several requests at once: a request queue and batched decoding with one KV cache per sequence (continuous batching); tokens/s against the number of concurrent users (overlaps C4's tokens/s against batch size) Claude after S2
S4 Speed: half-precision weights, torch.compile, the existing speculative decoding; make 4-bit GPTQ actually faster (today's layers turn the weights back into floats every token, which is slower); load-test time to first token and tokens/s (C4's prefill vs decode work) Claude after S3
S5 Packaging: the app and scripts on .venv (J2's last step), a config file, a health check, logging, an optional API key, a Dockerfile (CUDA or CPU; the Mac's GPU isn't available inside Docker) Claude after S2
S6 Where it runs, the author's choice, possibly several: the Mac on the local network or through a tunnel; a Hugging Face Space or Inference Endpoint; Ollama or LM Studio through GGUF or MLX if S1 works the author decides first part ✅ 18 Nov: published for anyone to run, one public repo per model with both formats (the author's choice): fineweb-edu-124m-base-qwen3 and fineweb-edu-124m-chat-dpo-qwen3 (Qwen3 safetensors for transformers, bf16 and Q8_0 GGUF for llama.cpp, Ollama, LM Studio), by serving/publish.py with cards in docs/model_cards/; checked by pulling them back (llama-server -hf …:Q8_0, AutoModelForCausalLM.from_pretrained). A hosted endpoint of our own is still open
S7 Check the served model Claude ✅ 18 Nov: the served model is the model. serving/check.py. llama-server can't score given text (it returns log probabilities only for tokens it generates), so the check has two parts. Engine: an adapter puts llama.cpp (llama-cpp-python on the GGUF) behind our model's interface, so our own held-out loss and benchmark code run unchanged: bfloat16 held-out losses within 0.0013 of the stored ones on all 10 datasets (Mini FineWeb-Edu 3.2959 vs 3.2957), every benchmark within 0.2 points, average 42.34 vs 42.35; 8-bit (Q8_0) within 0.001 on held-out loss, average 42.22. API: llama-server over HTTP: LAMBADA by greedy generation on /v1/completions 20.36% vs 20.34% (5,153 passages); the DPO chat model's 100 held-out questions (dpo.py eval) on /v1/chat/completions: repetition 6.6% vs 6.5%, answers that end 85% vs 86%, length 129.1 vs 129.1, prompt tokens identical 100/100, 70/100 answers identical word for word (bfloat16 vs our float32: a near-tie flips a token and the answer drifts); 8-bit 6.7% / 82% / 133.2. Results in outputs/s7/ and the results store (kind served); two tests

Open choices

  • ~~Stage 2 (E2) before or after post-training (D)~~ decided on 14 Nov: instruction tuning (D1) first, on today's base, then stage 2; D1 is redone on the stage 2 model afterwards.
  • Where GitHub Pages (G) goes: it's independent of the rest, and most worth doing once there's more to show.
  • What C3 covers depends on the author's ideas.
  • Reasoning (I): which synthetic tasks to start with, and whether to extend the context (I0) before or after stage 2; RL (I3, I6) builds on D3's GRPO, so D3 comes first.
  • ~~How to scale the diffusion model~~ decided 15 Nov: F6, option B (124M, the stage 2 data, 1B tokens then 2.5B); converting the AR model (option C) stays a later experiment.
  • The visualizations (H) can replay saved data, so they can also go on GitHub Pages (G2); H1's diffusion half and H2 come with the diffusion integration (F2–F4).

Why the order: the benchmarks showed that the remaining gap to GPT-2 is mostly data (more of it, and more kinds), so data and inference come before further architecture work. Post-training changes behaviour more than knowledge; it's cheap (~1 h), so it's done on today's base first and redone on the stage 2 base. GitHub Pages can only serve static files, so it shows saved results; training, the Lab and live generation stay in the local app.