From docs/model_cards/fineweb-edu-124m-base.md
license: mit language: en library_name: pytorch pipeline_tag: text-generation tags: - tinystories-arm - gpt - from-scratch datasets: - BaoMinhTran/mini-fineweb-edu - HuggingFaceTB/cosmopedia
FineWeb-Edu 124M base (stage 2)
A 124M-parameter GPT trained from scratch in tinystories-ARM-model: written by hand, improved one measured change at a time, trained on a Mac and on Colab GPUs. Related models: its chat versions (SFT, SFT + DPO) and a masked diffusion model of the same size and data (fineweb-edu-124m-diffusion).
Design: 12 layers × 768, 12 heads, RoPE, SwiGLU, no biases, QK-norm, 512-token context, GPT-2's tokenizer. Data: 2.5B tokens of Mini FineWeb-Edu (full) plus Cosmopedia subsets (stories, WikiHow, OpenStax, Stanford, Khan Academy), with the held-out documents of Mini FineWeb-Edu small removed; 152,600 updates.
| This model | OpenAI GPT-2 small (124M) | |
|---|---|---|
| Held-out Mini FineWeb-Edu (loss, lower is better) | 3.296 | 3.401 |
| TinyStories / WikiText-2 / Shakespeare (loss) | 2.751 / 5.047 / 5.446 | 2.553 / 4.180 / 4.807 |
| HellaSwag | 32.3% | 31.1% |
| ARC-Easy | 46.8% | 43.8% |
| PIQA | 62.4% | 62.9% |
| LAMBADA | 20.3% | 32.6% |
| Winogrande | 49.9% | — |
Zero-shot, lm-evaluation-harness style.
Use
The architecture is the project's own (model.py), so load it with the repository's code:
git clone https://github.com/a-ramakrishnan/tinystories-ARM-model && cd tinystories-ARM-model
python3 -m infra.model_store pull fineweb-edu-124m-base # -> models/mini-fineweb-edu-ful-202611151103-bb160c
import tiktoken, torch
from inference import load_model, load_model_config
d = "models/mini-fineweb-edu-ful-202611151103-bb160c"
cfg, _ = load_model_config(f"{d}/model_config.json")
model = load_model(cfg, f"{d}/best_model_params.pt", "cpu")
enc = tiktoken.get_encoding("gpt2")
ids = torch.tensor([enc.encode("Photosynthesis is")])
print(enc.decode(model.generate(ids, 60, temperature=0.8, top_k=50)[0].tolist()))
Files: best_model_params.pt (weights), model_config.json, meta.json (how it was trained), eval.json and
benchmarks.json (results), manifest.json (checksums).
It writes fluent educational-style English; at 124M parameters it knows few facts reliably, and greedy decoding repeats itself (use temperature and top-k, or a repetition penalty).