From docs/model_cards/fineweb-edu-124m-chat-dpo.md
license: mit language: en library_name: pytorch pipeline_tag: text-generation tags: - tinystories-arm - gpt - from-scratch datasets: - BaoMinhTran/mini-fineweb-edu - HuggingFaceTB/cosmopedia - HuggingFaceTB/smol-smoltalk base_model: aramkris/fineweb-edu-124m-chat-sft
FineWeb-Edu 124M chat (SFT + DPO)
A chat version of FineWeb-Edu 124M base, a GPT trained from scratch in tinystories-ARM-model. The SFT model plus DPO against repetition: 2,000 smol-smoltalk questions, the reference answer preferred to the SFT model's own greedy one; 19 minutes on an M5 Mac.
It answers in a plain User: … / Assistant: … format (no special tokens) and ends its answer with
<|endoftext|>. Measured on 100 held-out questions with greedy decoding:
| Model | Loss on the reference answers | Repeated 4-grams | Answers that end | Benchmark average |
|---|---|---|---|---|
| SFT | 1.156 | 12.4% | 70% | 38.5 |
| this model (SFT + DPO) | 1.196 | 6.5% | 86% | 38.4 |
The base model scores 42.3 on the same five benchmarks; instruction tuning costs about 4 points. DPO halves the repetition and lets most answers end by themselves, at a small cost in reference-answer loss.
Use
git clone https://github.com/a-ramakrishnan/tinystories-ARM-model && cd tinystories-ARM-model
python3 -m infra.model_store pull fineweb-edu-124m-chat-dpo # -> models/chat-mini-fineweb-ed-202611152150-2746ba
import tiktoken
from inference import load_model, load_model_config
from sft import answer
d = "models/chat-mini-fineweb-ed-202611152150-2746ba"
cfg, _ = load_model_config(f"{d}/model_config.json")
model = load_model(cfg, f"{d}/best_model_params.pt", "cpu")
print(answer(model, tiktoken.get_encoding("gpt2"), "Why do leaves change color in autumn?", "cpu"))
It answers in the right shape but with the base model's knowledge: plausible-sounding and often wrong on facts.