From docs/model_cards/fineweb-edu-124m-base-gptq-4bit.md
license: mit language: en library_name: pytorch pipeline_tag: text-generation base_model: aramkris/fineweb-edu-124m-base tags: - tinystories-arm - gpt - from-scratch - gptq - 4-bit - quantized datasets: - BaoMinhTran/mini-fineweb-edu - HuggingFaceTB/cosmopedia
FineWeb-Edu 124M base, GPTQ 4-bit
The FineWeb-Edu 124M base model with its weights
stored as 4-bit integers, quantized with GPTQ through llm-compressor's oneshot, in
tinystories-ARM-model. 84 MB instead of 494 MB.
How it was made (quantization/gptq.py):
- Our GPT, wrapped as a Hugging Face model (
quantization/hf_wrapper.py), went throughoneshot(recipe=GPTQModifier(scheme="W4A16", targets="Linear", ignore=["re:.*lm_head"]), pipeline="sequential"). That means 4-bit weights (−8…7 × one scale per group of 128) in all 48 transformer-block matrices, and 16-bit activations. - It was calibrated on 256 windows of 512 tokens from Mini FineWeb-Edu's training split, in 44 s on a Mac CPU.
- The integers and scales were then read into the project's own quantized layer. The shared embedding / output layer is stored at 8 bits, one scale per row.
Held-out loss, on the same 16,384 tokens per set (lower is better). The round-to-nearest column is a 4-bit copy with no calibration data, shown for comparison:
| Original (32-bit) | Round to nearest, 4-bit | This model (GPTQ) | |
|---|---|---|---|
| Weight file | 494 MB | 87 MB | 84 MB |
| Mini FineWeb-Edu | 3.296 | 3.326 | 3.319 |
| TinyStories | 2.751 | 2.793 | 2.774 |
| Cosmopedia (5 sets) | 2.336–2.785 | +0.035 to +0.041 | +0.030 to +0.037 |
| WikiText-2 | 5.047 | 5.069 | 5.072 |
| Tiny Shakespeare | 5.446 | 5.472 | 5.481 |
| Mean change over 10 sets | +0.034 | +0.030 |
The benchmarks (HellaSwag, ARC-Easy and the rest) haven't been run on the quantized model yet; see the base model's card for the original's.
Use
The quantized format is the project's own, so load the model with the repository's code. llm-compressor is only needed to make a GPTQ copy, not to run one:
git clone https://github.com/a-ramakrishnan/tinystories-ARM-model && cd tinystories-ARM-model
python3 -m infra.model_store pull fineweb-edu-124m-base-gptq-4bit
import tiktoken, torch
from inference import load_model, load_model_config
d = "models/mini-fineweb-edu-ful-202611151103-842ab9"
cfg, _ = load_model_config(f"{d}/model_config.json")
model = load_model(cfg, f"{d}/best_model_params.pt", "cpu") # recognises the quantized file
enc = tiktoken.get_encoding("gpt2")
ids = torch.tensor([enc.encode("Photosynthesis is")])
print(enc.decode(model.generate(ids, 60, temperature=0.8, top_k=50)[0].tolist()))
The model can also be used in the project's app (Generate, and the Quantize page, which compares it with the original).
Files:
best_model_params.pt: the 4-bit integers packed two per byte, their scales, and the norms;model_config.json,meta.json: the recipe;quantize.json: the comparison above, with a sample from each model;manifest.json: checksums.
Speed. The weights are turned back into floats on every token, so on a Mac it decodes at about 30 tokens/s against 80–100 for the float model. It saves memory, not time.