Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090

AI summary

Overview

• Research area: Natural Language Processing — low-cost, reproducible pretraining of large language models (the paper's arXiv identifier is 2608.27370v2 [cs.CL]).
• Technical level: Intermediate. The recipe spans optimizers, low-precision numerics, parallel training systems, and data curation, but the paper is written as a cost-and-reproducibility report rather than a theory paper.
• Scope: A full open pretraining recipe — hardware, FP8 precision, optimizer, data curriculum, data selection, cost accounting, and post-training case study — used to train a collection of roughly 2B-parameter "Puro-2B" models from scratch for under $6.9K in compute.

What This Paper Is About

Pretraining language models is so expensive that most academic and open-source labs cannot run or reproduce it, even at small scale: the authors estimate that training Llama-3.2-3B costs over $1.5M and reproducing SmolLM3-3B needs over $700K (stated elsewhere as $719K), while OLMoE-1B-7B reproduction would cost $200K. The goal is to show that a fully open, hardware-accessible recipe can produce a competitive small model at a cost that an individual "poor lab" can afford. Using consumer-grade RTX 5090 GPUs and FP8 training, the authors train Puro-2B from scratch on up to 1.4 trillion tokens for a measured production compute cost of about $6.9K.

Key Contributions

1. An affordable from-scratch pretraining recipe and model collection. The Puro-2B collection is trained with two-phase pretraining (438.8B Phase 1 tokens and 960.0B Phase 2 tokens, 1.4T scheduled tokens total) at a compute cost below $6.9K. Under the paper's evaluation protocol the best checkpoint surpasses Qwen2-1.5B overall and approaches Qwen2.5-1.5B.
2. A recipe-specific cost scaling relationship. The "Puro Cost Scaling Law" relates rental-equivalent training budget to average model performance for scale-down settings, fitted from Phase 2 training results with uniform data at different token budgets. The fitted law suggests about $4.4K — less than the $5,090 in the title — is sufficient to reach Qwen2-1.5B performance.
3. Ablations of the efficiency stack. The paper reports cost-efficiency estimates for RTX 5090 infrastructure, blockwise FP8, MuonH, and curriculum model averaging (CMA), plus a post-training case study showing that Phase 2 recipe differences persist through supervised adaptation.
4. Full artifact release under Apache 2.0. Data manifests and materialized components, training code (Puro-Megatron), data-processing code (Kaiyuan-Spark), and 10 checkpoint versions, published on Hugging Face and GitHub, including intermediate checkpoints and training configurations.

Main Findings

• Production cost and schedule: The canonical run uses 438.84B Phase 1 tokens on 24 GPUs over 10.43 days ($1.84K) and 959.99B Phase 2 tokens on 96 GPUs over 7.16 days ($5.05K), totaling 22,514 active-training GPU-hours, $6.89K, and 17.6 elapsed days. The uniform-data (UD) run uses 480.00B Phase 2 tokens over 3.58 days ($2.53K), totaling 14,262 active GPU-hours and $4.37K.
• Benchmark scope: Performance is measured as the average over 15 mathematics, code, reasoning, and knowledge benchmarks reported in Tables 3 and 4 of the paper.
• Surpasses baselines at lower cost: Puro-2B surpasses Qwen2-1.5B and Gemma-2-2B while using less than one sixth of the stated training cost of comparable open recipes. The $4.4K UD checkpoint exceeds Qwen2-1.5B in average performance over the 15 benchmarks.
• Hardware cost efficiency: On the reported price assumptions, RTX 5090 achieves 2.43 EFLOP/USD in BF16 and 4.87 EFLOP/USD in FP8, versus 0.89 and 1.78 for H200 SXM5 and 0.63 for A100 SXM4 in BF16. The paper states RTX 5090 BF16 and FP8 cost efficiency is about 2.7 times that of H200.
• FP8 quality and speed: Blockwise FP8 increases validation loss by 0.0031–0.0039 relative to BF16 across five tested model sizes in a matched 20-token-per-parameter scaling ladder; a shared-shape fit maps this to 98.0% BF16-equivalent compute retention. At the 1.7B scale it improves median training throughput by 1.36 times, giving a quality-adjusted net gain of 1.34 times. FP8 accounts for 72% of theoretical Tensor Core operations.
• Achieved utilization: Phase 1 sustains median 238 TFLOP/s per GPU on 24 GPUs across three nodes (global batch size 1536), and Phase 2 sustains median 192 TFLOP/s per GPU on 96 GPUs. The mixed-precision effective MFU under the precision-weighted peak convention (an effective peak of about 327 TFLOP/s per GPU) is approximately 73%.
• Communication tweaks matter: Enabling PCIe P2P on RTX 5090 raised one-way bandwidth from 31.5 GB/s to 56 GB/s, bidirectional bandwidth from 32 GB/s to 111 GB/s (theoretical limit 128 GB/s), and cut latency from 14.3 microseconds to 0.4 microseconds. Eight-GPU AllReduce busbw rose from 14.75 GB/s to 27.34 GB/s, and from 16.33 GB/s to 46.31 GB/s for four GPUs on one CPU socket (about 2.8 times). Enabling GPUDirect RDMA improved 24-GPU (3 servers) AllReduce busbw from approximately 8.87 GB/s to 19.93 GB/s.
• Curriculum beats uniform ordering under LR decay: At the $6.9K reproduction cost, the data curriculum with LR decay (CD) records 1.65 times cost-efficiency gains over the uniform data recipe, and adding curriculum model averaging (CMA) yields 2.40 times gains relative to the uniform scaling curve.
• Post-training effects persist: In GSM8K-based SFT and Math&Code SFT with replay, CMA-based SFT yields higher GSM8K accuracy in repeated runs; in Tulu-3 mixed-domain SFT it also improves comprehensive capabilities and instruction following.
• Narrow cost definition: The headline cost covers only the compute of rerunning the finalized two-phase pretraining once. It excludes data acquisition and preprocessing, proxy experiments, scaling and ablation studies, failed runs, post-training, evaluation, checkpoint averaging, research labor, CPU/storage/networking, taxes, and depreciation.
• Not reported: The truncated paper content does not report per-benchmark scores, the exact fitted functional form of the Puro Cost Scaling Law, or the detailed recipe contents of the evaluation Tables 3 and 4.

Methodology in Plain English

The authors treat pretraining cost as an engineering problem with several separate bottlenecks, and they attack all of them at once.

First, they train on consumer RTX 5090 GPUs instead of data-center accelerators, because the papers' price analysis shows more raw throughput per dollar on the consumer part, even though it has less memory (32 GB) and no NVLink. To make this work they use a modified open NVIDIA driver plus platform configuration changes to re-enable PCIe peer-to-peer communication between GPUs, and a modification to the CUDA user-space driver to enable GPUDirect RDMA between servers. They connect servers with a 400 Gbps InfiniBand network.

Second, they train in blockwise FP8 precision from random initialization, with no BF16 warm-up. Only the Transformer linear-layer matrix multiplications run in FP8 (E4M3 operands); master weights, optimizer states, and numerically sensitive operations stay in BF16 or FP32. Quantization scales are computed online for small groups — 128 consecutive values along the reduction dimension for activations and their gradients, and 128×128 blocks for weights.

Third, for the optimizer they wrap the Muon optimizer in a "Hyperball" constraint, applied to approximately scale-invariant matrices such as attention and MLP weights, while AdamW handles embeddings, normalization layers, the language-model head, and other parameters. Each wrapped matrix is projected back to its initial Frobenius radius after every update.

Fourth, training runs in two phases. Phase 1 processes 438.8B tokens. Phase 2 has three recipe variants: uniform data ordering with learning-rate decay (UD), a source-local data curriculum with learning-rate decay (CD), and curriculum model averaging (CMA), which adds a late constant-learning-rate continuation followed by averaging six checkpoints. The curriculum sorts examples within each dataset from lower to higher quality score, chunks them, and aligns chunks across sources by normalized within-source rank, so better-scored data appears later in training.

Fifth, because they have far more data than budget, they build a corpus from public sources spanning web text, code, mathematics, and synthetic examples, deduplicate the web portion with their Kaiyuan-Spark framework, and use "proxy experiments" — small controlled training runs on candidate data sources or slices — to produce benchmark-score profiles that guide source selection, mixture weights, and within-source filtering thresholds.

Finally, they distill the comparison into a cost accounting protocol, fit a scale-down cost-versus-performance curve from Phase 2 runs at different token budgets, and validate the design choices with ablations and with post-training case studies (GSM8K SFT, Math&Code SFT with replay, and Tulu-3 mixed-domain SFT) in which CMA and UD initializations receive identical supervised data and optimization.

Why This Matters

Impact on research. The paper argues that open weights and open recipes are not the same as affordable reproducibility. By releasing datasets, code, intermediate checkpoints, training configurations, and a transparent cost accounting, it gives academic labs a complete pretraining pipeline they can rerun, inspect, and modify at a budget measured in thousands of dollars rather than millions. It also introduces a cost-based scaling relationship aimed specifically at scale-down settings, a regime that mainstream scaling laws largely leave unaddressed.

Real-world applications:

• Academic and small-lab model development: A university group can pretrain, fine-tune, and study a 2B-class model on modest hardware rather than only evaluating released checkpoints.
• Domain-specific small models: The recipe and released tooling can be reused to train compact models on specialized corpora where a 2B model is sufficient.
• Low-cost deployment: A model of this size can run on constrained or on-premises hardware where large hosted models cannot be used.
• Reproducibility auditing: Because the cost ledger separates GPU-hours, tokens, and unit rates, others can recompute or challenge the cost claims rather than take them on trust.

Industry relevance. The results question the assumption that data-center GPUs are always the economically correct choice for small-scale pretraining, and they show how engineered low-precision training and communication fixes on consumer hardware can reach roughly 73% effective MFU on this workload. The paper also documents vendor restrictions (disabled PCIe P2P, no NVLink, no GPUDirect RDMA on non-data-center GPUs, driver and EULA constraints) that practitioners on consumer clusters must grapple with.

Future Directions

• Push the cost frontier lower. The authors explicitly state they expect future efforts to reduce cost and increase efficiency further; reproducing the recipe on newer or cheaper hardware is the natural next step.
• Extend the cost scaling law. The reported law is recipe-specific and fitted from Phase 2 runs with uniform data at different token budgets. Whether it transfers to other architectures, scales, data mixtures, or recipes beyond CD and CMA is open.
• Broaden the end-to-end post-training study. The current case study covers GSM8K-based SFT, Math&Code SFT with replay, and Tulu-3 mixed-domain SFT; how pretraining curricula shape other downstream adaptations (for example reinforcement-learning post-training) is not established.
• Consolidate the hardware modifications. The driver-level changes enabling GPU P2P and GPUDirect RDMA are unsupported by the vendor and cannot be disclosed in detail; whether equivalent performance can be reached through supported configurations (for example a PLX PCIe switch or a more balanced network topology) is raised as a prospective change rather than a solved question.

Target Audience

This paper is most useful to researchers and engineers who want to pretrain or reproduce language models on limited budgets: academic NLP groups, small labs and startups, and infrastructure engineers who work with consumer GPUs and low-precision training. It also suits readers interested in scaling laws framed around cost rather than FLOPs, and reviewers or practitioners who want to audit claims about open and reproducible pretraining. Readers looking for a purely architectural or algorithmic novelty paper may find the contribution more engineering- and accounting-oriented, since the individual techniques (FP8 blockwise training, MuonH, checkpoint averaging, proxy-based data selection) are adapted from prior work and combined rather than newly invented here.

Authors’ abstract

Language model pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities. Although strong open-source efforts already exist, including open-weight models and open-source training recipes, a cost-efficient, hardware-accessible, and open-source pretraining recipe has long been missing. Even at a small scale, training Llama-3.2-3B costs over \$1.5M, and reproducing SmolLM3-3B needs over \$700K. In this report, we present an open pretraining recipe designed to lower this barrier. Using this recipe, we train a collection of Puro-2B models from scratch on up to 1.4 trillion tokens with FP8 precision on consumer-grade RTX 5090 GPUs. The models in the collection differ in token budgets and selected recipe variants. Our best model is trained at a compute cost of less than \$6.9K and approaches Qwen2.5-1.5B performance under our evaluation protocol. This cost efficiency is enabled by a combination of approaches, including hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and the data recipe. Beyond the recipe itself, we provide two additional results. First, across the Puro-2B collection, we derive a Puro Cost Scaling Law that relates training cost to average model performance; the fitted law suggests that about \$4.4K, less than \$5,090, is sufficient to reach the performance of Qwen2-1.5B. Second, as an end-to-end case study, we examine how pretraining data curricula shape downstream performance after post-training. Such controlled studies are enabled by having access to the full pretraining pipeline rather than model weights alone. We release the full training recipe for Puro-2B, including data, code, and model weights under Apache 2.0 at https://huggingface.co/collections/thu-pacman/puro-2b.