Hugging Face Daily Papers · August 31, 2026 · 4 min read

Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090

#model-release #open-source #gpu #security

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

How much would it cost to pretrain a 2B LLM from scratch? $1M? $100K?\nPuro-2B matches Qwen2-1.5B for under $5,090, trained on RTX 5090s — with a fully open training recipe!\n","updatedAt":"2026-08-31T06:10:12.337Z","author":{"_id":"66a087eff89441587fdc53d0","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66a087eff89441587fdc53d0/riitxI2xqgJrxx0sQn9VF.jpeg","fullname":"Harry Chen","name":"harryleafchen","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9604126811027527},"editors":["harryleafchen"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/66a087eff89441587fdc53d0/riitxI2xqgJrxx0sQn9VF.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.27370","authors":[{"_id":"6a91275ea64059bab69c36fb","name":"Kairong Luo","hidden":false},{"_id":"6a91275ea64059bab69c36fc","name":"Jiarui Cui","hidden":false},{"_id":"6a91275ea64059bab69c36fd","name":"Yaorui Yin","hidden":false},{"_id":"6a91275ea64059bab69c36fe","user":{"_id":"66a087eff89441587fdc53d0","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66a087eff89441587fdc53d0/riitxI2xqgJrxx0sQn9VF.jpeg","isPro":false,"fullname":"Harry Chen","user":"harryleafchen","type":"user","name":"harryleafchen"},"name":"Shengqi Chen","status":"claimed_verified","statusLastChangedAt":"2026-08-28T13:20:43.004Z","hidden":false},{"_id":"6a91275ea64059bab69c36ff","name":"Yiming Yang","hidden":false},{"_id":"6a91275ea64059bab69c3700","name":"Linxiang Gao","hidden":false},{"_id":"6a91275ea64059bab69c3701","name":"Yanmohan Wang","hidden":false},{"_id":"6a91275ea64059bab69c3702","name":"Mingzhe Zhang","hidden":false},{"_id":"6a91275ea64059bab69c3703","name":"Kaiyue Wen","hidden":false},{"_id":"6a91275ea64059bab69c3704","name":"Kaifeng Lyu","hidden":false},{"_id":"6a91275ea64059bab69c3705","name":"Wenguang Chen","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/66a087eff89441587fdc53d0/tEwHehG7GEUmUEAps7ijn.mp4","https://cdn-uploads.huggingface.co/production/uploads/66a087eff89441587fdc53d0/p3aNYC8ASjkLTLcgD6-aS.png","https://cdn-uploads.huggingface.co/production/uploads/66a087eff89441587fdc53d0/ELfrHchmAI0goe6BIHZwo.png","https://cdn-uploads.huggingface.co/production/uploads/66a087eff89441587fdc53d0/ff7UFe0LpYO-i7NUqk6SE.png","https://cdn-uploads.huggingface.co/production/uploads/66a087eff89441587fdc53d0/am0WOVjdIt-6TbuSbTNRr.png"],"publishedAt":"2026-08-27T00:00:00.000Z","submittedOnDailyAt":"2026-08-31T00:00:00.000Z","title":"Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090","submittedOnDailyBy":{"_id":"66a087eff89441587fdc53d0","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66a087eff89441587fdc53d0/riitxI2xqgJrxx0sQn9VF.jpeg","isPro":false,"fullname":"Harry Chen","user":"harryleafchen","type":"user","name":"harryleafchen"},"summary":"Language model pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities. Although strong open-source efforts already exist, including open-weight models and open-source training recipes, a cost-efficient, hardware-accessible, and open-source pretraining recipe has long been missing. Even at a small scale, training Llama-3.2-3B costs over \\1.5M, and reproducing SmolLM3-3B needs over 700K. In this report, we present an open pretraining recipe designed to lower this barrier. Using this recipe, we train a collection of Puro-2B models from scratch on up to 1.4 trillion tokens with FP8 precision on consumer-grade RTX 5090 GPUs. The models in the collection differ in token budgets and selected recipe variants. Our best model is trained at a compute cost of less than \\6.9K and approaches Qwen2.5-1.5B performance under our evaluation protocol. This cost efficiency is enabled by a combination of approaches, including hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and the data recipe. Beyond the recipe itself, we provide two additional results. First, across the Puro-2B collection, we derive a Puro Cost Scaling Law that relates training cost to average model performance; the fitted law suggests that about 4.4K, less than \\$5,090, is sufficient to reach the performance of Qwen2-1.5B. Second, as an end-to-end case study, we examine how pretraining data curricula shape downstream performance after post-training. Such controlled studies are enabled by having access to the full pretraining pipeline rather than model weights alone. We release the full training recipe for Puro-2B, including data, code, and model weights under Apache 2.0 at https://huggingface.co/collections/thu-pacman/puro-2b.","upvotes":12,"discussionId":"6a91275ea64059bab69c3706","projectPage":"https://huggingface.co/collections/thu-pacman/puro-2b","githubRepo":"https://github.com/thu-pacman/Puro-Megatron","githubRepoAddedBy":"user","ai_summary":"A cost-efficient open-source pretraining recipe trains 2B-parameter models on consumer GPUs for under $7K, yielding performance near larger baselines while deriving cost scaling laws and studying data curricula.","ai_keywords":["FP8 precision","hyperball optimization","curriculum model averaging","cost scaling law","pretraining data curricula","low-precision training"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":37,"organization":{"_id":"69076fd2eb8bfb94ee8b0969","name":"thu-pacman","fullname":"PACMAN Group, Tsinghua University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/66a087eff89441587fdc53d0/2Jw62jk56H9sHnM-Ik4Hz.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"66a087eff89441587fdc53d0","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66a087eff89441587fdc53d0/riitxI2xqgJrxx0sQn9VF.jpeg","isPro":false,"fullname":"Harry Chen","user":"harryleafchen","type":"user"},{"_id":"630b4269e67c604e9b7a429c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/630b4269e67c604e9b7a429c/qsmA2ObMFfLwPIAyveo9F.jpeg","isPro":true,"fullname":"Steffen Röcker","user":"sroecker","type":"user"},{"_id":"64094eb49e9f790c905a3a59","avatarUrl":"/avatars/6137058af2eaa39131173291af714346.svg","isPro":false,"fullname":"lkr","user":"openhonor","type":"user"},{"_id":"6a5edf6b6478b8cdb6707b68","avatarUrl":"/avatars/e62290f7f149462e567ac12d3c8be0be.svg","isPro":false,"fullname":"Li Chengxia","user":"aigorgs","type":"user"},{"_id":"692fd8d6e9acbeb00d8f15e7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/692fd8d6e9acbeb00d8f15e7/uLAgs8Z9jGjAlbpPLUrq6.jpeg","isPro":false,"fullname":"Mingzhe Zhang","user":"ZMZ21","type":"user"},{"_id":"67edf5f7d7d16dd8d9cfc8d5","avatarUrl":"/avatars/6d74657563ed1c1b80b21c2d9c4d861f.svg","isPro":false,"fullname":"Fan Dang","user":"dangfan","type":"user"},{"_id":"64106205b27543634e392e04","avatarUrl":"/avatars/571ee7747b22774b7cb66107eadaf39a.svg","isPro":false,"fullname":"Jiajie Chen","user":"jiegec","type":"user"},{"_id":"6a9520ef5962d68590b5a297","avatarUrl":"/avatars/423b7d3270cd21f821ad2ed6484499db.svg","isPro":false,"fullname":"Yunyi Chen","user":"lenard-chen","type":"user"},{"_id":"6700016144384e570895609f","avatarUrl":"/avatars/029df89b7e24e1a99b717230e0733bbf.svg","isPro":false,"fullname":"Meow","user":"CircuitCoder","type":"user"},{"_id":"6a2c4a2af89dcb999b8ee921","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a2c4a2af89dcb999b8ee921/u6PZib3OpIgElAIKs4f78.png","isPro":false,"fullname":"Yanmohan Wang","user":"Dct-Strange","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"6407e5294edf9f5c4fd32228","avatarUrl":"/avatars/8e2d55460e9fe9c426eb552baf4b2cb0.svg","isPro":false,"fullname":"Stoney Kang","user":"sikang99","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"69076fd2eb8bfb94ee8b0969","name":"thu-pacman","fullname":"PACMAN Group, Tsinghua University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/66a087eff89441587fdc53d0/2Jw62jk56H9sHnM-Ik4Hz.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.27370.md","query":{}}">

Papers

arxiv:2608.27370

Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090

Published on Aug 27

· Submitted by

Harry Chen on Aug 31

PACMAN Group, Tsinghua University

Upvote

Authors:

Shengqi Chen ,

Abstract

A cost-efficient open-source pretraining recipe trains 2B-parameter models on consumer GPUs for under $7K, yielding performance near larger baselines while deriving cost scaling laws and studying data curricula.

Generated by thinkingmachines/Inkling-Small

Language model pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities. Although strong open-source efforts already exist, including open-weight models and open-source training recipes, a cost-efficient, hardware-accessible, and open-source pretraining recipe has long been missing. Even at a small scale, training Llama-3.2-3B costs over \1.5M, and reproducing SmolLM3-3B needs over 700K. In this report, we present an open pretraining recipe designed to lower this barrier. Using this recipe, we train a collection of Puro-2B models from scratch on up to 1.4 trillion tokens with FP8 precision on consumer-grade RTX 5090 GPUs. The models in the collection differ in token budgets and selected recipe variants. Our best model is trained at a compute cost of less than \6.9K and approaches Qwen2.5-1.5B performance under our evaluation protocol. This cost efficiency is enabled by a combination of approaches, including hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and the data recipe. Beyond the recipe itself, we provide two additional results. First, across the Puro-2B collection, we derive a Puro Cost Scaling Law that relates training cost to average model performance; the fitted law suggests that about 4.4K, less than \$5,090, is sufficient to reach the performance of Qwen2-1.5B. Second, as an end-to-end case study, we examine how pretraining data curricula shape downstream performance after post-training. Such controlled studies are enabled by having access to the full pretraining pipeline rather than model weights alone. We release the full training recipe for Puro-2B, including data, code, and model weights under Apache 2.0 at https://huggingface.co/collections/thu-pacman/puro-2b.