Daedalus-150M is a 160M-parameter language model designed backwards from its deployment target: one user, one token at a time, 4-bit weights, an ordinary CPU. Only 6 of its 18 blocks use full attention; the other 12 use short depthwise convolutions whose recurrent state is two timesteps wide no matter how long the context gets, so two thirds of the network never re-reads a growing KV cache.</p>\n<p>Trained from scratch on 59.9B tokens, it scores <strong>47.31</strong> on a five-task benchmark (HellaSwag, ARC-Easy, PIQA, OpenBookQA, WinoGrande) against a bar of 42.20 that was fixed before training began. It beats GPT-2 124M, Pythia-160M, OPT-125M and GPT-Neo-125M — all trained on 3–6× more data — and exceeds MobileLLM-125M's published score despite that model seeing a trillion tokens. Validation bits-per-byte is 0.8685.</p>\n<p>To test the architecture rather than the training recipe, a parameter-matched all-attention twin was trained on identical data, with the winning condition written down before either arm was scored. The hybrid won the pre-registered quality metric by 0.81%, matched the twin on downstream tasks, produced a 6.3% smaller 4-bit file, and decoded <strong>1.76× faster at 2048 tokens of context</strong> — 2.08× against an external 135M peer. In every measurement the speed advantage is near zero at empty context and grows with depth, which is what the mechanism predicts and what a merely leaner model would not show. A bandwidth-only cost model predicts just 1.17×, so memory volume alone does not explain the gap.</p>\n<p>The paper also reports what did not work: an unmitigated 4-bit quality cost after quantisation-aware training failed on its first step, roughly half the convolution channels ending up inert and impossible to prune under the stock runtime's shape checks, and a vocabulary larger than this model size warrants.</p>\n<p>Model: <a href=\"https://huggingface.co/Unseen1980/daedalus-150m-instruct\">https://huggingface.co/Unseen1980/daedalus-150m-instruct</a><br>Checkpoints: <a href=\"https://huggingface.co/Unseen1980/daedalus-checkpoints\">https://huggingface.co/Unseen1980/daedalus-checkpoints</a><br>Code: <a href=\"https://github.com/unseen1980/daedalus\" rel=\"nofollow\">https://github.com/unseen1980/daedalus</a></p>\n","updatedAt":"2026-08-24T06:47:50.314Z","author":{"_id":"6582e7f1326a9c04c9004cf4","avatarUrl":"/avatars/103d64489d3c88e7fa2fef936d97608a.svg","fullname":"Ck","name":"Unseen1980","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9205157160758972},"editors":["Unseen1980"],"editorAvatarUrls":["/avatars/103d64489d3c88e7fa2fef936d97608a.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.20210","authors":[{"_id":"6a89e9b23d26296ea3091567","name":"Christos Koutsiaris","hidden":false}],"publishedAt":"2026-08-20T00:00:00.000Z","submittedOnDailyAt":"2026-08-24T00:00:00.000Z","title":"Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference","submittedOnDailyBy":{"_id":"6582e7f1326a9c04c9004cf4","avatarUrl":"/avatars/103d64489d3c88e7fa2fef936d97608a.svg","isPro":false,"fullname":"Ck","user":"Unseen1980","type":"user","name":"Unseen1980"},"summary":"Small language models are usually built like large ones and then squeezed onto a CPU afterwards. We did the opposite: we fixed the target first, one user, one token at a time, 4-bit weights, ordinary CPU, and chose the architecture to suit it. The result keeps full attention in only 6 of its 18 blocks. The other 12 use short convolutions whose memory is two timesteps wide no matter how long the conversation gets, so two thirds of the network never re-reads a growing cache.\n Trained from scratch on 59.9B tokens, the model scores 47.31 on a five-task benchmark against a bar of 42.20 that was fixed before training began. It beats GPT-2 124M, Pythia-160M, OPT-125M and GPT-neo-125M, all trained on three to six times more data, and exceeds MobileLLM-125M's published score despite that model seeing a trillion tokens. Validation bits-per-byte is 0.8685.\n To check the architecture rather than the training recipe, we trained a conventional all-attention model of the same size on the same data, and wrote down the winning condition before scoring either. The hybrid won the chosen quality metric by 0.81%, matched it on downstream tasks, produced a 6.3% smaller 4-bit file, and decoded 1.76x faster at 2048 tokens of context, 2.08x against an external model of similar size. In every measurement the speed advantage is near zero at an empty context and grows with length, which is what the mechanism predicts and what a merely leaner model would not show. A simple bandwidth calculation predicts only 1.17x, so memory volume alone does not explain the gap.\n We also report what did not work: an unmitigated 4-bit quality cost, roughly half the convolution channels ending up inert and impossible to remove, and a vocabulary larger than this model size warrants.","upvotes":2,"discussionId":"6a89e9b23d26296ea3091568","projectPage":"https://huggingface.co/Unseen1980/daedalus-150m-instruct","githubRepo":"https://github.com/unseen1980/daedalus","githubRepoAddedBy":"user","ai_summary":"A small hybrid language model using sparse attention and short convolutions achieves faster CPU inference and better benchmark scores than larger conventional models despite training on far less data.","ai_keywords":["attention blocks","short convolutions","4-bit weights","hybrid architecture","bits-per-byte","context length","bandwidth"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":0},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6582e7f1326a9c04c9004cf4","avatarUrl":"/avatars/103d64489d3c88e7fa2fef936d97608a.svg","isPro":false,"fullname":"Ck","user":"Unseen1980","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.20210.md","query":{}}">
Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference
Published on Aug 20
· Submitted by Ck on Aug 24 Abstract
A small hybrid language model using sparse attention and short convolutions achieves faster CPU inference and better benchmark scores than larger conventional models despite training on far less data.
Small language models are usually built like large ones and then squeezed onto a CPU afterwards. We did the opposite: we fixed the target first, one user, one token at a time, 4-bit weights, ordinary CPU, and chose the architecture to suit it. The result keeps full attention in only 6 of its 18 blocks. The other 12 use short convolutions whose memory is two timesteps wide no matter how long the conversation gets, so two thirds of the network never re-reads a growing cache.
Trained from scratch on 59.9B tokens, the model scores 47.31 on a five-task benchmark against a bar of 42.20 that was fixed before training began. It beats GPT-2 124M, Pythia-160M, OPT-125M and GPT-neo-125M, all trained on three to six times more data, and exceeds MobileLLM-125M's published score despite that model seeing a trillion tokens. Validation bits-per-byte is 0.8685.
To check the architecture rather than the training recipe, we trained a conventional all-attention model of the same size on the same data, and wrote down the winning condition before scoring either. The hybrid won the chosen quality metric by 0.81%, matched it on downstream tasks, produced a 6.3% smaller 4-bit file, and decoded 1.76x faster at 2048 tokens of context, 2.08x against an external model of similar size. In every measurement the speed advantage is near zero at an empty context and grows with length, which is what the mechanism predicts and what a merely leaner model would not show. A simple bandwidth calculation predicts only 1.17x, so memory volume alone does not explain the gap.
We also report what did not work: an unmitigated 4-bit quality cost, roughly half the convolution channels ending up inert and impossible to remove, and a vocabulary larger than this model size warrants.
Community
Daedalus-150M is a 160M-parameter language model designed backwards from its deployment target: one user, one token at a time, 4-bit weights, an ordinary CPU. Only 6 of its 18 blocks use full attention; the other 12 use short depthwise convolutions whose recurrent state is two timesteps wide no matter how long the context gets, so two thirds of the network never re-reads a growing KV cache.
Trained from scratch on 59.9B tokens, it scores 47.31 on a five-task benchmark (HellaSwag, ARC-Easy, PIQA, OpenBookQA, WinoGrande) against a bar of 42.20 that was fixed before training began. It beats GPT-2 124M, Pythia-160M, OPT-125M and GPT-Neo-125M — all trained on 3–6× more data — and exceeds MobileLLM-125M's published score despite that model seeing a trillion tokens. Validation bits-per-byte is 0.8685.
To test the architecture rather than the training recipe, a parameter-matched all-attention twin was trained on identical data, with the winning condition written down before either arm was scored. The hybrid won the pre-registered quality metric by 0.81%, matched the twin on downstream tasks, produced a 6.3% smaller 4-bit file, and decoded 1.76× faster at 2048 tokens of context — 2.08× against an external 135M peer. In every measurement the speed advantage is near zero at empty context and grows with depth, which is what the mechanism predicts and what a merely leaner model would not show. A bandwidth-only cost model predicts just 1.17×, so memory volume alone does not explain the gap.
The paper also reports what did not work: an unmitigated 4-bit quality cost after quantisation-aware training failed on its first step, roughly half the convolution channels ending up inert and impossible to prune under the stock runtime's shape checks, and a vocabulary larger than this model size warrants.
Model: https://huggingface.co/Unseen1980/daedalus-150m-instruct
Checkpoints: https://huggingface.co/Unseen1980/daedalus-checkpoints
Code: https://github.com/unseen1980/daedalus
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.20210 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.20210 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.20210 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.