More information? Just ask</p>\n","updatedAt":"2026-07-28T03:27:17.429Z","author":{"_id":"67fcc5ca7f552d4eb0162533","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67fcc5ca7f552d4eb0162533/sqFNvSpyuvKmzgUaxayfd.jpeg","fullname":"sietse schelpe","name":"Corbenic","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7066448330879211},"editors":["Corbenic"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/67fcc5ca7f552d4eb0162533/sqFNvSpyuvKmzgUaxayfd.jpeg"],"reactions":[],"isReport":false}},{"id":"6a684d2d891e0b825c5c3415","author":{"_id":"668cc7e615511820b973007b","avatarUrl":"/avatars/5cd49fc5404a37ba806701a6929864c6.svg","fullname":"David Seyser","name":"DavidSeyserHF","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":5,"isUserFollowing":false},"createdAt":"2026-07-28T06:33:17.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"The core engineering idea here seems reasonable. Store verified, parameterized procedures and reuse them instead of asking an LLM to solve the same task repeatedly.\n\nBut the headline result feels seriously overstated. The reported 100% does not appear to show that a frozen 12B model “beats” frontier models. It appears to show that, once the correct procedure is already solved, verified, stored, retrieved, and executed on new parameters from the same task family, the system produces correct outputs.\n\nThat is much closer to a verified function registry, executable RAG, or tool library than to an improvement in the reasoning capability of the underlying model.","html":"<p>The core engineering idea here seems reasonable. Store verified, parameterized procedures and reuse them instead of asking an LLM to solve the same task repeatedly.</p>\n<p>But the headline result feels seriously overstated. The reported 100% does not appear to show that a frozen 12B model “beats” frontier models. It appears to show that, once the correct procedure is already solved, verified, stored, retrieved, and executed on new parameters from the same task family, the system produces correct outputs.</p>\n<p>That is much closer to a verified function registry, executable RAG, or tool library than to an improvement in the reasoning capability of the underlying model.</p>\n","updatedAt":"2026-07-28T06:33:17.776Z","author":{"_id":"668cc7e615511820b973007b","avatarUrl":"/avatars/5cd49fc5404a37ba806701a6929864c6.svg","fullname":"David Seyser","name":"DavidSeyserHF","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":5,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9246946573257446},"editors":["DavidSeyserHF"],"editorAvatarUrls":["/avatars/5cd49fc5404a37ba806701a6929864c6.svg"],"reactions":[{"reaction":"👍","users":["KRLLRZz","Corbenic"],"count":2}],"isReport":false}},{"id":"6a68a393d7096abd8d855f50","author":{"_id":"67fcc5ca7f552d4eb0162533","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67fcc5ca7f552d4eb0162533/sqFNvSpyuvKmzgUaxayfd.jpeg","fullname":"sietse schelpe","name":"Corbenic","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false},"createdAt":"2026-07-28T12:41:55.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"https://github.com/corbenicai/galahad-bench/tree/main/proof","html":"<p><a href=\"https://github.com/corbenicai/galahad-bench/tree/main/proof\" rel=\"nofollow\">https://github.com/corbenicai/galahad-bench/tree/main/proof</a></p>\n","updatedAt":"2026-07-28T12:41:55.591Z","author":{"_id":"67fcc5ca7f552d4eb0162533","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67fcc5ca7f552d4eb0162533/sqFNvSpyuvKmzgUaxayfd.jpeg","fullname":"sietse schelpe","name":"Corbenic","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.40557050704956055},"editors":["Corbenic"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/67fcc5ca7f552d4eb0162533/sqFNvSpyuvKmzgUaxayfd.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.23806","authors":[{"_id":"6a6820b373f69d5af2bec7eb","user":{"_id":"67fcc5ca7f552d4eb0162533","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67fcc5ca7f552d4eb0162533/sqFNvSpyuvKmzgUaxayfd.jpeg","isPro":true,"fullname":"sietse schelpe","user":"Corbenic","type":"user","name":"Corbenic"},"name":"Sietse Schelpe","status":"claimed_verified","statusLastChangedAt":"2026-07-28T16:45:04.654Z","hidden":false}],"publishedAt":"2026-07-26T00:00:00.000Z","submittedOnDailyAt":"2026-07-28T00:00:00.000Z","title":"A Frozen 12B Beats Frontier Models on Verified Work: 100% Accuracy, 0 Tokens, Bit-Exact, Forever","submittedOnDailyBy":{"_id":"67fcc5ca7f552d4eb0162533","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67fcc5ca7f552d4eb0162533/sqFNvSpyuvKmzgUaxayfd.jpeg","isPro":true,"fullname":"sietse schelpe","user":"Corbenic","type":"user","name":"Corbenic"},"summary":"Improving a language model today means retraining it: enormous compute, a new opaque model each cycle, non-deterministic output. We take the opposite path: the model stays frozen, and a persistent memory of verified solutions grows beside it. Once a problem family is solved and has passed an independent verification step that never consults the answer key, every new instance of that family is answered at zero generation tokens, bit-exact, deterministically. Across 180 fresh instances spanning nine problem families, four architectures from four vendors - dense and mixture-of-experts - each score 180/180 at zero generation tokens per answer: execution-bound capability decoupled from parameter scaling. A negative control attributes the capability fully to the memory: emptied, it solves nothing. The same verify-before-store contract holds for open-ended reasoning: 88/88 consistency-gated acceptances across all four models, machine-checked formal proof, and reasoning-method transfer at 77/80. Memory selection takes 1.4 microseconds; a full reuse completes in 6-23 ms at 36 mWh. Approximate similarity retrieval selects the wrong item 94.3% of the time on a 4,500-item verified store where exact addressing makes zero errors. The store also serves as working context at a scale no shipped engine matches: a 6,000,000-token movable window on a single 46 GB GPU at flat memory, where vLLM stops at 30,399 tokens and SGLang silently truncates past 32,000. On published benchmarks, frontier models remain far ahead of any 12B at raw from-scratch reasoning; on everything this system has solved and verified, the comparison inverts: a frontier API call pays a fresh generation pass on every query, forever, while verified reuse costs zero tokens and returns the identical bits every time. A public testbench with free, rate-limited access accompanies this report: https://corbenic-galahad-bench.hf.space","upvotes":3,"discussionId":"6a6820b373f69d5af2bec7ec","projectPage":"https://arxiv.org/abs/2607.23806","githubRepo":"https://github.com/corbenicai/galahad-bench","githubRepoAddedBy":"user","githubStars":1,"organization":{"_id":"6a1d1018e87b112ed3ab159d","name":"Corbenci","fullname":"Corbenic","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/67fcc5ca7f552d4eb0162533/JMN8IDsOLPOKK8t1Q6Rlz.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"67fcc5ca7f552d4eb0162533","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67fcc5ca7f552d4eb0162533/sqFNvSpyuvKmzgUaxayfd.jpeg","isPro":true,"fullname":"sietse schelpe","user":"Corbenic","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"697c8b15a7f796854ef333c4","avatarUrl":"/avatars/94de3a736fac914944f1b57609e3819a.svg","isPro":false,"fullname":"Joel Wang","user":"joelhenwang","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6a1d1018e87b112ed3ab159d","name":"Corbenci","fullname":"Corbenic","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/67fcc5ca7f552d4eb0162533/JMN8IDsOLPOKK8t1Q6Rlz.png"},"query":{}}">
A Frozen 12B Beats Frontier Models on Verified Work: 100% Accuracy, 0 Tokens, Bit-Exact, Forever
Abstract
Improving a language model today means retraining it: enormous compute, a new opaque model each cycle, non-deterministic output. We take the opposite path: the model stays frozen, and a persistent memory of verified solutions grows beside it. Once a problem family is solved and has passed an independent verification step that never consults the answer key, every new instance of that family is answered at zero generation tokens, bit-exact, deterministically. Across 180 fresh instances spanning nine problem families, four architectures from four vendors - dense and mixture-of-experts - each score 180/180 at zero generation tokens per answer: execution-bound capability decoupled from parameter scaling. A negative control attributes the capability fully to the memory: emptied, it solves nothing. The same verify-before-store contract holds for open-ended reasoning: 88/88 consistency-gated acceptances across all four models, machine-checked formal proof, and reasoning-method transfer at 77/80. Memory selection takes 1.4 microseconds; a full reuse completes in 6-23 ms at 36 mWh. Approximate similarity retrieval selects the wrong item 94.3% of the time on a 4,500-item verified store where exact addressing makes zero errors. The store also serves as working context at a scale no shipped engine matches: a 6,000,000-token movable window on a single 46 GB GPU at flat memory, where vLLM stops at 30,399 tokens and SGLang silently truncates past 32,000. On published benchmarks, frontier models remain far ahead of any 12B at raw from-scratch reasoning; on everything this system has solved and verified, the comparison inverts: a frontier API call pays a fresh generation pass on every query, forever, while verified reuse costs zero tokens and returns the identical bits every time. A public testbench with free, rate-limited access accompanies this report: https://corbenic-galahad-bench.hf.space
Community
More information? Just ask
The core engineering idea here seems reasonable. Store verified, parameterized procedures and reuse them instead of asking an LLM to solve the same task repeatedly.
But the headline result feels seriously overstated. The reported 100% does not appear to show that a frozen 12B model “beats” frontier models. It appears to show that, once the correct procedure is already solved, verified, stored, retrieved, and executed on new parameters from the same task family, the system produces correct outputs.
That is much closer to a verified function registry, executable RAG, or tool library than to an improvement in the reasoning capability of the underlying model.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2607.23806 in a model README.md to link it from this page.
Cite arxiv.org/abs/2607.23806 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.