For more information, please check out our:<br>📝 Blog: <a href=\"https://microsoft.github.io/debug-gym/blog/2026/09/programdistill/\" rel=\"nofollow\">https://microsoft.github.io/debug-gym/blog/2026/09/programdistill/</a><br>🔗 Paper: <a href=\"https://microsoft.github.io/debug-gym/static/papers/ProgramDistill_arxiv.pdf\" rel=\"nofollow\">https://microsoft.github.io/debug-gym/static/papers/ProgramDistill_arxiv.pdf</a></p>\n<p>We’re working on a public release of the ProgramDistill benchmark, so you'll be able to play with it yourself soon.</p>\n","updatedAt":"2026-09-17T01:46:40.563Z","author":{"_id":"63e48f6d9db5da2dc1f6288e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1676046878664-63e48f6d9db5da2dc1f6288e.png","fullname":"JeonghyeKim","name":"beanie00","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.46441349387168884},"editors":["beanie00"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1676046878664-63e48f6d9db5da2dc1f6288e.png"],"reactions":[{"reaction":"🔥","users":["eryua","leee99"],"count":2}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2609.18805","authors":[{"_id":"6aab3fdb1d9cc4dec79623ac","name":"Jeonghye Kim","hidden":false},{"_id":"6aab3fdb1d9cc4dec79623ad","name":"Minseon Kim","hidden":false},{"_id":"6aab3fdb1d9cc4dec79623ae","name":"Young Jin Kim","hidden":false},{"_id":"6aab3fdb1d9cc4dec79623af","name":"Matheus Pereira","hidden":false},{"_id":"6aab3fdb1d9cc4dec79623b0","name":"Marc-Alexandre Côté","hidden":false},{"_id":"6aab3fdb1d9cc4dec79623b1","name":"Alessandro Sordoni","hidden":false},{"_id":"6aab3fdb1d9cc4dec79623b2","name":"Xingdi Yuan","hidden":false},{"_id":"6aab3fdb1d9cc4dec79623b3","name":"Zhengyan Shi","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/63e48f6d9db5da2dc1f6288e/zREUngFoSrCcZv-h7OcAf.mp4"],"publishedAt":"2026-09-16T00:00:00.000Z","submittedOnDailyAt":"2026-09-17T00:00:00.000Z","title":"ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks","submittedOnDailyBy":{"_id":"63e48f6d9db5da2dc1f6288e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1676046878664-63e48f6d9db5da2dc1f6288e.png","isPro":false,"fullname":"JeonghyeKim","user":"beanie00","type":"user","name":"beanie00"},"summary":"Coding agents are typically evaluated with desired behavior specified through issues or instructions. In practical web development, however, agents may need to infer behavior from working software and implement it in an incomplete application. We introduce ProgramDistill, a benchmark evaluating coding agents on features discovered through interaction with fully functional reference applications. We build ProgramDistill by factorizing applications into features of different granularities, each associated with replayable behaviors executable via its gold patch. Our pipeline, mine-craft-patch, discovers 1,975 replay-verified behaviors across 26 applications and constructs 4,063 tasks without human intervention. Across nine frontier coding agents, GPT-6 Astra and Claude Opus 5 achieve 49.2% and 28.8% success on cumulative workflows in full-application reconstruction. In partial-application reconstruction, success falls from 100% to 64.0% and from 96% to 32% as restoration depth increases from 1 to 8. ProgramDistill thus provides a scalable benchmark with controlled difficulty for evaluating and diagnosing coding agents, and a natural basis for future curriculum-based training.","upvotes":43,"discussionId":"6aab3fdb1d9cc4dec79623b4","projectPage":"https://microsoft.github.io/debug-gym/blog/2026/09/programdistill/","organization":{"_id":"68151d0f51add3813f3f7d1b","name":"MicrosoftResearch","fullname":"Microsoft Research","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6529a4f2f1205983224fa513/PeuVr7jSuJflmDBBGxoDX.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"63e48f6d9db5da2dc1f6288e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1676046878664-63e48f6d9db5da2dc1f6288e.png","isPro":false,"fullname":"JeonghyeKim","user":"beanie00","type":"user"},{"_id":"68346316bce23898b21d1de0","avatarUrl":"/avatars/c69e35b75083fb8e767b56db5d0ee939.svg","isPro":false,"fullname":"Dong Hyeok Kim","user":"styxsecutiry","type":"user"},{"_id":"6495947b3eaaf416d924daeb","avatarUrl":"/avatars/2e79b75e0f5f54f11471affc2bb377b6.svg","isPro":false,"fullname":"Eric Yuan","user":"eryua","type":"user"},{"_id":"64aba0744135aae75f51bf8e","avatarUrl":"/avatars/87ba3b0551756261795ca8ef029d81fd.svg","isPro":false,"fullname":"Do Hyung Kim","user":"kimdohyung","type":"user"},{"_id":"65b9fa2ac29f995b6df1c1b2","avatarUrl":"/avatars/a1077af79f41da50925f1c3931a78bd4.svg","isPro":false,"fullname":"Lee SangMook","user":"sangmook12","type":"user"},{"_id":"668e4af81173ab43d9c95202","avatarUrl":"/avatars/7f0dcaa7b8523aa46afa0cb717af3e9c.svg","isPro":false,"fullname":"DongGyu Kim","user":"8dgkim","type":"user"},{"_id":"6440a11d757aa3c2ad87c8db","avatarUrl":"/avatars/7f4cafddc76e3720ed161dc7b5d0bb65.svg","isPro":false,"fullname":"minbeomkim","user":"mbkim","type":"user"},{"_id":"6aab4aee1f88dc502db04641","avatarUrl":"/avatars/f758165b5fd035a134c2857565669f05.svg","isPro":false,"fullname":"Kim, JaeHwan","user":"Kim3882","type":"user"},{"_id":"667a955526fccf94e22fc3d2","avatarUrl":"/avatars/8aa9183f16d40e02d3cdf78ec0e8d211.svg","isPro":false,"fullname":"minhae","user":"minhaeoh","type":"user"},{"_id":"67d9202f5958eee282cd3da2","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/TOWRglfA0OqtrPDgUBU_6.png","isPro":true,"fullname":"Seungpil Lee","user":"iamseungpil","type":"user"},{"_id":"64eeda0c06bfd13e03896055","avatarUrl":"/avatars/b0617fa5d305588448a72e16f4d59087.svg","isPro":false,"fullname":"박종의","user":"bitsandscraps","type":"user"},{"_id":"64cfa0b9749587dbe01d0079","avatarUrl":"/avatars/93ca0a1d9c5578d052c5af0d4d1a0252.svg","isPro":false,"fullname":"Yumin Choi","user":"YuminChoi","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"68151d0f51add3813f3f7d1b","name":"MicrosoftResearch","fullname":"Microsoft Research","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6529a4f2f1205983224fa513/PeuVr7jSuJflmDBBGxoDX.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2609/2609.18805.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2609.18805 in a model README.md to link it from this page.

Cite arxiv.org/abs/2609.18805 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2609.18805 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.