We show that reasoning traces can be rewritten and compressed by models smaller than the teacher before distillation, substantially reducing training and inference cost while retaining most of the accuracy. Compression is not a free lunch: raw traces still perform best, but it offers a practical accuracy–efficiency trade-off for distilling reasoning models.</p>\n","updatedAt":"2026-06-08T11:15:46.977Z","author":{"_id":"64d8cd6f99eb22b8da66dd81","avatarUrl":"/avatars/b418ed1d059119550ca57b150537692e.svg","fullname":"Maxime Griot","name":"maximegmd","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.943023145198822},"editors":["maximegmd"],"editorAvatarUrls":["/avatars/b418ed1d059119550ca57b150537692e.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.05988","authors":[{"_id":"6a26a35ce4c258a02949232a","name":"Maxime Griot","hidden":false},{"_id":"6a26a35ce4c258a02949232b","name":"Paul Steven Scotti","hidden":false},{"_id":"6a26a35ce4c258a02949232c","name":"Tanishq Mathew Abraham","hidden":false}],"publishedAt":"2026-06-04T00:00:00.000Z","submittedOnDailyAt":"2026-06-08T00:00:00.000Z","title":"Compress-Distill: Reasoning Trace Compression for Efficient Knowledge Distillation","submittedOnDailyBy":{"_id":"64d8cd6f99eb22b8da66dd81","avatarUrl":"/avatars/b418ed1d059119550ca57b150537692e.svg","isPro":false,"fullname":"Maxime Griot","user":"maximegmd","type":"user","name":"maximegmd"},"summary":"Reasoning models produce long chain-of-thought traces that are costly to distill and encourage verbose student outputs. We study post-hoc compression of such traces before knowledge distillation. Two teachers, Qwen3.5-397B-A17B and gpt-oss-120B, generate about 283k correct traces each; two instruction-tuned models then compress them to 8.6-21.0% of their original character length. Across a 48-run main grid plus seven Qwen-teacher truncation ablations, compressed traces reduce training tokens to 12-30% of raw, speed up training by 2.0-7.6x, and shorten inference outputs by 3-19x with smaller reductions under the shorter gpt-oss teacher. However, raw traces retain the highest downstream accuracy at every scale and for both teachers. A length-matched raw-trace truncation ablation shows that compression is not merely benefiting from a smaller token budget: model-compressed traces usually beat or match naive truncation, especially for smaller students, while maintaining shorter inference outputs. Overall, reasoning-trace compression offers an accuracy-efficiency trade-off rather than a free improvement: students retain up to 96% of raw-trace accuracy while gaining up to 18x higher per-token efficiency, and at the 0.8B scale under LoRA compressed traces narrow the raw-vs-compressed gap but do not exceed raw.","upvotes":1,"discussionId":"6a26a35ce4c258a02949232d","ai_summary":"Post-hoc compression of reasoning traces reduces computational costs and inference lengths while maintaining high accuracy, offering an accuracy-efficiency trade-off in knowledge distillation.","ai_keywords":["chain-of-thought traces","knowledge distillation","post-hoc compression","instruction-tuned models","LoRA"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","organization":{"_id":"633ed1d3c11d723b1804828b","name":"UCLouvain","fullname":"Université Catholique de Louvain","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1665061312301-633ed1704a7a5d7dfbd1a613.jpeg"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64d8cd6f99eb22b8da66dd81","avatarUrl":"/avatars/b418ed1d059119550ca57b150537692e.svg","isPro":false,"fullname":"Maxime Griot","user":"maximegmd","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"633ed1d3c11d723b1804828b","name":"UCLouvain","fullname":"Université Catholique de Louvain","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1665061312301-633ed1704a7a5d7dfbd1a613.jpeg"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.05988.md"}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2606.05988 in a model README.md to link it from this page.

Cite arxiv.org/abs/2606.05988 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2606.05988 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.