Hugging Face Daily Papers · August 5, 2026 · 3 min read

PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

<a href=\"https://cdn-uploads.huggingface.co/production/uploads/68d35c61a4faedb167e99546/rLWpSVOhq6mv_hFws773j.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68d35c61a4faedb167e99546/rLWpSVOhq6mv_hFws773j.png\" alt=\"pcsd-framework\"></a></p>\n","updatedAt":"2026-08-05T02:56:24.561Z","author":{"_id":"68d35c61a4faedb167e99546","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U2552CQOGaymVamgxcmfM.png","fullname":"chunji lv","name":"yaya234","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.5359033942222595},"editors":["yaya234"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U2552CQOGaymVamgxcmfM.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.01837","authors":[{"_id":"6a7197edec5082b9f872cfec","user":{"_id":"68d35c61a4faedb167e99546","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U2552CQOGaymVamgxcmfM.png","isPro":false,"fullname":"chunji lv","user":"yaya234","type":"user","name":"yaya234"},"name":"Chunji Lv","status":"claimed_verified","statusLastChangedAt":"2026-08-04T08:45:04.780Z","hidden":false},{"_id":"6a7197edec5082b9f872cfed","name":"Yangguang Wei","hidden":false},{"_id":"6a7197edec5082b9f872cfee","name":"Junlin Liu","hidden":false},{"_id":"6a7197edec5082b9f872cfef","name":"Yang Gao","hidden":false},{"_id":"6a7197edec5082b9f872cff0","name":"Ming Liu","hidden":false},{"_id":"6a7197edec5082b9f872cff1","name":"Xinming Wang","hidden":false},{"_id":"6a7197edec5082b9f872cff2","name":"Jinyang Wu","hidden":false},{"_id":"6a7197edec5082b9f872cff3","name":"Guoren Wang","hidden":false},{"_id":"6a7197edec5082b9f872cff4","name":"Changsheng Li","hidden":false}],"publishedAt":"2026-08-03T00:00:00.000Z","submittedOnDailyAt":"2026-08-05T00:00:00.000Z","title":"PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning","submittedOnDailyBy":{"_id":"68d35c61a4faedb167e99546","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U2552CQOGaymVamgxcmfM.png","isPro":false,"fullname":"chunji lv","user":"yaya234","type":"user","name":"yaya234"},"summary":"Large language model agents have shown strong potential in complex interactive tasks, yet their reinforcement learning (RL) is often hindered by sparse rewards, as a long multi-turn trajectory may receive only a single outcome-level signal. On-policy self-distillation (OPSD) provides dense token-level supervision from a privileged teacher, but the teacher may not be reliable at every position. Existing methods commonly rely on isolated token-level discrepancies, which can be sensitive to noise, or assign a shared step-level weight that may overlook positional variation. We propose Persistent Consistency Self-Distillation (PCSD), which derives token-level distillation weights from the local persistence of teacher-favoring signals. PCSD combines adaptive windows with exponentially decayed aggregation to capture persistent relative teacher support, applies trend-aware modulation to attenuate locally declining support, and produces continuous weights through sigmoid gating. The resulting objective is jointly optimized with GRPO, combining dense teacher guidance with sparse environmental feedback. Without inference-time skills, PCSD achieves the best ALFWorld Overall results among all baselines on both backbones, exceeding GRPO by 15.6 and 13.3 points and SDAR by 6.2 and 5.5 points, while remaining competitive on WebShop and gaining 15.8 points over GRPO on unseen ALFWorld split.","upvotes":24,"discussionId":"6a7197edec5082b9f872cff5","projectPage":"https://pcsd.vercel.app/","organization":{"_id":"623df197158b6354baef12e2","name":"BIT","fullname":"Beijing Institute of Technology","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1665508603955-623df1201733752a68c76d38.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"68d35c61a4faedb167e99546","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U2552CQOGaymVamgxcmfM.png","isPro":false,"fullname":"chunji lv","user":"yaya234","type":"user"},{"_id":"65f3ee92c2d214f8848d687c","avatarUrl":"/avatars/10c7eac57273e8dcb4b2a08fb00270f6.svg","isPro":false,"fullname":"Juntong Fang","user":"hellexf","type":"user"},{"_id":"652c92255aec376f5537fbfb","avatarUrl":"/avatars/60ec58c5011b5af1feee972ea812cd4e.svg","isPro":false,"fullname":"WangXinming","user":"Eincasia","type":"user"},{"_id":"6a6a829fa698558ca76157ee","avatarUrl":"/avatars/98e9a42de130397e7f4efce0039cdacd.svg","isPro":false,"fullname":"Richard Williams","user":"richardwilliams","type":"user"},{"_id":"6a6a9d781fcc333c083e0a2f","avatarUrl":"/avatars/31b1c26caa8af0dd423cc1ff6ed9af43.svg","isPro":false,"fullname":"Brian Jones","user":"brian-jones","type":"user"},{"_id":"6a6c7c0e7cd39895c75922a5","avatarUrl":"/avatars/0a127612b74e9afe24ec89549b85d93f.svg","isPro":false,"fullname":"Edward Anderson","user":"orbitcraft","type":"user"},{"_id":"6715b493d54796e4b99d90e8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6715b493d54796e4b99d90e8/X-VqGsRrjPlfb94GRn13x.jpeg","isPro":false,"fullname":"黄炜锴","user":"tsrigo","type":"user"},{"_id":"6a6c8517bd71e954a590d9de","avatarUrl":"/avatars/95120238f43f8486bde2ef6f36e6cc49.svg","isPro":false,"fullname":"Joshua Thompson","user":"graniteP","type":"user"},{"_id":"6a6aa720a698558ca7632e02","avatarUrl":"/avatars/d82c5171d9e05a56cdfd1fa9c097da12.svg","isPro":false,"fullname":"Patricia Miller","user":"walter-6023081","type":"user"},{"_id":"6a6dc8c8d7fe07d7924fc6d7","avatarUrl":"/avatars/a6d040478c4136292fee1105b2010299.svg","isPro":false,"fullname":"John Rodriguez","user":"mary-0279018","type":"user"},{"_id":"6a6de3ec77d1834fc3f1d3ca","avatarUrl":"/avatars/2e8c043efd5c5602cd4b6a448753c9da.svg","isPro":false,"fullname":"Charles Moore","user":"sarah-2913228","type":"user"},{"_id":"6a701c6c518039c9718c3b96","avatarUrl":"/avatars/2e6fa06a429776f305200495892836d6.svg","isPro":false,"fullname":"Daniel Martinez","user":"james-2606086","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"623df197158b6354baef12e2","name":"BIT","fullname":"Beijing Institute of Technology","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1665508603955-623df1201733752a68c76d38.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.01837.md","query":{}}">

Papers

arxiv:2608.01837

PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning

Published on Aug 3

· Submitted by

chunji lv on Aug 5

Beijing Institute of Technology

Upvote

Authors:

Chunji Lv ,

Abstract

Large language model agents have shown strong potential in complex interactive tasks, yet their reinforcement learning (RL) is often hindered by sparse rewards, as a long multi-turn trajectory may receive only a single outcome-level signal. On-policy self-distillation (OPSD) provides dense token-level supervision from a privileged teacher, but the teacher may not be reliable at every position. Existing methods commonly rely on isolated token-level discrepancies, which can be sensitive to noise, or assign a shared step-level weight that may overlook positional variation. We propose Persistent Consistency Self-Distillation (PCSD), which derives token-level distillation weights from the local persistence of teacher-favoring signals. PCSD combines adaptive windows with exponentially decayed aggregation to capture persistent relative teacher support, applies trend-aware modulation to attenuate locally declining support, and produces continuous weights through sigmoid gating. The resulting objective is jointly optimized with GRPO, combining dense teacher guidance with sparse environmental feedback. Without inference-time skills, PCSD achieves the best ALFWorld Overall results among all baselines on both backbones, exceeding GRPO by 15.6 and 13.3 points and SDAR by 6.2 and 5.5 points, while remaining competitive on WebShop and gaining 15.8 points over GRPO on unseen ALFWorld split.

View arXiv page View PDF Project page Add to collection