We therefore propose AgentOPSD, a critic-free recursive turn-level credit assignment for agentic reinforcement learning. AgentOPSD aggregates token-level teacher-student log-probability gaps into turn-level evidence, recursively updates a Bayesian belief state in log-odds space.</p>\n","updatedAt":"2026-08-07T02:14:42.591Z","author":{"_id":"69b7a148b71e93b5b8d4cb51","avatarUrl":"/avatars/544e2fc71b81e85df396e7d03e462138.svg","fullname":"Zi-Han Wang","name":"zzzzhw","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8391207456588745},"editors":["zzzzhw"],"editorAvatarUrls":["/avatars/544e2fc71b81e85df396e7d03e462138.svg"],"reactions":[{"reaction":"👍","users":["LZXzju","zzzzhw","21223wj","SapnapHalo"],"count":4},{"reaction":"🔥","users":["LZXzju","zzzzhw","21223wj"],"count":3},{"reaction":"🚀","users":["LZXzju","zzzzhw","21223wj"],"count":3}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.05987","authors":[{"_id":"6a753d3ae1228e04b323817e","name":"Zi-Han Wang","hidden":false},{"_id":"6a753d3ae1228e04b323817f","user":{"_id":"676127cf11b19ea602bb202a","avatarUrl":"/avatars/dfd802a24bd63e509728159ebb1769f6.svg","isPro":false,"fullname":"Zhengxi Lu","user":"LZXzju","type":"user","name":"LZXzju"},"name":"Zhengxi Lu","status":"claimed_verified","statusLastChangedAt":"2026-08-07T08:45:04.509Z","hidden":false},{"_id":"6a753d3ae1228e04b3238180","name":"Zhiyuan Yao","hidden":false},{"_id":"6a753d3ae1228e04b3238181","user":{"_id":"6747de57f8cab58c22ec94a2","avatarUrl":"/avatars/5bae0341862fac24564781c0fa32aac5.svg","isPro":false,"fullname":"Jinyang Wu","user":"Jinyang23","type":"user","name":"Jinyang23"},"name":"Jinyang Wu","status":"claimed_verified","statusLastChangedAt":"2026-08-07T08:45:04.515Z","hidden":false},{"_id":"6a753d3ae1228e04b3238182","name":"Jie Wu","hidden":false},{"_id":"6a753d3ae1228e04b3238183","name":"Zhengzhou Cai","hidden":false},{"_id":"6a753d3ae1228e04b3238184","name":"Yueqing Sun","hidden":false},{"_id":"6a753d3ae1228e04b3238185","name":"Ziang Ye","hidden":false},{"_id":"6a753d3ae1228e04b3238186","name":"Linji Hao","hidden":false},{"_id":"6a753d3ae1228e04b3238187","name":"Qi Gu","hidden":false},{"_id":"6a753d3ae1228e04b3238188","name":"Xunliang Cai","hidden":false},{"_id":"6a753d3ae1228e04b3238189","name":"Yongliang Shen","hidden":false},{"_id":"6a753d3ae1228e04b323818a","name":"Yujiu Yang","hidden":false}],"publishedAt":"2026-08-06T00:00:00.000Z","submittedOnDailyAt":"2026-08-07T00:00:00.000Z","title":"AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning","submittedOnDailyBy":{"_id":"69b7a148b71e93b5b8d4cb51","avatarUrl":"/avatars/544e2fc71b81e85df396e7d03e462138.svg","isPro":false,"fullname":"Zi-Han Wang","user":"zzzzhw","type":"user","name":"zzzzhw"},"summary":"Reinforcement learning (RL) with verifiable rewards constructs trajectory-level advantage estimates, yet it often fails to credit the few pivotal decisions that determine outcomes in long-horizon, multi-turn agentic tasks. Recent work introduces privileged self-distillation for credit assignment, providing denser supervision, but it remains unclear how such local signals should represent sequential credit. We propose AgentOPSD, a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning. AgentOPSD aggregates token-level teacher-student log-probability gaps into turn-level evidence and recursively updates a Bayesian belief state in log-odds space. This yields a principled reweighting scheme that converts sparse outcome supervision into turn-level credit signals and identifies pivotal turns through the marginal belief revision between consecutive states. The method is fully compatible with standard policy optimization and requires neither an additional critic nor extra rollouts. We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B). AgentOPSD outperforms GRPO and strong self-distillation baselines, achieving 89.1% success on ALFWorld with Qwen2.5-7B. Ablation studies attribute the gains to turn-level aggregation and history-dependent recursive belief updates.","upvotes":62,"discussionId":"6a753d3ae1228e04b323818b","githubRepo":"https://github.com/ZethWang/AgentOPSD","githubRepoAddedBy":"user","githubStars":11,"organization":{"_id":"628735cbc83a2d6ab8d14a66","name":"Tsinghua","fullname":"Tsinghua University","avatar":"https://www.gravatar.com/avatar/6c5c1441e3283e7543342e59277ea219?d=retro&size=100"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"69b7a148b71e93b5b8d4cb51","avatarUrl":"/avatars/544e2fc71b81e85df396e7d03e462138.svg","isPro":false,"fullname":"Zi-Han Wang","user":"zzzzhw","type":"user"},{"_id":"6a6a92243eb4da2061193abb","avatarUrl":"/avatars/e6116c77711f56986bea726ffbdc81ea.svg","isPro":false,"fullname":"Karen Thompson","user":"karen-thompson","type":"user"},{"_id":"6a6a9207ee246d8b226648d7","avatarUrl":"/avatars/549aafb4c48353b827bd5d83abb024ee.svg","isPro":false,"fullname":"Richard Smith","user":"richard-smith","type":"user"},{"_id":"6a6a9506a698558ca76242e3","avatarUrl":"/avatars/0a0eff98d0336961ce4b5c0f0f2db559.svg","isPro":false,"fullname":"John Perez","user":"Ember-Wisp","type":"user"},{"_id":"6a6aa236144c90a0d41741b8","avatarUrl":"/avatars/3f1f7a14fa7466daf8161e65b0ecbeeb.svg","isPro":false,"fullname":"Linda Perez","user":"PixelEvan","type":"user"},{"_id":"6a6c7edf33ba949670870a48","avatarUrl":"/avatars/7b5ac17124afb1b3c27315f7307e858c.svg","isPro":false,"fullname":"Michael Jones","user":"SableLens","type":"user"},{"_id":"6a6c81f3ffc872511f2f5233","avatarUrl":"/avatars/8b8f19bb50bd785df358cb9173c345d3.svg","isPro":false,"fullname":"Barbara Jackson","user":"lunartrail","type":"user"},{"_id":"6a6c849d7827edbb08463d78","avatarUrl":"/avatars/fa58f398553193ff34c11959b184129a.svg","isPro":false,"fullname":"Jessica Anderson","user":"Jessica-Anderson","type":"user"},{"_id":"6a6c8859ac4f2e43d4f6a580","avatarUrl":"/avatars/4e3e3470f655f2825935bb371bb1e321.svg","isPro":false,"fullname":"Kenneth Gonzalez","user":"Kenneth-Gonzalez","type":"user"},{"_id":"6a6d58010067cd4ef9922c73","avatarUrl":"/avatars/3f6b770e7bf155a974580cfc9a0bce48.svg","isPro":false,"fullname":"Brian Smith","user":"brian-smith","type":"user"},{"_id":"6a6aa7be2af00ac1140c6958","avatarUrl":"/avatars/a3911dd8f180b53b659d67ce714d6fb7.svg","isPro":false,"fullname":"Matthew Garcia","user":"nimbusRuneE","type":"user"},{"_id":"6a6da89b0f8a360db5286730","avatarUrl":"/avatars/4824d2edbfde383b848c0d6bf594f61b.svg","isPro":false,"fullname":"Patricia Davis","user":"Cobalt-Ian","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":1,"organization":{"_id":"628735cbc83a2d6ab8d14a66","name":"Tsinghua","fullname":"Tsinghua University","avatar":"https://www.gravatar.com/avatar/6c5c1441e3283e7543342e59277ea219?d=retro&size=100"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.05987.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2608.05987 in a model README.md to link it from this page.

Cite arxiv.org/abs/2608.05987 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2608.05987 in a Space README.md to link it from this page.