Hugging Face Daily Papers · June 10, 2026 · 6 min read

Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

Project Page: <a href=\"https://chongqichuizi875.github.io/CPPO-Project-Page/\" rel=\"nofollow\">https://chongqichuizi875.github.io/CPPO-Project-Page/</a>\n","updatedAt":"2026-06-10T09:46:47.653Z","author":{"_id":"663f6e938732bcd0766f17df","avatarUrl":"/avatars/c0eaf3a5cecf0a7129e8bf2aa76cbb22.svg","fullname":"sumail","name":"sumailmao","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.4736770689487457},"editors":["sumailmao"],"editorAvatarUrls":["/avatars/c0eaf3a5cecf0a7129e8bf2aa76cbb22.svg"],"reactions":[],"isReport":false}},{"id":"6a29f7b76a0c01e11c952e9e","author":{"_id":"66129c7b50350afe76757262","avatarUrl":"/avatars/a2f4fac076b9d658a0d904ed54960f6f.svg","fullname":"Xiangxin Zhou","name":"zhouxiangxin","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":5,"isUserFollowing":false},"createdAt":"2026-06-10T23:48:07.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"![20260611-074726](https://cdn-uploads.huggingface.co/production/uploads/66129c7b50350afe76757262/-4DutF1vZQmI0jpkSd3Gp.png)","html":"<a href=\"https://cdn-uploads.huggingface.co/production/uploads/66129c7b50350afe76757262/-4DutF1vZQmI0jpkSd3Gp.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/66129c7b50350afe76757262/-4DutF1vZQmI0jpkSd3Gp.png\" alt=\"20260611-074726\"></a>\n","updatedAt":"2026-06-10T23:48:07.492Z","author":{"_id":"66129c7b50350afe76757262","avatarUrl":"/avatars/a2f4fac076b9d658a0d904ed54960f6f.svg","fullname":"Xiangxin Zhou","name":"zhouxiangxin","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":5,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.48373186588287354},"editors":["zhouxiangxin"],"editorAvatarUrls":["/avatars/a2f4fac076b9d658a0d904ed54960f6f.svg"],"reactions":[],"isReport":false}},{"id":"6a2a15f5093961743494e40a","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":363,"isUserFollowing":false},"createdAt":"2026-06-11T01:57:09.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective](https://huggingface.co/papers/2605.07331) (2026)\n* [Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization](https://huggingface.co/papers/2604.13197) (2026)\n* [Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards](https://huggingface.co/papers/2605.20865) (2026)\n* [EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance](https://huggingface.co/papers/2605.04960) (2026)\n* [Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing](https://huggingface.co/papers/2605.14978) (2026)\n* [Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVR](https://huggingface.co/papers/2604.11056) (2026)\n* [OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning](https://huggingface.co/papers/2605.21851) (2026)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"This is an automated message from the <a href=\"https://huggingface.co/librarian-bots\">Librarian Bot</a>. I found the following papers similar to this paper. \nThe following papers were recommended by the Semantic Scholar API \n<ul>\n<li><a href=\"https://huggingface.co/papers/2605.07331\">Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2604.13197\">Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.20865\">Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.04960\">EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.14978\">Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2604.11056\">Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVR</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.21851\">OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning</a> (2026)</li>\n</ul>\n Please give a thumbs up to this comment if you found it helpful!\n If you want recommendations for any Paper on Hugging Face checkout <a href=\"https://huggingface.co/spaces/librarian-bots/recommend_similar_papers\">this</a> Space\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: <code><a href=\"/librarian-bot\">@librarian-bot</a> recommend</code>\n","updatedAt":"2026-06-11T01:57:09.286Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":363,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7437193989753723},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.10968","authors":[{"_id":"6a28f367e7d78ea7587e55ab","name":"Renjie Mao","hidden":false},{"_id":"6a28f367e7d78ea7587e55ac","name":"Xiangxin Zhou","hidden":false},{"_id":"6a28f367e7d78ea7587e55ad","name":"Lvfang Tao","hidden":false},{"_id":"6a28f367e7d78ea7587e55ae","name":"Yixin Ding","hidden":false},{"_id":"6a28f367e7d78ea7587e55af","name":"Yu Shi","hidden":false},{"_id":"6a28f367e7d78ea7587e55b0","name":"Yongguang Lin","hidden":false},{"_id":"6a28f367e7d78ea7587e55b1","name":"Yuheng Wu","hidden":false},{"_id":"6a28f367e7d78ea7587e55b2","name":"Honglin Zhu","hidden":false},{"_id":"6a28f367e7d78ea7587e55b3","name":"Qian Qiu","hidden":false},{"_id":"6a28f367e7d78ea7587e55b4","name":"Wenxi Zhu","hidden":false}],"publishedAt":"2026-06-09T00:00:00.000Z","submittedOnDailyAt":"2026-06-10T00:00:00.000Z","title":"Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning","submittedOnDailyBy":{"_id":"66129c7b50350afe76757262","avatarUrl":"/avatars/a2f4fac076b9d658a0d904ed54960f6f.svg","isPro":false,"fullname":"Xiangxin Zhou","user":"zhouxiangxin","type":"user","name":"zhouxiangxin"},"summary":"Reinforcement learning with verifiable rewards (RLVR) has become standard for improving LLM reasoning. However, existing PPO-style trust-region mechanisms remain position-agnostic by enforcing uniform thresholds across all tokens independently. This pointwise treatment conflicts with autoregressive generation in two critical ways. First, uniform thresholds ignore autoregressive asymmetry. Early-stage deviations produce compounding sequence-level drift, causing static thresholds to under-regulate early divergence and excessively constrain late-stage exploration. Second, evaluating token-level divergence in isolation overlooks cumulative prefix drift, granting the same divergence allowance regardless of how far the conditioning history has already deviated from the rollout policy. To address this limitation, we propose CPPO (Cumulative Prefix-divergence Policy Optimization), a token-level masking rule that aligns updates with a finite-horizon policy-improvement bound via two coupled mechanisms. First, a position-weighted threshold imposes stricter limits at early positions whose effects persist longer, relaxing constraints for late-stage tokens. Second, a cumulative prefix budget tracks historical deviations, dynamically restricting further token-level deviation to prevent compounding errors along the prefix. Empirically, CPPO enhances training stability and significantly improves reasoning accuracy across various model scales.","upvotes":41,"discussionId":"6a28f367e7d78ea7587e55b5","ai_summary":"CPPO addresses limitations in reinforcement learning with verifiable rewards by introducing position-weighted thresholds and cumulative prefix budgeting to better handle autoregressive generation challenges.","ai_keywords":["PPO-style trust-region mechanisms","autoregressive generation","token-level masking","policy-improvement bound","position-weighted threshold","cumulative prefix budget"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","organization":{"_id":"6a24e31c749f04abbbb5105d","name":"Tencent-Hunyuan-Multimodal-RL","fullname":"Tencent-Hunyuan-Multimodal-RL","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/66129c7b50350afe76757262/AMk8iMwAaEjA0M5Cdq0NE.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64b76c8453d91a364aae131f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64b76c8453d91a364aae131f/4fHfPJ8QT8zVssBAnDPQ1.png","isPro":false,"fullname":"Lvfang Tao","user":"MeowFET","type":"user"},{"_id":"653a274836c4c769cd67aac1","avatarUrl":"/avatars/81b07bee46da203196afafd92df2621b.svg","isPro":false,"fullname":"wenxizhu","user":"wenxizhu","type":"user"},{"_id":"65fd8dfe706c1d587b6c4745","avatarUrl":"/avatars/fad02de1116180923c7bc2d2af0e008d.svg","isPro":false,"fullname":"yixiang dai","user":"yixiang666","type":"user"},{"_id":"6856995d756b6b4241dbd041","avatarUrl":"/avatars/ffd668b562c507c6a7382955d6d6758a.svg","isPro":false,"fullname":"QQ","user":"BruceQQ0","type":"user"},{"_id":"66129c7b50350afe76757262","avatarUrl":"/avatars/a2f4fac076b9d658a0d904ed54960f6f.svg","isPro":false,"fullname":"Xiangxin Zhou","user":"zhouxiangxin","type":"user"},{"_id":"69b1289eeb8c5f1fc8e2411f","avatarUrl":"/avatars/542050f7e1ae2bcb65536ad8903ecfe9.svg","isPro":false,"fullname":"yanghl","user":"selphe","type":"user"},{"_id":"647587a6e0b188d3cb265916","avatarUrl":"/avatars/0b0166a6d9e29e32f97721aaac8269fc.svg","isPro":false,"fullname":"zwh","user":"zzzwh","type":"user"},{"_id":"6a28fd283bd971d12cd94d92","avatarUrl":"/avatars/2cb6df0778616584cc4ab8e845bb5ade.svg","isPro":false,"fullname":"cyzheng","user":"pty123","type":"user"},{"_id":"636e4b2b8ba65db4a0957f69","avatarUrl":"/avatars/40e14c73a9cd3e851bc7738b5c4a8cac.svg","isPro":false,"fullname":"Zhao Haohan","user":"zhhxte","type":"user"},{"_id":"6a28ff325160513fef81af26","avatarUrl":"/avatars/5c3038ecdcd9fbb1fa816ec8d9ff368f.svg","isPro":false,"fullname":"yohunawu","user":"yohuna","type":"user"},{"_id":"6a28ff6b12b6c5ac38944dd1","avatarUrl":"/avatars/86a098eb74d1f20f7198615225d94cce.svg","isPro":false,"fullname":"zhaolong li","user":"zli2026","type":"user"},{"_id":"694c930ed3e40c5289face86","avatarUrl":"/avatars/f2a23fc19808efa30440ccba487370e6.svg","isPro":false,"fullname":"aiden","user":"aidenjhwu","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6a24e31c749f04abbbb5105d","name":"Tencent-Hunyuan-Multimodal-RL","fullname":"Tencent-Hunyuan-Multimodal-RL","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/66129c7b50350afe76757262/AMk8iMwAaEjA0M5Cdq0NE.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.10968.md"}">

Papers

arxiv:2606.10968

Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning

Published on Jun 9

· Submitted by

Xiangxin Zhou on Jun 10

Tencent-Hunyuan-Multimodal-RL

Upvote

Authors:

Abstract

CPPO addresses limitations in reinforcement learning with verifiable rewards by introducing position-weighted thresholds and cumulative prefix budgeting to better handle autoregressive generation challenges.

Generated by Qwen/Qwen2.5-Coder-32B-Instruct

Reinforcement learning with verifiable rewards (RLVR) has become standard for improving LLM reasoning. However, existing PPO-style trust-region mechanisms remain position-agnostic by enforcing uniform thresholds across all tokens independently. This pointwise treatment conflicts with autoregressive generation in two critical ways. First, uniform thresholds ignore autoregressive asymmetry. Early-stage deviations produce compounding sequence-level drift, causing static thresholds to under-regulate early divergence and excessively constrain late-stage exploration. Second, evaluating token-level divergence in isolation overlooks cumulative prefix drift, granting the same divergence allowance regardless of how far the conditioning history has already deviated from the rollout policy. To address this limitation, we propose CPPO (Cumulative Prefix-divergence Policy Optimization), a token-level masking rule that aligns updates with a finite-horizon policy-improvement bound via two coupled mechanisms. First, a position-weighted threshold imposes stricter limits at early positions whose effects persist longer, relaxing constraints for late-stage tokens. Second, a cumulative prefix budget tracks historical deviations, dynamically restricting further token-level deviation to prevent compounding errors along the prefix. Empirically, CPPO enhances training stability and significantly improves reasoning accuracy across various model scales.

View arXiv page View PDF Add to collection