token credit assignment for reinforcement learning based on guidance signals</p>\n","updatedAt":"2026-06-01T16:34:38.677Z","author":{"_id":"6310531914aa81e1044363ed","avatarUrl":"/avatars/ae7767e591cb7199ea2f62d2db89fc7f.svg","fullname":"Shufan Li","name":"jacklishufan","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":8,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9437058568000793},"editors":["jacklishufan"],"editorAvatarUrls":["/avatars/ae7767e591cb7199ea2f62d2db89fc7f.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.29198","authors":[{"_id":"6a1db40d808ddbc3c7d439a0","name":"Shufan Li","hidden":false},{"_id":"6a1db40d808ddbc3c7d439a1","name":"Konstantinos Kallidromitis","hidden":false},{"_id":"6a1db40d808ddbc3c7d439a2","name":"Akash Gokul","hidden":false},{"_id":"6a1db40d808ddbc3c7d439a3","name":"Yuta Kyuragi","hidden":false},{"_id":"6a1db40d808ddbc3c7d439a4","name":"Aditya Grover","hidden":false}],"publishedAt":"2026-05-29T00:00:00.000Z","submittedOnDailyAt":"2026-06-01T00:00:00.000Z","title":"Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization","submittedOnDailyBy":{"_id":"6310531914aa81e1044363ed","avatarUrl":"/avatars/ae7767e591cb7199ea2f62d2db89fc7f.svg","isPro":false,"fullname":"Shufan Li","user":"jacklishufan","type":"user","name":"jacklishufan"},"summary":"Group-advantage-based reinforcement learning methods, such as GRPO and DAPO, have demonstrated strong performance across diverse domains, including mathematical reasoning and text-to-image generation. However, their reliance on sample-level rewards introduces a key limitation as uniform credit assignment across all tokens fails to capture fine-grained, token-level contributions. To address this issue, we propose Guidance Contrastive Policy Optimization (GCPO), a novel algorithm that enables per-token credit assignment by contrasting model predictions under positive and negative prompts. Rather than uniformly broadcasting sample-level advantages, GCPO assigns token-level advantages proportional to the difference between these contrastive predictions, allowing more precise and informative learning signals. Empirically, we find that GCPO emphasizes semantically relevant regions such as visual areas aligned with textual prompts in text-to-image generation, and critical keywords within reasoning traces for chain-of-thought tasks. Through extensive experiments, GCPO consistently outperforms GRPO and DAPO baselines on both text-to-image generation and chain-of-thought reasoning benchmarks, demonstrating its effectiveness as a general and scalable optimization strategy for discrete policy learning.","upvotes":0,"discussionId":"6a1db40d808ddbc3c7d439a5","githubRepo":"https://github.com/jacklishufan/gcpo","githubRepoAddedBy":"user","ai_summary":"GCPO enables per-token credit assignment in reinforcement learning by contrasting model predictions under positive and negative prompts, improving performance in text-to-image generation and chain-of-thought reasoning tasks.","ai_keywords":["reinforcement learning","GRPO","DAPO","token-level credit assignment","guidance contrastive policy optimization","contrastive predictions","discrete policy learning","chain-of-thought reasoning","text-to-image generation"],"githubStars":0,"organization":{"_id":"67784c39dac147922d8d09f0","name":"UCLA","fullname":"University of California, Los Angeles","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/67784bd637dfa531fbce95a2/Nf0seEMEn66sPL3QsJXj4.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[],"acceptLanguages":["en"],"organization":{"_id":"67784c39dac147922d8d09f0","name":"UCLA","fullname":"University of California, Los Angeles","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/67784bd637dfa531fbce95a2/Nf0seEMEn66sPL3QsJXj4.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.29198.md"}">
Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization
Abstract
GCPO enables per-token credit assignment in reinforcement learning by contrasting model predictions under positive and negative prompts, improving performance in text-to-image generation and chain-of-thought reasoning tasks.
AI-generated summary
Group-advantage-based reinforcement learning methods, such as GRPO and DAPO, have demonstrated strong performance across diverse domains, including mathematical reasoning and text-to-image generation. However, their reliance on sample-level rewards introduces a key limitation as uniform credit assignment across all tokens fails to capture fine-grained, token-level contributions. To address this issue, we propose Guidance Contrastive Policy Optimization (GCPO), a novel algorithm that enables per-token credit assignment by contrasting model predictions under positive and negative prompts. Rather than uniformly broadcasting sample-level advantages, GCPO assigns token-level advantages proportional to the difference between these contrastive predictions, allowing more precise and informative learning signals. Empirically, we find that GCPO emphasizes semantically relevant regions such as visual areas aligned with textual prompts in text-to-image generation, and critical keywords within reasoning traces for chain-of-thought tasks. Through extensive experiments, GCPO consistently outperforms GRPO and DAPO baselines on both text-to-image generation and chain-of-thought reasoning benchmarks, demonstrating its effectiveness as a general and scalable optimization strategy for discrete policy learning.
Community
token credit assignment for reinforcement learning based on guidance signals
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.29198 in a model README.md to link it from this page.
Cite arxiv.org/abs/2605.29198 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2605.29198 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.