On-Policy Feedback for Reward Model Self-Supervised Improvement</p>\n","updatedAt":"2026-06-01T04:51:20.655Z","author":{"_id":"65e933fcacb29a596cd6eb6e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65e933fcacb29a596cd6eb6e/dlH14rO2I4EUMVyjEjdRK.jpeg","fullname":"Xiaobo Wang","name":"Yofuria","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7411664724349976},"editors":["Yofuria"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/65e933fcacb29a596cd6eb6e/dlH14rO2I4EUMVyjEjdRK.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.30888","authors":[{"_id":"6a1d0e7d808ddbc3c7d435e4","user":{"_id":"65e933fcacb29a596cd6eb6e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65e933fcacb29a596cd6eb6e/dlH14rO2I4EUMVyjEjdRK.jpeg","isPro":false,"fullname":"Xiaobo Wang","user":"Yofuria","type":"user","name":"Yofuria"},"name":"Xiaobo Wang","status":"claimed_verified","statusLastChangedAt":"2026-06-01T09:32:44.215Z","hidden":false},{"_id":"6a1d0e7d808ddbc3c7d435e5","name":"Tong Wu","hidden":false},{"_id":"6a1d0e7d808ddbc3c7d435e6","name":"Min Tang","hidden":false},{"_id":"6a1d0e7d808ddbc3c7d435e7","name":"Jiaqi Li","hidden":false},{"_id":"6a1d0e7d808ddbc3c7d435e8","name":"Qi Liu","hidden":false},{"_id":"6a1d0e7d808ddbc3c7d435e9","name":"Zilong Zheng","hidden":false}],"publishedAt":"2026-05-29T00:00:00.000Z","submittedOnDailyAt":"2026-06-01T00:00:00.000Z","title":"The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement","submittedOnDailyBy":{"_id":"65e933fcacb29a596cd6eb6e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65e933fcacb29a596cd6eb6e/dlH14rO2I4EUMVyjEjdRK.jpeg","isPro":false,"fullname":"Xiaobo Wang","user":"Yofuria","type":"user","name":"Yofuria"},"summary":"Building strong reward models (RMs) for language model alignment is bottlenecked by the cost and difficulty of acquiring diverse and reliable preference data from human annotation or judge models. It is dramatically worse as the policy evolves beyond the static RM training. Therefore, we propose SAVE (Self-supervised reward model improvement via Value-Anchored On-policy feedback), a framework that grades on-policy responses as feedback by using the value function for on-policy RM training. SAVE naturally converts the reward-graded on-policy responses into supervision with a prompt-specific value head as an adaptive anchor. It computes RM advantages and filters ambiguous samples to update the RM via a contrastive objective. The effectiveness of SAVE for enhancing RM training is strongly validated through rigorous empirical evaluation across six diverse benchmarks. It achieves outperforming results across all datasets while maintaining consistent improvements across three RL algorithms (GRPO, RLOO, GSPO) and different policy backbones.","upvotes":2,"discussionId":"6a1d0e7e808ddbc3c7d435ea","ai_summary":"SAVE framework improves reward model training by using value functions to grade on-policy responses and update models through contrastive objectives.","ai_keywords":["reward models","on-policy responses","value function","contrastive objective","reinforcement learning","GRPO","RLOO","GSPO"]},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"65e933fcacb29a596cd6eb6e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65e933fcacb29a596cd6eb6e/dlH14rO2I4EUMVyjEjdRK.jpeg","isPro":false,"fullname":"Xiaobo Wang","user":"Yofuria","type":"user"},{"_id":"6696423a0761ae8772b8b73e","avatarUrl":"/avatars/bc490b2d17a9841b35da6a543e5d7106.svg","isPro":false,"fullname":"tang min","user":"tangmin","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.30888.md"}">
The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement
Abstract
SAVE framework improves reward model training by using value functions to grade on-policy responses and update models through contrastive objectives.
AI-generated summary
Building strong reward models (RMs) for language model alignment is bottlenecked by the cost and difficulty of acquiring diverse and reliable preference data from human annotation or judge models. It is dramatically worse as the policy evolves beyond the static RM training. Therefore, we propose SAVE (Self-supervised reward model improvement via Value-Anchored On-policy feedback), a framework that grades on-policy responses as feedback by using the value function for on-policy RM training. SAVE naturally converts the reward-graded on-policy responses into supervision with a prompt-specific value head as an adaptive anchor. It computes RM advantages and filters ambiguous samples to update the RM via a contrastive objective. The effectiveness of SAVE for enhancing RM training is strongly validated through rigorous empirical evaluation across six diverse benchmarks. It achieves outperforming results across all datasets while maintaining consistent improvements across three RL algorithms (GRPO, RLOO, GSPO) and different policy backbones.
Community
On-Policy Feedback for Reward Model Self-Supervised Improvement
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.30888 in a model README.md to link it from this page.
Cite arxiv.org/abs/2605.30888 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2605.30888 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.