On-policy training has emerged as a powerful post-training paradigm for improving the reasoning capabilities of large language models, and is often enhanced by golden trajectories from stronger expert models. However, when the expert fails on harder problems, existing trajectory-guided methods lose their main source of supervision, and these failed trajectories are typically discarded as negative samples. We argue that such failures, which we call Golden Negative Trajectories, can still provide valuable reasoning signals when treated not as demonstrations to imitate, but as flawed trajectories to reflect upon. We identify a Reflection Advantage: for hard problems, reflecting on a flawed trajectory can be easier and more effective than solving the problem directly from scratch. Motivated by this, we propose ReflectRL, a lightweight plug-and-play framework that learns from Golden Negative Trajectories during on-policy training. ReflectRL first uses these trajectories to elicit Reflective Reasoning, then applies Reflective-to-Direct Policy Transition to transfer the acquired reasoning behavior back to Direct Reasoning. Experiments across 9 benchmarks, 4 LLM backbones, and 4 on-policy training methods show that ReflectRL consistently improves reasoning performance with minimal overhead.</p>\n","updatedAt":"2026-08-05T16:53:34.146Z","author":{"_id":"63b28be9677046a1428284fe","avatarUrl":"/avatars/e4f36949862eac7a183ea0de20b0e500.svg","fullname":"Jinhe Bi","name":"Jinhe","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9239597916603088},"editors":["Jinhe"],"editorAvatarUrls":["/avatars/e4f36949862eac7a183ea0de20b0e500.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.03972","authors":[{"_id":"6a7369b921d743496486c54a","name":"Jinhe Bi","hidden":false},{"_id":"6a7369b921d743496486c54b","name":"Chennan Zhou","hidden":false},{"_id":"6a7369b921d743496486c54c","name":"Zengjie Jin","hidden":false},{"_id":"6a7369b921d743496486c54d","name":"Aniri","hidden":false},{"_id":"6a7369b921d743496486c54e","name":"Shuo Lu","hidden":false},{"_id":"6a7369b921d743496486c54f","name":"Wenke Huang","hidden":false},{"_id":"6a7369b921d743496486c550","name":"Hu Cao","hidden":false},{"_id":"6a7369b921d743496486c551","name":"Xun Xiao","hidden":false},{"_id":"6a7369b921d743496486c552","name":"Zhihong Zhu","hidden":false},{"_id":"6a7369b921d743496486c553","name":"Volker Tresp","hidden":false},{"_id":"6a7369b921d743496486c554","name":"Fei Shen","hidden":false},{"_id":"6a7369b921d743496486c555","name":"Yunpu Ma","hidden":false},{"_id":"6a7369b921d743496486c556","name":"Tat-Seng Chua","hidden":false}],"publishedAt":"2026-08-04T00:00:00.000Z","submittedOnDailyAt":"2026-08-05T00:00:00.000Z","title":"ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoning","submittedOnDailyBy":{"_id":"63b28be9677046a1428284fe","avatarUrl":"/avatars/e4f36949862eac7a183ea0de20b0e500.svg","isPro":false,"fullname":"Jinhe Bi","user":"Jinhe","type":"user","name":"Jinhe"},"summary":"On-policy training has emerged as a powerful post-training paradigm for improving the reasoning capabilities of large language models, and is often enhanced by golden trajectories from stronger expert models. However, when the expert fails on harder problems, existing trajectory-guided methods lose their main source of supervision, and these failed trajectories are typically discarded as negative samples. We argue that such failures, which we call Golden Negative Trajectories, can still provide valuable reasoning signals when treated not as demonstrations to imitate, but as flawed trajectories to reflect upon. We identify a Reflection Advantage: for hard problems, reflecting on a flawed trajectory can be easier and more effective than solving the problem directly from scratch. Motivated by this, we propose ReflectRL, a lightweight plug-and-play framework that learns from Golden Negative Trajectories during on-policy training. ReflectRL first uses these trajectories to elicit Reflective Reasoning, then applies Reflective-to-Direct Policy Transition to transfer the acquired reasoning behavior back to Direct Reasoning. Experiments across 9 benchmarks, 4 LLM backbones, and 4 on-policy training methods show that ReflectRL consistently improves reasoning performance with minimal overhead.","upvotes":1,"discussionId":"6a7369b921d743496486c557","githubRepo":"https://github.com/bibisbar/ReflectRL","githubRepoAddedBy":"user","githubStars":1,"organization":{"_id":"6508ab2b349930913196378b","name":"NationalUniversityofSingapore","fullname":"National University of Singapore","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/630ca0817dacb93b33506ce7/ZYUmpSMsa5Whihw3me2Bw.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"63b28be9677046a1428284fe","avatarUrl":"/avatars/e4f36949862eac7a183ea0de20b0e500.svg","isPro":false,"fullname":"Jinhe Bi","user":"Jinhe","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6508ab2b349930913196378b","name":"NationalUniversityofSingapore","fullname":"National University of Singapore","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/630ca0817dacb93b33506ce7/ZYUmpSMsa5Whihw3me2Bw.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.03972.md","query":{}}">
ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoning
Abstract
On-policy training has emerged as a powerful post-training paradigm for improving the reasoning capabilities of large language models, and is often enhanced by golden trajectories from stronger expert models. However, when the expert fails on harder problems, existing trajectory-guided methods lose their main source of supervision, and these failed trajectories are typically discarded as negative samples. We argue that such failures, which we call Golden Negative Trajectories, can still provide valuable reasoning signals when treated not as demonstrations to imitate, but as flawed trajectories to reflect upon. We identify a Reflection Advantage: for hard problems, reflecting on a flawed trajectory can be easier and more effective than solving the problem directly from scratch. Motivated by this, we propose ReflectRL, a lightweight plug-and-play framework that learns from Golden Negative Trajectories during on-policy training. ReflectRL first uses these trajectories to elicit Reflective Reasoning, then applies Reflective-to-Direct Policy Transition to transfer the acquired reasoning behavior back to Direct Reasoning. Experiments across 9 benchmarks, 4 LLM backbones, and 4 on-policy training methods show that ReflectRL consistently improves reasoning performance with minimal overhead.
Community
On-policy training has emerged as a powerful post-training paradigm for improving the reasoning capabilities of large language models, and is often enhanced by golden trajectories from stronger expert models. However, when the expert fails on harder problems, existing trajectory-guided methods lose their main source of supervision, and these failed trajectories are typically discarded as negative samples. We argue that such failures, which we call Golden Negative Trajectories, can still provide valuable reasoning signals when treated not as demonstrations to imitate, but as flawed trajectories to reflect upon. We identify a Reflection Advantage: for hard problems, reflecting on a flawed trajectory can be easier and more effective than solving the problem directly from scratch. Motivated by this, we propose ReflectRL, a lightweight plug-and-play framework that learns from Golden Negative Trajectories during on-policy training. ReflectRL first uses these trajectories to elicit Reflective Reasoning, then applies Reflective-to-Direct Policy Transition to transfer the acquired reasoning behavior back to Direct Reasoning. Experiments across 9 benchmarks, 4 LLM backbones, and 4 on-policy training methods show that ReflectRL consistently improves reasoning performance with minimal overhead.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.03972 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.03972 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.03972 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.