FlowBalance is a verifier-grounded self-improvement method for reasoning models. It addresses one known failure mode of OPSD on thinking models: privileged feedback can reinforce locally plausible but verifier-wrong traces. FlowBalance combines sparse outcome rewards with dense self-feedback while flipping misleading teacher support on failures.</p>\n","updatedAt":"2026-09-08T05:58:52.520Z","author":{"_id":"68266b5261ed4d89177c3612","avatarUrl":"/avatars/e9d214c78916e3cfa554e545fd831446.svg","fullname":"Kishan Panaganti","name":"kishanpb","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.847001314163208},"editors":["kishanpb"],"editorAvatarUrls":["/avatars/e9d214c78916e3cfa554e545fd831446.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2609.03241","authors":[{"_id":"6a9fa2f66c8e10537d563bba","name":"Zixun Huang","hidden":false},{"_id":"6a9fa2f66c8e10537d563bbb","name":"Kishan Panaganti","hidden":false},{"_id":"6a9fa2f66c8e10537d563bbc","name":"Haitao Mi","hidden":false},{"_id":"6a9fa2f66c8e10537d563bbd","name":"Leowei Liang","hidden":false}],"publishedAt":"2026-09-03T00:00:00.000Z","submittedOnDailyAt":"2026-09-08T00:00:00.000Z","title":"FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience","submittedOnDailyBy":{"_id":"68266b5261ed4d89177c3612","avatarUrl":"/avatars/e9d214c78916e3cfa554e545fd831446.svg","isPro":false,"fullname":"Kishan Panaganti","user":"kishanpb","type":"user","name":"kishanpb"},"summary":"A reasoning model can improve from its own on-policy experience, but this inner loop is fragile: terminal verifiers provide reliable yet sparse supervision, while dense same-model guidance can reinforce false confidence or overconcentrate learning on a narrow solution mode. We introduce FlowBalance, a verifier-grounded self-improvement method that learns a normalized distribution over complete responses. For each on-policy trajectory, a frozen training-time view of the same policy uses privileged context to produce token-level log-probability gains, which are aggregated into a trajectory-level self-guidance score. FlowBalance calibrates this score with the verifier-derived group advantage: guidance is retained on positive-advantage trajectories, reversed on negative-advantage trajectories, and disabled when the rollout group provides no outcome preference. The resulting energy exponentially reweights a reference policy, and profiled trajectory balance fits the normalized target with one log-partition estimate per rollout group. This realizes outcome-calibrated self-guidance via trajectory balance, without a separate token-level imitation loss. Our analysis establishes within-group contrast preservation, a minimum-change reverse-KL characterization, monotonic verifier control of target reward, and an exact correction against false-positive self-guidance on rejected responses. On mathematical reasoning, FlowBalance improves average performance over FlowRL on both Qwen3-4B and Qwen3-8B, while also improving training speed and stability, avoiding direct OPSD's response-length collapse, and exhibiting higher correct-strategy diversity in a controlled AIME24 diagnostic.","upvotes":1,"discussionId":"6a9fa2f66c8e10537d563bbe","projectPage":"https://alexhuang13.github.io/FlowBalance-Blog/","githubRepo":"https://github.com/alexhuang13/FlowBalance","githubRepoAddedBy":"user","ai_summary":"FlowBalance improves reasoning models via verifier-calibrated self-guidance using trajectory-level score reweighting and profile-based trajectory balance.","ai_keywords":["FlowBalance","on-policy self-improvement","terminal verifiers","token-level log-probability gains","trajectory-level self-guidance","group advantage","trajectory balance","reverse-KL","false-positive self-guidance","mathematical reasoning"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":0,"organization":{"_id":"6645f953c39288df638dbdd5","name":"Tencent-Hunyuan","fullname":"Tencent Hunyuan","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/62d22496c58f969c152bcefd/woKSjt2wXvBNKussyYPsa.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"68266b5261ed4d89177c3612","avatarUrl":"/avatars/e9d214c78916e3cfa554e545fd831446.svg","isPro":false,"fullname":"Kishan Panaganti","user":"kishanpb","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":3,"organization":{"_id":"6645f953c39288df638dbdd5","name":"Tencent-Hunyuan","fullname":"Tencent Hunyuan","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/62d22496c58f969c152bcefd/woKSjt2wXvBNKussyYPsa.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2609/2609.03241.md","query":{}}">
FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience
Abstract
FlowBalance improves reasoning models via verifier-calibrated self-guidance using trajectory-level score reweighting and profile-based trajectory balance.
A reasoning model can improve from its own on-policy experience, but this inner loop is fragile: terminal verifiers provide reliable yet sparse supervision, while dense same-model guidance can reinforce false confidence or overconcentrate learning on a narrow solution mode. We introduce FlowBalance, a verifier-grounded self-improvement method that learns a normalized distribution over complete responses. For each on-policy trajectory, a frozen training-time view of the same policy uses privileged context to produce token-level log-probability gains, which are aggregated into a trajectory-level self-guidance score. FlowBalance calibrates this score with the verifier-derived group advantage: guidance is retained on positive-advantage trajectories, reversed on negative-advantage trajectories, and disabled when the rollout group provides no outcome preference. The resulting energy exponentially reweights a reference policy, and profiled trajectory balance fits the normalized target with one log-partition estimate per rollout group. This realizes outcome-calibrated self-guidance via trajectory balance, without a separate token-level imitation loss. Our analysis establishes within-group contrast preservation, a minimum-change reverse-KL characterization, monotonic verifier control of target reward, and an exact correction against false-positive self-guidance on rejected responses. On mathematical reasoning, FlowBalance improves average performance over FlowRL on both Qwen3-4B and Qwen3-8B, while also improving training speed and stability, avoiding direct OPSD's response-length collapse, and exhibiting higher correct-strategy diversity in a controlled AIME24 diagnostic.
Community
FlowBalance is a verifier-grounded self-improvement method for reasoning models. It addresses one known failure mode of OPSD on thinking models: privileged feedback can reinforce locally plausible but verifier-wrong traces. FlowBalance combines sparse outcome rewards with dense self-feedback while flipping misleading teacher support on failures.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2609.03241 in a model README.md to link it from this page.
Cite arxiv.org/abs/2609.03241 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2609.03241 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.