ESPO (Early-Stopping Proximal Policy Optimization) tackles a key waste in RL training of reasoning LLMs: when a model errs early in a trajectory, standard algorithms keep generating to the maximum length, burning compute on never-rewarded tokens and polluting advantage estimates with post-failure noise. ESPO detects failure on-the-fly—using a surrogate regret computed from logits already produced during sampling—and truncates the rollout, treating it as an absorbing failure state with a terminal reward that concentrates negative TD errors at the failure step, all without any extra reward model or human annotation.</p>\n","updatedAt":"2026-06-02T02:05:23.972Z","author":{"_id":"62e0ef42edb0462c8d51818d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62e0ef42edb0462c8d51818d/3YM7DUynIWiiRFM6_enpg.jpeg","fullname":"Ting-En Lin","name":"tnlin","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":15,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8763687014579773},"editors":["tnlin"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/62e0ef42edb0462c8d51818d/3YM7DUynIWiiRFM6_enpg.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.29860","authors":[{"_id":"6a1e3659808ddbc3c7d43bb3","name":"Zihang Li","hidden":false},{"_id":"6a1e3659808ddbc3c7d43bb4","name":"Rui Zhou","hidden":false},{"_id":"6a1e3659808ddbc3c7d43bb5","name":"Yingcheng Shi","hidden":false},{"_id":"6a1e3659808ddbc3c7d43bb6","name":"Wenhan Yu","hidden":false},{"_id":"6a1e3659808ddbc3c7d43bb7","name":"Zhewen Tan","hidden":false},{"_id":"6a1e3659808ddbc3c7d43bb8","name":"Zixiang Liu","hidden":false},{"_id":"6a1e3659808ddbc3c7d43bb9","name":"Zeming Li","hidden":false},{"_id":"6a1e3659808ddbc3c7d43bba","name":"Binhua Li","hidden":false},{"_id":"6a1e3659808ddbc3c7d43bbb","name":"Yongbin Li","hidden":false},{"_id":"6a1e3659808ddbc3c7d43bbc","name":"Tong Yang","hidden":false},{"_id":"6a1e3659808ddbc3c7d43bbd","name":"Jieping Ye","hidden":false}],"publishedAt":"2026-05-28T00:00:00.000Z","submittedOnDailyAt":"2026-06-02T00:00:00.000Z","title":"ESPO: Early-Stopping Proximal Policy Optimization","submittedOnDailyBy":{"_id":"62e0ef42edb0462c8d51818d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62e0ef42edb0462c8d51818d/3YM7DUynIWiiRFM6_enpg.jpeg","isPro":false,"fullname":"Ting-En Lin","user":"tnlin","type":"user","name":"tnlin"},"summary":"When a large language model under reinforcement learning commits a wrong reasoning step early in a trajectory, standard algorithms force it to keep generating until the maximum horizon, spending compute on tokens that never receive positive reward and polluting advantage estimates with post-failure noise. We propose ESPO (Early-Stopping Proximal Policy Optimization), which detects trajectory failure on-the-fly and terminates rollouts early. At each generation step, ESPO computes a surrogate regret using only the logits already computed during sampling, and terminates when the smoothed cumulative regret significantly exceeds its estimated values. Truncated trajectories are treated as absorbing failure states with a terminal reward, concentrating negative temporal-difference (TD) errors near the detected failure step without any additional reward model or human annotation. On DeepSeek-R1-Distill-Qwen-7B trained for mathematical reasoning, ESPO surpasses PPO on AIME~2024 (46.28% vs. 45.25%), AMC~2023 (85.83% vs. 82.94%), and MATH-500 (87.42% vs. 85.43%), while saving more than 20% rollout tokens cumulatively.","upvotes":11,"discussionId":"6a1e3659808ddbc3c7d43bbe","ai_summary":"ESPO improves mathematical reasoning in large language models by detecting and terminating failed trajectories early, leading to better performance and reduced computational waste.","ai_keywords":["reinforcement learning","proximal policy optimization","trajectory failure","surrogate regret","logits","temporal-difference errors","absorbing failure states","rollout tokens","mathematical reasoning"],"organization":{"_id":"67d15cca6e2cf0e062dbfb54","name":"AlibabaTongyiLab","fullname":"TongyiLab","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/67d1502bfabfe9974d1f77bb/XdUSVf6HqBzE7zFBfSDQP.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"62e0ef42edb0462c8d51818d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62e0ef42edb0462c8d51818d/3YM7DUynIWiiRFM6_enpg.jpeg","isPro":false,"fullname":"Ting-En Lin","user":"tnlin","type":"user"},{"_id":"6407e5294edf9f5c4fd32228","avatarUrl":"/avatars/8e2d55460e9fe9c426eb552baf4b2cb0.svg","isPro":false,"fullname":"Stoney Kang","user":"sikang99","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"660e5cc5d0a9664d8becaadf","avatarUrl":"/avatars/9806e1c7dfa272b0a1b5f1997bc2a63f.svg","isPro":false,"fullname":"Raymond","user":"Raysada","type":"user"},{"_id":"6721d8a882531630c7b30c4b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/JAnRGpiq3_SnCFtxG8PxJ.png","isPro":false,"fullname":"tanzhewen","user":"tanzhewen","type":"user"},{"_id":"64a3897a34612d376415545c","avatarUrl":"/avatars/3d69ce7f59783e51fd8a97830333ead6.svg","isPro":false,"fullname":"shiyingcheng","user":"shiyingcheng","type":"user"},{"_id":"63469d24a6f101c7f9051319","avatarUrl":"/avatars/a3f5f3e34f2ac2cb5185b413dafc765f.svg","isPro":false,"fullname":"Caoyc","user":"cyc125065","type":"user"},{"_id":"64228bd266d8fdee07b402bf","avatarUrl":"/avatars/89c90ca3d94d8cb99924746d2c3522c4.svg","isPro":false,"fullname":"yibo","user":"rookielyb","type":"user"},{"_id":"69ccb155b2619f648f8e0e5b","avatarUrl":"/avatars/f233cf7c702dd3ef50c9a9fcc034ab83.svg","isPro":false,"fullname":"朱怡然","user":"MatthewNelson20","type":"user"},{"_id":"63ca8e060609f1def7e6548a","avatarUrl":"/avatars/1da7947840cb87d5f77c0af9ee11f9c2.svg","isPro":true,"fullname":"Yi Jung","user":"YJ-142150","type":"user"},{"_id":"699eda17233213e661d69edb","avatarUrl":"/avatars/72e15701bad7d6b9ba6a62837ea15624.svg","isPro":false,"fullname":"韩千怡","user":"noahga","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"67d15cca6e2cf0e062dbfb54","name":"AlibabaTongyiLab","fullname":"TongyiLab","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/67d1502bfabfe9974d1f77bb/XdUSVf6HqBzE7zFBfSDQP.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.29860.md"}">
ESPO: Early-Stopping Proximal Policy Optimization
Authors: ,
,
,
,
,
,
,
,
,
,
Abstract
ESPO improves mathematical reasoning in large language models by detecting and terminating failed trajectories early, leading to better performance and reduced computational waste.
AI-generated summary
When a large language model under reinforcement learning commits a wrong reasoning step early in a trajectory, standard algorithms force it to keep generating until the maximum horizon, spending compute on tokens that never receive positive reward and polluting advantage estimates with post-failure noise. We propose ESPO (Early-Stopping Proximal Policy Optimization), which detects trajectory failure on-the-fly and terminates rollouts early. At each generation step, ESPO computes a surrogate regret using only the logits already computed during sampling, and terminates when the smoothed cumulative regret significantly exceeds its estimated values. Truncated trajectories are treated as absorbing failure states with a terminal reward, concentrating negative temporal-difference (TD) errors near the detected failure step without any additional reward model or human annotation. On DeepSeek-R1-Distill-Qwen-7B trained for mathematical reasoning, ESPO surpasses PPO on AIME~2024 (46.28% vs. 45.25%), AMC~2023 (85.83% vs. 82.94%), and MATH-500 (87.42% vs. 85.43%), while saving more than 20% rollout tokens cumulatively.
Community
ESPO (Early-Stopping Proximal Policy Optimization) tackles a key waste in RL training of reasoning LLMs: when a model errs early in a trajectory, standard algorithms keep generating to the maximum length, burning compute on never-rewarded tokens and polluting advantage estimates with post-failure noise. ESPO detects failure on-the-fly—using a surrogate regret computed from logits already produced during sampling—and truncates the rollout, treating it as an absorbing failure state with a terminal reward that concentrates negative TD errors at the failure step, all without any extra reward model or human annotation.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.29860 in a model README.md to link it from this page.
Cite arxiv.org/abs/2605.29860 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2605.29860 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.