EMNLP’26<br>✅ Query-KL Replaces Policy-KL<br>✅ Plug-and-Play with DAPO &amp; Other RL Methods<br>✅ Stronger Reasoning Across 6 Benchmarks<br>Moving regularization from responses to queries for more stable and exploratory LLM reasoning. 🚀</p>\n","updatedAt":"2026-08-25T07:23:55.431Z","author":{"_id":"63006304a9df0b1a3285326f","avatarUrl":"/avatars/030f9aa81d1d6370a439e03514ef770f.svg","fullname":"he","name":"xinyu1607","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7496104836463928},"editors":["xinyu1607"],"editorAvatarUrls":["/avatars/030f9aa81d1d6370a439e03514ef770f.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.23311","authors":[{"_id":"6a8d41e05add2537c32e98df","name":"Xianlei Zhou","hidden":false},{"_id":"6a8d41e05add2537c32e98e0","name":"Xiangdi Meng","hidden":false},{"_id":"6a8d41e05add2537c32e98e1","name":"Yu He","hidden":false},{"_id":"6a8d41e05add2537c32e98e2","name":"Tianyu Qi","hidden":false},{"_id":"6a8d41e05add2537c32e98e3","name":"Shuyan Guan","hidden":false},{"_id":"6a8d41e05add2537c32e98e4","name":"Xianli Zhang","hidden":false},{"_id":"6a8d41e05add2537c32e98e5","name":"Jian Zhang","hidden":false},{"_id":"6a8d41e05add2537c32e98e6","name":"Xin Li","hidden":false},{"_id":"6a8d41e05add2537c32e98e7","name":"Qika Lin","hidden":false},{"_id":"6a8d41e05add2537c32e98e8","name":"Jun Liu","hidden":false}],"publishedAt":"2026-08-24T00:00:00.000Z","submittedOnDailyAt":"2026-08-25T00:00:00.000Z","title":"Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization","submittedOnDailyBy":{"_id":"63006304a9df0b1a3285326f","avatarUrl":"/avatars/030f9aa81d1d6370a439e03514ef770f.svg","isPro":false,"fullname":"he","user":"xinyu1607","type":"user","name":"xinyu1607"},"summary":"Policy optimization (PO) for Large Language Models faces a stability--exploration trade-off, currently mediated by an action-side Policy-KL regularizer. This puts practitioners in a double bind: keeping Policy-KL constrains response behavior and consumes the action-side exploration budget, while dropping it leaves the optimization without an explicit drift control. We argue for an alternative that breaks the dilemma by moving regularization to the input side. As training progresses, the distribution over training queries induced by the current policy drifts unchecked from its pre-RL reference distribution.\n Concretely, Environment-Regularized Policy Optimization (ERPO) introduces a Query-KL (QKL) term that bounds this query distribution shift, together with a dataset-static reference-derived per-query weight that biases each per-query update toward queries typical under the reference. The QKL gradient flows strictly through the query likelihood; the response score function used by policy-gradient estimators does not appear in the QKL term, so QKL exerts no direct gradient pressure on the response distribution---exploration is preserved. ERPO plugs into GRPO/PPO/REINFORCE-style pipelines without additional forward passes. On six mathematical reasoning benchmarks, ERPO replaces the standard Policy-KL regularizer while achieving effective control over query distribution drift, delivering stronger accuracy and substantially more stable behavior under high-temperature decoding and long-horizon training.Our source code are available at https://github.com/alibaba/ERPO","upvotes":10,"discussionId":"6a8d41e05add2537c32e98e9","ai_summary":"ERPO replaces action-side policy regularization with input-side query distribution control to stabilize reinforcement learning for language models while preserving response exploration.","ai_keywords":["Policy Optimization","Policy-KL regularizer","Environment-Regularized Policy Optimization","Query-KL","GRPO","PPO","REINFORCE","policy-gradient estimators"],"ai_summary_model":"thinkingmachines/Inkling-Small","organization":{"_id":"68be41370a3fcebdcad6516a","name":"alibabagroup","fullname":"alibaba","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/68be3ab7e52df040b2cf80dc/li4G29u_EGswyTN1Sm_Kq.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"63006304a9df0b1a3285326f","avatarUrl":"/avatars/030f9aa81d1d6370a439e03514ef770f.svg","isPro":false,"fullname":"he","user":"xinyu1607","type":"user"},{"_id":"67e0dc49daf1e39a7d15e67f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/GsHxvMtp5jW58LunxVorc.png","isPro":false,"fullname":"Zhiyuan Wang","user":"Pekku","type":"user"},{"_id":"658be7fe135580745c510323","avatarUrl":"/avatars/830e5cec4565efdc23226a86a0fcef0e.svg","isPro":false,"fullname":"Jian Zhang","user":"VentureZJ","type":"user"},{"_id":"658d6101e7f846313c275353","avatarUrl":"/avatars/6633d6702c9502e87712586b294d5e2d.svg","isPro":false,"fullname":"zhou-xian-lei","user":"zhou-xl","type":"user"},{"_id":"67e0e28d9a10dad714bb1a46","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/rVPGvGnHJD1f6d6Uehgk3.png","isPro":false,"fullname":"Tt kingdom","user":"Sorale","type":"user"},{"_id":"67e0e08d372cafa0dd5ae245","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/EzBYs9iyjOinnKfZZJg_3.png","isPro":false,"fullname":"Mickey","user":"Jhapter","type":"user"},{"_id":"67e0e44c78e66717a1065bfa","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/tLGH704kWcVJleLOoieTH.png","isPro":false,"fullname":"Kacn","user":"Rankery","type":"user"},{"_id":"6311ef8241ebdfa5f11a4ecc","avatarUrl":"/avatars/2f6a5741c883f4a0839ead051723aedd.svg","isPro":false,"fullname":"carlo meng","user":"CCCarloooo","type":"user"},{"_id":"66ac77011cfb12c087605acb","avatarUrl":"/avatars/54c06bd1c4c9d491470ed4162c2301ae.svg","isPro":false,"fullname":"Lin","user":"Qika","type":"user"},{"_id":"6a8d54a22370f401d3df4794","avatarUrl":"/avatars/dadefdf2ee32b72f0d136d9bf66c57f1.svg","isPro":false,"fullname":"toooom","user":"aazjjasd","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"68be41370a3fcebdcad6516a","name":"alibabagroup","fullname":"alibaba","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/68be3ab7e52df040b2cf80dc/li4G29u_EGswyTN1Sm_Kq.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.23311.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2608.23311 in a model README.md to link it from this page.

Cite arxiv.org/abs/2608.23311 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2608.23311 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.