Hugging Face Daily Papers · June 12, 2026 · 3 min read

N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization<br>Github: <a href=\"https://github.com/ZJUSCL/N-GRPO\" rel=\"nofollow\">https://github.com/ZJUSCL/N-GRPO</a></p>\n","updatedAt":"2026-06-12T02:14:04.129Z","author":{"_id":"656da08ae9f2c52b98d59b14","avatarUrl":"/avatars/93e36b07b16108aaa7be55a75f9f6239.svg","fullname":"Xukun Zhu","name":"MoonKuun","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.6660481691360474},"editors":["MoonKuun"],"editorAvatarUrls":["/avatars/93e36b07b16108aaa7be55a75f9f6239.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.10768","authors":[{"_id":"6a28c690e7d78ea7587e5331","name":"Xukun Zhu","hidden":false},{"_id":"6a28c690e7d78ea7587e5332","name":"Hang Yu","hidden":false},{"_id":"6a28c690e7d78ea7587e5333","name":"Peng Di","hidden":false},{"_id":"6a28c690e7d78ea7587e5334","name":"Linchao Zhu","hidden":false}],"publishedAt":"2026-06-09T00:00:00.000Z","submittedOnDailyAt":"2026-06-12T00:00:00.000Z","title":"N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization","submittedOnDailyBy":{"_id":"656da08ae9f2c52b98d59b14","avatarUrl":"/avatars/93e36b07b16108aaa7be55a75f9f6239.svg","isPro":false,"fullname":"Xukun Zhu","user":"MoonKuun","type":"user","name":"MoonKuun"},"summary":"The success of Large Language Models in mathematical reasoning relies heavily on the generation of diverse and valid solution paths during the rollout phase. However, current rollout techniques face a fundamental trade-off: token-level sampling often yields redundant trajectories that differ only in rephrasing, while embedding-level methods utilizing random noise frequently disrupt semantic consistency. To resolve this, we introduce N-GRPO, a novel exploration strategy integrated into the Group Relative Policy Optimization (GRPO) framework. Rather than relying on token-level sampling or native embedding-level noise, our approach leverages Semantic Neighbor Mixing. This mechanism dynamically constructs input representations by mixing the embeddings of an anchor token and its nearest semantic neighbors, thereby injecting diversity while strictly adhering to the local semantic manifold. Experimental evaluations on the DeepSeek-R1-Distill-Qwen models across different sizes show that N-GRPO not only achieves consistent improvements over strong baselines on math reasoning benchmarks but also exhibits robust generalization capabilities on out-of-distribution tasks.","upvotes":16,"discussionId":"6a28c690e7d78ea7587e5335","ai_summary":"N-GRPO, a novel exploration strategy within GRPO framework, enhances mathematical reasoning in large language models through semantic neighbor mixing that maintains semantic consistency while injecting diversity.","ai_keywords":["Large Language Models","mathematical reasoning","rollout phase","token-level sampling","embedding-level methods","Group Relative Policy Optimization","Semantic Neighbor Mixing","semantic manifold","math reasoning benchmarks","out-of-distribution tasks"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"656da08ae9f2c52b98d59b14","avatarUrl":"/avatars/93e36b07b16108aaa7be55a75f9f6239.svg","isPro":false,"fullname":"Xukun Zhu","user":"MoonKuun","type":"user"},{"_id":"66aa0c7ad9b96f88862a192f","avatarUrl":"/avatars/5ebf92c2ca8df51a481dcda3cf43d3c5.svg","isPro":false,"fullname":"xinyu qiu","user":"x1nyu","type":"user"},{"_id":"64f5f8dd9b17cd59c453c57f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64f5f8dd9b17cd59c453c57f/MulhwLcePFUWUQel8LQZ8.jpeg","isPro":false,"fullname":"Xinyu Fang","user":"nebulae09","type":"user"},{"_id":"65785ee4eb02736add74db01","avatarUrl":"/avatars/e175ee0989dff644ed91d0b5f22df5f3.svg","isPro":false,"fullname":"Cheng_Chen","user":"Zacks-Chen","type":"user"},{"_id":"64f9c64c698370119fb36b6a","avatarUrl":"/avatars/dae735d8c471bcfa5d1a7fc1d921ca13.svg","isPro":false,"fullname":"Shoumobutangshou","user":"SPpump","type":"user"},{"_id":"67e53e5a89cbbcca859b588c","avatarUrl":"/avatars/30831b76965255d3a849e23419f0e32b.svg","isPro":false,"fullname":"MTandHJ","user":"MTandHJ","type":"user"},{"_id":"6222cdbe088b146ce3c8af89","avatarUrl":"/avatars/71abc2949998516183682367c6150c74.svg","isPro":false,"fullname":"GongZi","user":"GoZion","type":"user"},{"_id":"67481b67fe46f9a025c6c26e","avatarUrl":"/avatars/6ef248a7476848ea98790e683d608c5d.svg","isPro":false,"fullname":"Rongyao Cai","user":"RyCai","type":"user"},{"_id":"6625fc61f5c285535ccdfaa4","avatarUrl":"/avatars/0684395ec693f24089d26403fcabdafe.svg","isPro":false,"fullname":"Yang Wj","user":"yangwj211","type":"user"},{"_id":"67172e6269532dd147d0d00e","avatarUrl":"/avatars/3d4f6ea2e17dae2b9236533d2b929e0a.svg","isPro":false,"fullname":"newshawn","user":"newshawn","type":"user"},{"_id":"6837243353739ee0d588d04c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/CYU8xxEHxxGkwNVCpua6X.png","isPro":false,"fullname":"Hang Yu","user":"fhlyhv","type":"user"},{"_id":"640d3e548036cc21422866b7","avatarUrl":"/avatars/116e361cb43230e5fc5e16e62a1c48fe.svg","isPro":false,"fullname":"feather820","user":"feather820","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.10768.md","query":{}}">

Papers

arxiv:2606.10768

N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization

Published on Jun 9

· Submitted by

Xukun Zhu on Jun 12

Upvote

Authors:

Abstract

N-GRPO, a novel exploration strategy within GRPO framework, enhances mathematical reasoning in large language models through semantic neighbor mixing that maintains semantic consistency while injecting diversity.

Generated by Qwen/Qwen2.5-Coder-32B-Instruct

The success of Large Language Models in mathematical reasoning relies heavily on the generation of diverse and valid solution paths during the rollout phase. However, current rollout techniques face a fundamental trade-off: token-level sampling often yields redundant trajectories that differ only in rephrasing, while embedding-level methods utilizing random noise frequently disrupt semantic consistency. To resolve this, we introduce N-GRPO, a novel exploration strategy integrated into the Group Relative Policy Optimization (GRPO) framework. Rather than relying on token-level sampling or native embedding-level noise, our approach leverages Semantic Neighbor Mixing. This mechanism dynamically constructs input representations by mixing the embeddings of an anchor token and its nearest semantic neighbors, thereby injecting diversity while strictly adhering to the local semantic manifold. Experimental evaluations on the DeepSeek-R1-Distill-Qwen models across different sizes show that N-GRPO not only achieves consistent improvements over strong baselines on math reasoning benchmarks but also exhibits robust generalization capabilities on out-of-distribution tasks.