Proceedings of the 2026 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Bellevue, WA, USA</p>\n","updatedAt":"2026-07-29T01:49:42.807Z","author":{"_id":"688661c83581aa7d9c1b7c58","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/pnNWojIRacuYGDMvWK104.png","fullname":"Md Rezwanul Haque","name":"mr3haque","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.6656498312950134},"editors":["mr3haque"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/pnNWojIRacuYGDMvWK104.png"],"reactions":[{"reaction":"🔥","users":["GhostScientist"],"count":1}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.25091","authors":[{"_id":"6a695b979d3a1231d492b78b","user":{"_id":"688661c83581aa7d9c1b7c58","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/pnNWojIRacuYGDMvWK104.png","isPro":false,"fullname":"Md Rezwanul Haque","user":"mr3haque","type":"user","name":"mr3haque"},"name":"Md Rezwanul Haque","status":"claimed_verified","statusLastChangedAt":"2026-07-29T08:45:04.413Z","hidden":false},{"_id":"6a695b979d3a1231d492b78c","name":"Md. Milon Islam","hidden":false},{"_id":"6a695b979d3a1231d492b78d","name":"Fakhri Karray","hidden":false}],"publishedAt":"2026-07-27T00:00:00.000Z","submittedOnDailyAt":"2026-07-29T00:00:00.000Z","title":"Towards Robust Reinforcement Learning for Small-Scale Language Model Agents","submittedOnDailyBy":{"_id":"688661c83581aa7d9c1b7c58","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/pnNWojIRacuYGDMvWK104.png","isPro":false,"fullname":"Md Rezwanul Haque","user":"mr3haque","type":"user","name":"mr3haque"},"summary":"The alignment of Small Language Models (SLMs) in the 70--500M parameter range using reinforcement learning is often considered unstable, though the underlying failure mechanisms have not been systematically investigated. In the State-of-the-Art (SOTA) research, fifteen (model, corpus) configurations were trained using Proximal Policy Optimization (PPO). The experiments included Pythia-70M, 160M, 410M and SmolLM2-135M, 360M on the TinyStories, CNN/DailyMail, and Wikitext-103 corpora. Three reproducible failure modes were identified in small-scale language models: silent LoRA parameter freezing in standard PEFT/TRL pipelines, numerical overflow in importance ratios when using bfloat16, and catastrophic policy collapse due to reward-model error. These issues were addressed using a merge-and-reinitialize adapter technique, float32 precision during PPO updates, and a three-layer safety mechanism comprising reward whitening, importance-ratio guarding, and weight rollback. In this paper, a capacity-headroom hypothesis is proposed, which states that PPO performance at the SLM scale depends on both a fluent supervised model (PPL<20) and a discriminative reward signal, rather than on the number of model parameters. The proposed system converged stably in all experiments and improved preference win rate over the SFT baseline in configurations with a fluent prior and an informative reward signal. Furthermore, it outperformed instruction-tuned baselines while requiring significantly less training data. All checkpoints, preference datasets, and training scripts are publicly released^§.","upvotes":3,"discussionId":"6a695b979d3a1231d492b78e","githubRepo":"https://github.com/rezwanh001/SLM-RL-Agents","githubRepoAddedBy":"user","githubStars":2},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"6418d9100956be7233a4f4d7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6418d9100956be7233a4f4d7/m1luRm7lnGdPPrmtEoBr2.png","isPro":false,"fullname":"Dakota Kim","user":"GhostScientist","type":"user"},{"_id":"66d8512c54209e9101811e8e","avatarUrl":"/avatars/62dfd8e6261108f2508efe678d5a2a57.svg","isPro":false,"fullname":"M Saad Salman","user":"MSS444","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.25091.md","query":{}}">
Towards Robust Reinforcement Learning for Small-Scale Language Model Agents
Abstract
The alignment of Small Language Models (SLMs) in the 70--500M parameter range using reinforcement learning is often considered unstable, though the underlying failure mechanisms have not been systematically investigated. In the State-of-the-Art (SOTA) research, fifteen (model, corpus) configurations were trained using Proximal Policy Optimization (PPO). The experiments included Pythia-70M, 160M, 410M and SmolLM2-135M, 360M on the TinyStories, CNN/DailyMail, and Wikitext-103 corpora. Three reproducible failure modes were identified in small-scale language models: silent LoRA parameter freezing in standard PEFT/TRL pipelines, numerical overflow in importance ratios when using bfloat16, and catastrophic policy collapse due to reward-model error. These issues were addressed using a merge-and-reinitialize adapter technique, float32 precision during PPO updates, and a three-layer safety mechanism comprising reward whitening, importance-ratio guarding, and weight rollback. In this paper, a capacity-headroom hypothesis is proposed, which states that PPO performance at the SLM scale depends on both a fluent supervised model (PPL<20) and a discriminative reward signal, rather than on the number of model parameters. The proposed system converged stably in all experiments and improved preference win rate over the SFT baseline in configurations with a fluent prior and an informative reward signal. Furthermore, it outperformed instruction-tuned baselines while requiring significantly less training data. All checkpoints, preference datasets, and training scripts are publicly released^§.
Community
Proceedings of the 2026 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Bellevue, WA, USA
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2607.25091 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.