Hugging Face Daily Papers · June 3, 2026 · 4 min read

A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RL

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

Our new work on multi-domain RL: <strong>A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RL</strong></p>\n","updatedAt":"2026-06-03T04:34:02.924Z","author":{"_id":"655310e0ecf367738e137f12","avatarUrl":"/avatars/c41f41f80c3a03b7be66e802069d95fd.svg","fullname":"Lei Yang","name":"yl-9","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7463562488555908},"editors":["yl-9"],"editorAvatarUrls":["/avatars/c41f41f80c3a03b7be66e802069d95fd.svg"],"reactions":[{"reaction":"🔥","users":["ShuaiyiNie","Zhuowen02","beiweixiaoxu"],"count":3}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.02398","authors":[{"_id":"6a1e7e7e808ddbc3c7d43ef6","user":{"_id":"655310e0ecf367738e137f12","avatarUrl":"/avatars/c41f41f80c3a03b7be66e802069d95fd.svg","isPro":false,"fullname":"Lei Yang","user":"yl-9","type":"user","name":"yl-9"},"name":"Lei Yang","status":"claimed_verified","statusLastChangedAt":"2026-06-02T12:09:20.303Z","hidden":false},{"_id":"6a1e7e7e808ddbc3c7d43ef7","name":"Siyu Ding","hidden":false},{"_id":"6a1e7e7e808ddbc3c7d43ef8","name":"Deyi Xiong","hidden":false}],"publishedAt":"2026-06-01T00:00:00.000Z","submittedOnDailyAt":"2026-06-03T00:00:00.000Z","title":"A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RL","submittedOnDailyBy":{"_id":"655310e0ecf367738e137f12","avatarUrl":"/avatars/c41f41f80c3a03b7be66e802069d95fd.svg","isPro":false,"fullname":"Lei Yang","user":"yl-9","type":"user","name":"yl-9"},"summary":"Reinforcement learning (RL) post-training improves large language models (LLMs) on individual domains such as mathematical reasoning, code generation, question answering, and creative writing (CW), but training on one domain often degrades performance on others. Existing explanations based on catastrophic forgetting or global gradient conflict are incomplete: substantial interference can occur even when full-model gradients are nearly orthogonal. We show that single-domain RL produces sparse, small-magnitude parameter edits with weak overlap among top-changed neurons, while different domains still share substantial active computation routes on which update directions determine whether they act synergistically or conflict. Guided by this observation, we prove under a local perturbation model of multi-domain RL that later-domain training harms an earlier domain mainly through a second-order damage term, which under the observed sparse route structure concentrates in a low-dimensional shared conflict subspace. Moreover, a short domain refresh contracts the harmful component on this subspace, enabling selective recovery with limited collateral damage. Consistent with the theory, a brief Re-Math refresh after Code rightarrow Math rightarrow QA rightarrow CW recovers Math from 57.66 to 66.04 while largely preserving performance on the other domains, yielding the best average score of 66.39. Beyond refresh, a training-free rollback on a sparse proxy conflict coordinate set for the Math-QA pair partially restores Math, providing direct proxy-level evidence for localized damage. These results provide a localized mechanistic account of interference and recovery in multi-domain RL.","upvotes":19,"discussionId":"6a1e7e7e808ddbc3c7d43ef9","ai_summary":"Multi-domain reinforcement learning in language models causes performance degradation through shared computational pathways, but targeted refresh and rollback techniques can selectively recover lost capabilities with minimal side effects.","ai_keywords":["reinforcement learning","large language models","catastrophic forgetting","gradient conflict","parameter edits","neuron overlap","computational routes","local perturbation model","second-order damage term","conflict subspace","domain refresh","rollback","sparse proxy conflict coordinates"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"655310e0ecf367738e137f12","avatarUrl":"/avatars/c41f41f80c3a03b7be66e802069d95fd.svg","isPro":false,"fullname":"Lei Yang","user":"yl-9","type":"user"},{"_id":"66122add438aed66bd3e983d","avatarUrl":"/avatars/e6014713126b69e841cfdca17e9508e0.svg","isPro":false,"fullname":"Bojian Xiong","user":"Gengar0215","type":"user"},{"_id":"66e3fec42cdfb787e6d8c21f","avatarUrl":"/avatars/34ea43a74452c30959b31147822c60bf.svg","isPro":false,"fullname":"Lei Yang","user":"yl-tmp","type":"user"},{"_id":"6747129d92a17b34b63d67a8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/BSC_3kmB910VglrrPCcVU.png","isPro":false,"fullname":"WangZhen","user":"Waldron9898","type":"user"},{"_id":"665e84f6152658fe8d478b1f","avatarUrl":"/avatars/9f08ce6aa78d7576c97e4feaddf77c1e.svg","isPro":false,"fullname":"Shuaiyi Nie","user":"ShuaiyiNie","type":"user"},{"_id":"673ccde008cd1d1dfe43b428","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/BdnQWGqMULuAJaJ7jS2Wr.png","isPro":false,"fullname":"Shijun Xiao","user":"LanWink","type":"user"},{"_id":"641e69355c348064a8251471","avatarUrl":"/avatars/acad3877df27ff44ea3921bb43e34d53.svg","isPro":false,"fullname":"Linhao Yu","user":"HasuerYu","type":"user"},{"_id":"6549d6aaa4635977149027c0","avatarUrl":"/avatars/2de9499a4c198ded51a3e6712957a8e4.svg","isPro":false,"fullname":"Shaowei ZHang","user":"Shaowei0326","type":"user"},{"_id":"65e9781eb35f91eca2666491","avatarUrl":"/avatars/cdb9fe13c07e630f84ae8c7d45429510.svg","isPro":false,"fullname":"xinr chen","user":"chen459664","type":"user"},{"_id":"6884c66504403c53c976a5ba","avatarUrl":"/avatars/5937e916695b4821e0d8b7ec159117c4.svg","isPro":false,"fullname":"Dan Shi","user":"dansdfki","type":"user"},{"_id":"66253d6e535ce92a4ae0a8cb","avatarUrl":"/avatars/13bcb8de32af1f7a8767d411056b5d32.svg","isPro":false,"fullname":"lll","user":"llll555666","type":"user"},{"_id":"6963c7c504fb94ecfdbab02f","avatarUrl":"/avatars/a527dcde621be60ff9221d7645a0e440.svg","isPro":false,"fullname":"Jason","user":"pjasonx","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":3,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.02398.md"}">

Papers

arxiv:2606.02398

A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RL

Published on Jun 1

· Submitted by

Lei Yang on Jun 3

#3 Paper of the day

Upvote

Authors:

Lei Yang ,

Abstract

Multi-domain reinforcement learning in language models causes performance degradation through shared computational pathways, but targeted refresh and rollback techniques can selectively recover lost capabilities with minimal side effects.

Generated by Qwen/Qwen2.5-Coder-32B-Instruct

Reinforcement learning (RL) post-training improves large language models (LLMs) on individual domains such as mathematical reasoning, code generation, question answering, and creative writing (CW), but training on one domain often degrades performance on others. Existing explanations based on catastrophic forgetting or global gradient conflict are incomplete: substantial interference can occur even when full-model gradients are nearly orthogonal. We show that single-domain RL produces sparse, small-magnitude parameter edits with weak overlap among top-changed neurons, while different domains still share substantial active computation routes on which update directions determine whether they act synergistically or conflict. Guided by this observation, we prove under a local perturbation model of multi-domain RL that later-domain training harms an earlier domain mainly through a second-order damage term, which under the observed sparse route structure concentrates in a low-dimensional shared conflict subspace. Moreover, a short domain refresh contracts the harmful component on this subspace, enabling selective recovery with limited collateral damage. Consistent with the theory, a brief Re-Math refresh after Code rightarrow Math rightarrow QA rightarrow CW recovers Math from 57.66 to 66.04 while largely preserving performance on the other domains, yielding the best average score of 66.39. Beyond refresh, a training-free rollback on a sparse proxy conflict coordinate set for the Math-QA pair partially restores Math, providing direct proxy-level evidence for localized damage. These results provide a localized mechanistic account of interference and recovery in multi-domain RL.