Hugging Face Daily Papers · August 10, 2026 · 3 min read

SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

This work investigates the fundamental differences between SFT and RL in multi-task reasoning. We find that SFT suffers from severe task conflicts under multi-stage training, while RL allows different task capabilities to coexist and improve stably. Through parameter-level and theoretical analyses, we show that RL induces smaller, sparser, and approximately orthogonal updates across tasks: SFT interference is norm-limited, whereas RL interference is variance-limited. Based on this insight, we further propose Parallel-RL, which decouples multi-task RL into parallel task-specific training.</p>\n","updatedAt":"2026-08-10T06:54:48.179Z","author":{"_id":"650fa426877b574970bc1f0c","avatarUrl":"/avatars/55945605b53f92f715f447aab0b1ee95.svg","fullname":"zkj","name":"JokerJan","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9032282829284668},"editors":["JokerJan"],"editorAvatarUrls":["/avatars/55945605b53f92f715f447aab0b1ee95.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.03573","authors":[{"_id":"6a7973858e9301703eaa5fa6","name":"Kejian Zhu","hidden":false},{"_id":"6a7973858e9301703eaa5fa7","name":"Zhuoran Jin","hidden":false},{"_id":"6a7973858e9301703eaa5fa8","name":"Shangqing Tu","hidden":false},{"_id":"6a7973858e9301703eaa5fa9","name":"Hongbang Yuan","hidden":false},{"_id":"6a7973858e9301703eaa5faa","name":"Yushi Bai","hidden":false},{"_id":"6a7973858e9301703eaa5fab","name":"Kang Liu","hidden":false},{"_id":"6a7973858e9301703eaa5fac","name":"Juanzi Li","hidden":false},{"_id":"6a7973858e9301703eaa5fad","name":"Jun Zhao","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/650fa426877b574970bc1f0c/7bWCu0bqVULTDlKiEKR_j.png"],"publishedAt":"2026-08-06T00:00:00.000Z","submittedOnDailyAt":"2026-08-10T00:00:00.000Z","title":"SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs","submittedOnDailyBy":{"_id":"650fa426877b574970bc1f0c","avatarUrl":"/avatars/55945605b53f92f715f447aab0b1ee95.svg","isPro":false,"fullname":"zkj","user":"JokerJan","type":"user","name":"JokerJan"},"summary":"Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in enhancing multi-task reasoning for large language models (LLMs). Our preliminary experiments revealed a phenomenon: SFT suffers from severe task conflicts under multi-stage training, whereas RL enables stable coexistence across diverse tasks. Empirically, we trace this to the parameter level, observing that RL induces sparse and approximately orthogonal updates across tasks. We provide a theoretical explanation for this mechanism by analyzing multi-task gradient interference. Our results reveal a distinction: interference in SFT is norm-limited, scaling with the absolute gradient magnitude, whereas interference in RL is variance-limited, bounded by the gradient variance induced by advantage normalization and on-policy optimization. This small variance bound yields near-orthogonal optimization directions across tasks. Leveraging this insight, we propose Parallel-RL, a paradigm that decouples multi-task training, significantly improving efficiency and flexibility.","upvotes":25,"discussionId":"6a7973858e9301703eaa5fae","githubRepo":"https://github.com/GaryStack/Parallel-RL","githubRepoAddedBy":"user","githubStars":5,"organization":{"_id":"640a887796aae649741a586f","name":"CASIA","fullname":"Chinese Academic of Science Institute of Automation","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1678411888885-6388984e8a5dbe2f3dc5afee.jpeg"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"650fa426877b574970bc1f0c","avatarUrl":"/avatars/55945605b53f92f715f447aab0b1ee95.svg","isPro":false,"fullname":"zkj","user":"JokerJan","type":"user"},{"_id":"6782344cabc09a34497d6475","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6782344cabc09a34497d6475/NTnU1Ko6ko5CgPeYlBrmf.jpeg","isPro":false,"fullname":"MMQ","user":"kiomio","type":"user"},{"_id":"643379416c6ecd58798421b3","avatarUrl":"/avatars/831db7eab2663abc33b176cf386b02f2.svg","isPro":false,"fullname":"Zhuoran Jin","user":"jinzhuoran","type":"user"},{"_id":"6772d6d23eee71eafc6a4fe4","avatarUrl":"/avatars/6b1aba467a20b5deee17ae90a03557a2.svg","isPro":false,"fullname":"ying1973","user":"ying1973","type":"user"},{"_id":"67d15f29bacfd19231a6e178","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/I0CxXy7s-Huy495htZlJC.png","isPro":false,"fullname":"Lu Wang","user":"wanglu666","type":"user"},{"_id":"67c6744ffaf82ef97dbe871d","avatarUrl":"/avatars/fe9f2c8bb3c98af3177a1170a6829913.svg","isPro":false,"fullname":"Longxiang Wang","user":"wlxxx","type":"user"},{"_id":"66027544a7a608d71163d24e","avatarUrl":"/avatars/3c168dd086f9f92c527d33615dc7f976.svg","isPro":false,"fullname":"stars shine","user":"starsshine","type":"user"},{"_id":"6538d3089d66a6c304ee2f4b","avatarUrl":"/avatars/6e93ea6c937128765e5323fb4a5d1cea.svg","isPro":false,"fullname":"HongbangYuan","user":"HongbangYuan","type":"user"},{"_id":"691e8129fdf62018ece3b64f","avatarUrl":"/avatars/2a1ad66699029437e313c5e08cbe2ef5.svg","isPro":false,"fullname":"double_x","user":"doublexxx","type":"user"},{"_id":"648c48d8c0ddeee6df5b6d22","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/648c48d8c0ddeee6df5b6d22/BlrYDv3eQxZ-Y5vtVGegX.jpeg","isPro":false,"fullname":"Shangqing Tu","user":"tsq2000","type":"user"},{"_id":"6307612bfd79b417f1bc3fa3","avatarUrl":"/avatars/e86ed202106c43d5ba65bc3ff1f0c1fd.svg","isPro":false,"fullname":"ricky_33","user":"ricky333","type":"user"},{"_id":"6555ca4b5891609e4558dd2a","avatarUrl":"/avatars/30395217fc2e6dbed8e261e1fa4883a6.svg","isPro":false,"fullname":"Tianyi Men","user":"MultimodalAgent","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":2,"organization":{"_id":"640a887796aae649741a586f","name":"CASIA","fullname":"Chinese Academic of Science Institute of Automation","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1678411888885-6388984e8a5dbe2f3dc5afee.jpeg"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.03573.md","query":{}}">

Papers

arxiv:2608.03573

SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs

Published on Aug 6

· Submitted by

zkj on Aug 10

#2 Paper of the day

Chinese Academic of Science Institute of Automation

Upvote

Authors:

Abstract

Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in enhancing multi-task reasoning for large language models (LLMs). Our preliminary experiments revealed a phenomenon: SFT suffers from severe task conflicts under multi-stage training, whereas RL enables stable coexistence across diverse tasks. Empirically, we trace this to the parameter level, observing that RL induces sparse and approximately orthogonal updates across tasks. We provide a theoretical explanation for this mechanism by analyzing multi-task gradient interference. Our results reveal a distinction: interference in SFT is norm-limited, scaling with the absolute gradient magnitude, whereas interference in RL is variance-limited, bounded by the gradient variance induced by advantage normalization and on-policy optimization. This small variance bound yields near-orthogonal optimization directions across tasks. Leveraging this insight, we propose Parallel-RL, a paradigm that decouples multi-task training, significantly improving efficiency and flexibility.

View arXiv page View PDF GitHub 5 Add to collection