To mitigate severe LLM capability forgetting while fine-tuning on the GenRetrieval task, weight averaging according to inter-model distance can retain performance.</p>\n","updatedAt":"2026-05-13T14:57:10.822Z","author":{"_id":"6138e0f027fc7c37930c7a12","avatarUrl":"/avatars/28f19853c7c365b5222caf8cf5297958.svg","fullname":"Neha Verma","name":"nverma","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8521483540534973},"editors":["nverma"],"editorAvatarUrls":["/avatars/28f19853c7c365b5222caf8cf5297958.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.12419","authors":[{"_id":"6a0490f6b1a8cbabc9f08455","name":"Neha Verma","hidden":false},{"_id":"6a0490f6b1a8cbabc9f08456","name":"Nikhil Mehta","hidden":false},{"_id":"6a0490f6b1a8cbabc9f08457","name":"Shao-Chuan Wang","hidden":false},{"_id":"6a0490f6b1a8cbabc9f08458","name":"Naijing Zhang","hidden":false},{"_id":"6a0490f6b1a8cbabc9f08459","name":"Alicia Tsai","hidden":false},{"_id":"6a0490f6b1a8cbabc9f0845a","name":"Li Wei","hidden":false},{"_id":"6a0490f6b1a8cbabc9f0845b","name":"Lukasz Heldt","hidden":false},{"_id":"6a0490f6b1a8cbabc9f0845c","name":"Lichan Hong","hidden":false},{"_id":"6a0490f6b1a8cbabc9f0845d","name":"Ed Chi","hidden":false},{"_id":"6a0490f6b1a8cbabc9f0845e","name":"Xinyang Yi","hidden":false}],"publishedAt":"2026-05-12T00:00:00.000Z","submittedOnDailyAt":"2026-05-13T00:00:00.000Z","title":"ORBIT: Preserving Foundational Language Capabilities in GenRetrieval via Origin-Regulated Merging","submittedOnDailyBy":{"_id":"6138e0f027fc7c37930c7a12","avatarUrl":"/avatars/28f19853c7c365b5222caf8cf5297958.svg","isPro":false,"fullname":"Neha Verma","user":"nverma","type":"user","name":"nverma"},"summary":"Despite the rapid advancements in large language model (LLM) development, fine-tuning them for specific tasks often results in the catastrophic forgetting of their general, language-based reasoning abilities. This work investigates and addresses this challenge in the context of the Generative Retrieval (GenRetrieval) task. During GenRetrieval fine-tuning, we find this forgetting occurs rapidly and correlates with the distance between the fine-tuned and original model parameters. Given these observations, we propose ORBIT, a novel approach that actively tracks the distance between fine-tuned and initial model weights, and uses a weight averaging strategy to constrain model drift during GenRetrieval fine-tuning when this inter-model distance exceeds a maximum threshold. Our results show that ORBIT retains substantial text and retrieval performance by outperforming both common continual learning baselines and related regularization methods that also employ weight averaging.","upvotes":1,"discussionId":"6a0490f7b1a8cbabc9f0845f","ai_summary":"ORBIT addresses catastrophic forgetting in large language model fine-tuning for generative retrieval by tracking parameter distances and employing weight averaging to maintain model performance.","ai_keywords":["large language model","catastrophic forgetting","fine-tuning","Generative Retrieval","weight averaging","parameter distance","model drift","continual learning"]},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6138e0f027fc7c37930c7a12","avatarUrl":"/avatars/28f19853c7c365b5222caf8cf5297958.svg","isPro":false,"fullname":"Neha Verma","user":"nverma","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.12419.md"}">
ORBIT: Preserving Foundational Language Capabilities in GenRetrieval via Origin-Regulated Merging
Authors: ,
,
,
,
,
,
,
,
,
Abstract
ORBIT addresses catastrophic forgetting in large language model fine-tuning for generative retrieval by tracking parameter distances and employing weight averaging to maintain model performance.
AI-generated summary
Despite the rapid advancements in large language model (LLM) development, fine-tuning them for specific tasks often results in the catastrophic forgetting of their general, language-based reasoning abilities. This work investigates and addresses this challenge in the context of the Generative Retrieval (GenRetrieval) task. During GenRetrieval fine-tuning, we find this forgetting occurs rapidly and correlates with the distance between the fine-tuned and original model parameters. Given these observations, we propose ORBIT, a novel approach that actively tracks the distance between fine-tuned and initial model weights, and uses a weight averaging strategy to constrain model drift during GenRetrieval fine-tuning when this inter-model distance exceeds a maximum threshold. Our results show that ORBIT retains substantial text and retrieval performance by outperforming both common continual learning baselines and related regularization methods that also employ weight averaging.
Community
To mitigate severe LLM capability forgetting while fine-tuning on the GenRetrieval task, weight averaging according to inter-model distance can retain performance.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.12419 in a model README.md to link it from this page.
Cite arxiv.org/abs/2605.12419 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2605.12419 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.