Fine-tuning degrades what a model already knew. Omega-S is a drop-in penalty computed from the weight matrix alone: no previous-task data, no Fisher matrix, no stored copy of the old weights, three lines in the training loop, under 4% added cost per step.</p>\n<p>On Llama-3-8B with LoRA, code to prose, HumanEval over ten seeds: 84.1% retention against 62.9% unregularised, ahead on 9 of 10 seeds, 10 of 10 against tuned weight decay, 8 of 10 against tuned EWC.</p>\n<p>The mechanism section reports what we measured rather than what the name promises. Three of the four factors in the index are numerically inert at this scale; as implemented it reduces to a penalty on the variance of node degrees. The repository ships the per-seed data, the script to check that on your own model.</p>\n","updatedAt":"2026-08-11T17:23:19.774Z","author":{"_id":"6a57db532776471cfc65598b","avatarUrl":"/avatars/97d0181e626938fa811ca6c785bbd43b.svg","fullname":"Alberto Acedo","name":"Acedo","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9197069406509399},"editors":["Acedo"],"editorAvatarUrls":["/avatars/97d0181e626938fa811ca6c785bbd43b.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.03887","authors":[{"_id":"6a7b0b34b7183653340c1200","user":{"_id":"6a57db532776471cfc65598b","avatarUrl":"/avatars/97d0181e626938fa811ca6c785bbd43b.svg","isPro":false,"fullname":"Alberto Acedo","user":"Acedo","type":"user","name":"Acedo"},"name":"Alberto Acedo","status":"claimed_verified","statusLastChangedAt":"2026-08-11T16:45:04.714Z","hidden":false}],"publishedAt":"2026-08-04T00:00:00.000Z","submittedOnDailyAt":"2026-08-11T00:00:00.000Z","title":"Omega-S: A Functional Resilience Index for LLM Fine-Tuning","submittedOnDailyBy":{"_id":"6a57db532776471cfc65598b","avatarUrl":"/avatars/97d0181e626938fa811ca6c785bbd43b.svg","isPro":false,"fullname":"Alberto Acedo","user":"Acedo","type":"user","name":"Acedo"},"summary":"Fine-tuning a large language model on new data degrades what it previously learned. We present Omega-S, a drop-in penalty computed from the weight matrix alone: it needs no previous-task data, no Fisher matrix and no stored copy of the old weights. It is three lines in an existing training loop and adds under 4% to the cost of a step.\n Retention. On Llama-3-8B with LoRA, fine-tuned from code to prose and measured by HumanEval over ten seeds, Omega-S retains more of the original capability than no regularisation on 9 of 10 seeds (0.173 -> 0.238 absolute pass@1; sign test one-sided p=0.011, Wilcoxon p=0.006), as a retention ratio, 62.9% -> 84.1%. It also beats tuned weight decay on 10 of 10 seeds (p=0.002) and tuned EWC on 8 of 10 (p=0.014), every arm re-measured in the same session.\n Mechanism, measured rather than asserted. Omega-S is topological by construction, its objective built from Tr(A^3), but we measured which of its four factors actually moves and three do not: their elasticity with respect to the weights is at or below 1e-4, against 9e-3 for the degree-variance term. As implemented, the composite reduces to a penalty on the variance of node degrees, which means row magnitude in square modules and directional alignment in non-square ones. We report this because a method whose name promises one thing and whose gradient does another should say so. We also enumerate the open design choices, including a contrast-preserving construction that does what it was designed to do and makes retention worse on all ten seeds.\n Repeating an identical configuration, same seed and same hardware, gives a standard deviation of 0.104 in retention ratio. We have not found this quantified for low-rank fine-tuning of language models, and it bounds every seed-paired comparison in this literature, ours included.\n Code, per-seed results and the full record of negative results are available.","upvotes":1,"discussionId":"6a7b0b35b7183653340c1201","ai_summary":"Omega-S is a lightweight, data-free regularization penalty for low-rank fine-tuning that improves retention of original model capabilities by penalizing variance in weight-matrix node degrees.","ai_keywords":["LoRA","Fisher matrix","EWC","weight decay","Tr(A^3)","node degrees","low-rank fine-tuning","retention ratio","HumanEval","Llama-3-8B"],"ai_summary_model":"thinkingmachines/Inkling-Small"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.03887.md","query":{}}">
Omega-S: A Functional Resilience Index for LLM Fine-Tuning
Abstract
Omega-S is a lightweight, data-free regularization penalty for low-rank fine-tuning that improves retention of original model capabilities by penalizing variance in weight-matrix node degrees.
Fine-tuning a large language model on new data degrades what it previously learned. We present Omega-S, a drop-in penalty computed from the weight matrix alone: it needs no previous-task data, no Fisher matrix and no stored copy of the old weights. It is three lines in an existing training loop and adds under 4% to the cost of a step.
Retention. On Llama-3-8B with LoRA, fine-tuned from code to prose and measured by HumanEval over ten seeds, Omega-S retains more of the original capability than no regularisation on 9 of 10 seeds (0.173 -> 0.238 absolute pass@1; sign test one-sided p=0.011, Wilcoxon p=0.006), as a retention ratio, 62.9% -> 84.1%. It also beats tuned weight decay on 10 of 10 seeds (p=0.002) and tuned EWC on 8 of 10 (p=0.014), every arm re-measured in the same session.
Mechanism, measured rather than asserted. Omega-S is topological by construction, its objective built from Tr(A^3), but we measured which of its four factors actually moves and three do not: their elasticity with respect to the weights is at or below 1e-4, against 9e-3 for the degree-variance term. As implemented, the composite reduces to a penalty on the variance of node degrees, which means row magnitude in square modules and directional alignment in non-square ones. We report this because a method whose name promises one thing and whose gradient does another should say so. We also enumerate the open design choices, including a contrast-preserving construction that does what it was designed to do and makes retention worse on all ten seeds.
Repeating an identical configuration, same seed and same hardware, gives a standard deviation of 0.104 in retention ratio. We have not found this quantified for low-rank fine-tuning of language models, and it bounds every seed-paired comparison in this literature, ours included.
Code, per-seed results and the full record of negative results are available.
Community
Fine-tuning degrades what a model already knew. Omega-S is a drop-in penalty computed from the weight matrix alone: no previous-task data, no Fisher matrix, no stored copy of the old weights, three lines in the training loop, under 4% added cost per step.
On Llama-3-8B with LoRA, code to prose, HumanEval over ten seeds: 84.1% retention against 62.9% unregularised, ahead on 9 of 10 seeds, 10 of 10 against tuned weight decay, 8 of 10 against tuned EWC.
The mechanism section reports what we measured rather than what the name promises. Three of the four factors in the index are numerically inert at this scale; as implemented it reduces to a penalty on the variance of node degrees. The repository ships the per-seed data, the script to check that on your own model.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.03887 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.03887 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.03887 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.