Method for learning scaling matrices to aid SVD based LLM compression</p>\n","updatedAt":"2026-06-09T13:53:04.496Z","author":{"_id":"638f0f9ab0525fa370479467","avatarUrl":"/avatars/0ee239721747bf0b5d976bfceb393707.svg","fullname":"Ernests Lavrinovics","name":"ernlavr","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9082667827606201},"editors":["ernlavr"],"editorAvatarUrls":["/avatars/0ee239721747bf0b5d976bfceb393707.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.07098","authors":[{"_id":"6a27dcc66dde1c5ef75bd2c3","user":{"_id":"638f0f9ab0525fa370479467","avatarUrl":"/avatars/0ee239721747bf0b5d976bfceb393707.svg","isPro":false,"fullname":"Ernests Lavrinovics","user":"ernlavr","type":"user","name":"ernlavr"},"name":"Ernests Lavrinovics","status":"claimed_verified","statusLastChangedAt":"2026-06-09T12:40:35.952Z","hidden":false},{"_id":"6a27dcc66dde1c5ef75bd2c4","name":"Marco Letizia","hidden":false},{"_id":"6a27dcc66dde1c5ef75bd2c5","name":"Roy Janco","hidden":false},{"_id":"6a27dcc66dde1c5ef75bd2c6","name":"Shai Segal","hidden":false},{"_id":"6a27dcc66dde1c5ef75bd2c7","name":"Johannes Bjerva","hidden":false},{"_id":"6a27dcc66dde1c5ef75bd2c8","name":"Maurizio Pierini","hidden":false}],"publishedAt":"2026-06-05T09:48:58.000Z","submittedOnDailyAt":"2026-06-09T00:00:00.000Z","title":"SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices","submittedOnDailyBy":{"_id":"638f0f9ab0525fa370479467","avatarUrl":"/avatars/0ee239721747bf0b5d976bfceb393707.svg","isPro":false,"fullname":"Ernests Lavrinovics","user":"ernlavr","type":"user","name":"ernlavr"},"summary":"We present SigmaScale, a method for learning auxiliary scaling matrices S to aid truncated Singular Value Decomposition (SVD) based Large Language Model (LLM) compression. Instead of deriving scaling matrices analytically, SigmaScale optimizes two sets of vectors that define diagonal row and column scaling transformations under an activation-aware compression loss. We show that learned scaling lowers the effective intrinsic rank of weight matrices, as reflected by reductions in effective-rank entropy, and that this reduction is strongly correlated with compression loss. Experiments on Llama 3.1 8B Instruct and Qwen3-8B show that SigmaScale is competitive with closely related state-of-the-art SVD-based compression methods across perplexity and zero-shot benchmarks. By using learned activation-aware transformations, SigmaScale explores a more flexible route to low-rank LLM compression by adapting to the structure of individual model weights. The advantage observed in specific tasks makes our approach a valid option for applications requiring a reduced LLM-inference computing cost.","upvotes":1,"discussionId":"6a27dcc66dde1c5ef75bd2c9","githubRepo":"https://github.com/ernlavr/SigmaScale","githubRepoAddedBy":"user","ai_summary":"SigmaScale learns auxiliary scaling matrices to improve truncated SVD-based LLM compression by adapting to individual weight structures through activation-aware transformations.","ai_keywords":["Singular Value Decomposition","Large Language Model","compression","effective-rank entropy","activation-aware compression loss","truncated SVD","scaling matrices","low-rank compression"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":0,"organization":{"_id":"64f034dc8a4cf3e5e6b1e70c","name":"AalborgUniversitet","fullname":"Aalborg Universitet","avatar":"https://www.gravatar.com/avatar/6b151c1e8ffe200d75d2181561ba385b?d=retro&size=100"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"638f0f9ab0525fa370479467","avatarUrl":"/avatars/0ee239721747bf0b5d976bfceb393707.svg","isPro":false,"fullname":"Ernests Lavrinovics","user":"ernlavr","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"64f034dc8a4cf3e5e6b1e70c","name":"AalborgUniversitet","fullname":"Aalborg Universitet","avatar":"https://www.gravatar.com/avatar/6b151c1e8ffe200d75d2181561ba385b?d=retro&size=100"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.07098.md"}">
SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices
Abstract
SigmaScale learns auxiliary scaling matrices to improve truncated SVD-based LLM compression by adapting to individual weight structures through activation-aware transformations.
We present SigmaScale, a method for learning auxiliary scaling matrices S to aid truncated Singular Value Decomposition (SVD) based Large Language Model (LLM) compression. Instead of deriving scaling matrices analytically, SigmaScale optimizes two sets of vectors that define diagonal row and column scaling transformations under an activation-aware compression loss. We show that learned scaling lowers the effective intrinsic rank of weight matrices, as reflected by reductions in effective-rank entropy, and that this reduction is strongly correlated with compression loss. Experiments on Llama 3.1 8B Instruct and Qwen3-8B show that SigmaScale is competitive with closely related state-of-the-art SVD-based compression methods across perplexity and zero-shot benchmarks. By using learned activation-aware transformations, SigmaScale explores a more flexible route to low-rank LLM compression by adapting to the structure of individual model weights. The advantage observed in specific tasks makes our approach a valid option for applications requiring a reduced LLM-inference computing cost.
Community
Method for learning scaling matrices to aid SVD based LLM compression
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.07098 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.07098 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.