We introduce <strong>The Shape of Addition</strong>, a mechanistic interpretability study of why LLMs can still fail at basic multi-operand addition.</p>\n<p>By probing residual-stream activations at each generated digit, we find that arithmetic states are organized into <strong>Iso-Raw-Sum Trajectories (IRSTs)</strong>: continuous raw-sum fibers passing through digit basins and further stratified by carry states. This geometry explains common off-by-one arithmetic errors as <strong>geometric slippages</strong>, where noisy latent carry representations cross quantization thresholds before discrete token output.</p>\n<p>We further propose a <strong>Noisy Quantization Model</strong> to characterize these failures, and validate the framework with a dual-stream consistency check that can detect and correct some quantization errors during inference. The results suggest that LLMs may internally retain correct arithmetic components even when the final token prediction is wrong.</p>\n","updatedAt":"2026-06-05T15:15:52.306Z","author":{"_id":"67e223b8a62bd61520a20763","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/46OmLBz_sQRq5ai1Jdx3I.png","fullname":"Liuyuan Wen","name":"FlushWen","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8923778533935547},"editors":["FlushWen"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/46OmLBz_sQRq5ai1Jdx3I.png"],"reactions":[{"reaction":"🔥","users":["FlushWen"],"count":1}],"isReport":false}},{"id":"6a237b739c9c7a503b9a6a2f","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":362,"isUserFollowing":false},"createdAt":"2026-06-06T01:44:19.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Reasoning on the Manifold: Bidirectional Consistency for Self-Verification in Diffusion Language Models](https://huggingface.co/papers/2604.16565) (2026)\n* [Causal Probing for Internal Visual Representations in Multimodal Large Language Models](https://huggingface.co/papers/2605.05593) (2026)\n* [Unveiling the Visual Counting Bottleneck in Vision-Language Models](https://huggingface.co/papers/2605.30170) (2026)\n* [Reading Calibrated Uncertainty from Language Model Trajectories](https://huggingface.co/papers/2605.22864) (2026)\n* [When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models](https://huggingface.co/papers/2605.08245) (2026)\n* [The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations](https://huggingface.co/papers/2605.09195) (2026)\n* [Transformers Linearly Represent Highly Structured World Models](https://huggingface.co/papers/2605.18847) (2026)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"<p>This is an automated message from the <a href=\"https://huggingface.co/librarian-bots\">Librarian Bot</a>. I found the following papers similar to this paper. </p>\n<p>The following papers were recommended by the Semantic Scholar API </p>\n<ul>\n<li><a href=\"https://huggingface.co/papers/2604.16565\">Reasoning on the Manifold: Bidirectional Consistency for Self-Verification in Diffusion Language Models</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.05593\">Causal Probing for Internal Visual Representations in Multimodal Large Language Models</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.30170\">Unveiling the Visual Counting Bottleneck in Vision-Language Models</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.22864\">Reading Calibrated Uncertainty from Language Model Trajectories</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.08245\">When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.09195\">The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.18847\">Transformers Linearly Represent Highly Structured World Models</a> (2026)</li>\n</ul>\n<p> Please give a thumbs up to this comment if you found it helpful!</p>\n<p> If you want recommendations for any Paper on Hugging Face checkout <a href=\"https://huggingface.co/spaces/librarian-bots/recommend_similar_papers\">this</a> Space</p>\n<p> You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: <code><span class=\"SVELTE_PARTIAL_HYDRATER contents\" data-target=\"UserMention\" data-props=\"{&quot;user&quot;:&quot;librarian-bot&quot;}\"><span class=\"inline-block\"><span class=\"contents\"><a href=\"/librarian-bot\">@<span class=\"underline\">librarian-bot</span></a></span> </span></span> recommend</code></p>\n","updatedAt":"2026-06-06T01:44:19.146Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":362,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7514756321907043},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.03645","authors":[{"_id":"6a21a0543490a593e87b10cc","user":{"_id":"67e223b8a62bd61520a20763","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/46OmLBz_sQRq5ai1Jdx3I.png","isPro":false,"fullname":"Liuyuan Wen","user":"FlushWen","type":"user","name":"FlushWen"},"name":"Liuyuan Wen","status":"claimed_verified","statusLastChangedAt":"2026-06-05T15:07:51.576Z","hidden":false},{"_id":"6a21a0543490a593e87b10cd","name":"Xun Zhu","hidden":false},{"_id":"6a21a0543490a593e87b10ce","name":"Lihao Huang","hidden":false},{"_id":"6a21a0543490a593e87b10cf","name":"Wenbin Li","hidden":false},{"_id":"6a21a0543490a593e87b10d0","name":"Yang Gao","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/67e223b8a62bd61520a20763/w1VGFO_FkiZArtJE3RBb6.png","https://cdn-uploads.huggingface.co/production/uploads/67e223b8a62bd61520a20763/p8jlkzOa-XcEMGRjjZXgP.png"],"publishedAt":"2026-05-29T00:00:00.000Z","submittedOnDailyAt":"2026-06-05T00:00:00.000Z","title":"The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models","submittedOnDailyBy":{"_id":"67e223b8a62bd61520a20763","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/46OmLBz_sQRq5ai1Jdx3I.png","isPro":false,"fullname":"Liuyuan Wen","user":"FlushWen","type":"user","name":"FlushWen"},"summary":"Large Language Models exhibit paradoxical fragility in fundamental arithmetic, implying a disconnect between internal computation and discrete output. By analyzing the residual stream geometry during multi-operand addition, we identify the Iso-Raw-Sum Trajectory (IRST), a geometric structure where representations are anchored by semantic digits and modulated by continuous carry fibers. We propose the Noisy Quantization Model to explain this geometry, framing arithmetic errors as Geometric Slippages caused by internal neural noise pushing a continuous, latent Carry Potential across quantization thresholds. This geometric framework further elucidates Probe Versatility, explaining how lightweight probes can disentangle coexisting latent signals (such as ground truth versus hallucination) from a single activation vector. Finally, we validate these insights through a geometric consistency check method that effectively detects and corrects these quantization failures during inference. Our code is available at https://github.com/RL-MIND/Shape-of-Addition.","upvotes":2,"discussionId":"6a21a0543490a593e87b10d1","githubRepo":"https://github.com/RL-MIND/Shape-of-Addition","githubRepoAddedBy":"user","ai_summary":"Large language models show arithmetic fragility due to geometric structures in residual streams, where neural noise causes quantization failures that can be detected and corrected through geometric analysis.","ai_keywords":["residual stream","Iso-Raw-Sum Trajectory","Noisy Quantization Model","Geometric Slippages","Carry Potential","probe versatility","geometric consistency check","quantization failures"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":2,"organization":{"_id":"638f70e8f1256a80d4288555","name":"nanjinguniv","fullname":"Nanjing University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/638f706ef1256a80d42880f9/6M6-JzwJGiLxjIJzvCflf.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"67e223b8a62bd61520a20763","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/46OmLBz_sQRq5ai1Jdx3I.png","isPro":false,"fullname":"Liuyuan Wen","user":"FlushWen","type":"user"},{"_id":"68b53b361c0b4b3b65d62afe","avatarUrl":"/avatars/54e4b780d3040d6dd7c81144a32e35b8.svg","isPro":false,"fullname":"Zhu Xun","user":"TheKiteRunner","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"638f70e8f1256a80d4288555","name":"nanjinguniv","fullname":"Nanjing University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/638f706ef1256a80d42880f9/6M6-JzwJGiLxjIJzvCflf.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.03645.md"}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2606.03645 in a model README.md to link it from this page.

Cite arxiv.org/abs/2606.03645 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2606.03645 in a Space README.md to link it from this page.