We identify a previously overlooked failure mode of ALiBi positional encoding: its linear bias scaling underflows floating-point precision, which zeroes out a large fraction of attention weights and renders the affected attention heads partially blind. </p>\n<p>We analyze this failure mode, characterize its impact, and examine four mitigation strategies. We further demonstrate its occurrence in state-of-the-art pretrained models based on ALiBi. Comprehensive pretraining experiments with 148M-parameter decoder models help us to disentangle its effects from out-of-context degradation. We find that ALiBi's failure mode can substantially impair token retrieval while having only a minor effect on standard decoder benchmarks. We propose four training-time mitigation strategies and evaluate them individually and in combinations, finding that log-scaled distances yield the most consistent improvements in passkey retrieval. </p>\n<p>Despite this problem, default ALiBi slopes remain a surprisingly strong baseline, particularly for needle-in-a-haystack retrieval. Based on these findings we provide concrete recommendations on how to train models with ALiBi.</p>\n","updatedAt":"2026-08-05T11:56:12.707Z","author":{"_id":"625026749d39e8be3166132f","avatarUrl":"/avatars/f32291df2054c1bb4a01889d1b41c0d5.svg","fullname":"Christopher Schröder","name":"cschroeder","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":27,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8913861513137817},"editors":["cschroeder"],"editorAvatarUrls":["/avatars/f32291df2054c1bb4a01889d1b41c0d5.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.03994","authors":[{"_id":"6a72f8755e0a61a8ccd03bbf","name":"Christopher Schröder","hidden":false},{"_id":"6a72f8755e0a61a8ccd03bc0","name":"Lukas Gienapp","hidden":false},{"_id":"6a72f8755e0a61a8ccd03bc1","name":"Ferdinand Schlatt","hidden":false},{"_id":"6a72f8755e0a61a8ccd03bc2","name":"Martin Potthast","hidden":false},{"_id":"6a72f8755e0a61a8ccd03bc3","name":"Gerhard Heyer","hidden":false}],"publishedAt":"2026-08-04T00:00:00.000Z","submittedOnDailyAt":"2026-08-05T00:00:00.000Z","title":"When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings","submittedOnDailyBy":{"_id":"625026749d39e8be3166132f","avatarUrl":"/avatars/f32291df2054c1bb4a01889d1b41c0d5.svg","isPro":false,"fullname":"Christopher Schröder","user":"cschroeder","type":"user","name":"cschroeder"},"summary":"We identify a previously overlooked failure mode of ALiBi positional encoding: its linear bias scaling underflows floating-point precision, which zeroes out a large fraction of attention weights and renders the affected attention heads partially blind. We analyze this failure mode, characterize its impact, and examine four mitigation strategies. We further demonstrate its occurrence in state-of-the-art pretrained models based on ALiBi. Comprehensive pretraining experiments with 148M-parameter decoder models help us to disentangle its effects from out-of-context degradation. We find that ALiBi's failure mode can substantially impair token retrieval while having only a minor effect on standard decoder benchmarks. We propose four training-time mitigation strategies and evaluate them individually and in combinations, finding that log-scaled distances yield the most consistent improvements in passkey retrieval. Despite this problem, default ALiBi slopes remain a surprisingly strong baseline, particularly for needle-in-a-haystack retrieval. Based on these findings we provide concrete recommendations on how to train models with ALiBi.","upvotes":4,"discussionId":"6a72f8755e0a61a8ccd03bc4","projectPage":"https://coral-nlp.github.io/","organization":{"_id":"6819f7e39d37753b22820e10","name":"coral-nlp","fullname":"CORAL NLP Research","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/625026749d39e8be3166132f/MedDY5R5dpN_RYXpdMo4C.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"5e6a3d4ea9afd5125d9ec064","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1584020801691-noauth.jpeg","isPro":true,"fullname":"Stefan Schweter","user":"stefan-it","type":"user"},{"_id":"625026749d39e8be3166132f","avatarUrl":"/avatars/f32291df2054c1bb4a01889d1b41c0d5.svg","isPro":false,"fullname":"Christopher Schröder","user":"cschroeder","type":"user"},{"_id":"6270324ebecab9e2dcf245de","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6270324ebecab9e2dcf245de/cMbtWSasyNlYc9hvsEEzt.jpeg","isPro":false,"fullname":"Kye Gomez","user":"kye","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6819f7e39d37753b22820e10","name":"coral-nlp","fullname":"CORAL NLP Research","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/625026749d39e8be3166132f/MedDY5R5dpN_RYXpdMo4C.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.03994.md","query":{}}">
When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings
Abstract
We identify a previously overlooked failure mode of ALiBi positional encoding: its linear bias scaling underflows floating-point precision, which zeroes out a large fraction of attention weights and renders the affected attention heads partially blind. We analyze this failure mode, characterize its impact, and examine four mitigation strategies. We further demonstrate its occurrence in state-of-the-art pretrained models based on ALiBi. Comprehensive pretraining experiments with 148M-parameter decoder models help us to disentangle its effects from out-of-context degradation. We find that ALiBi's failure mode can substantially impair token retrieval while having only a minor effect on standard decoder benchmarks. We propose four training-time mitigation strategies and evaluate them individually and in combinations, finding that log-scaled distances yield the most consistent improvements in passkey retrieval. Despite this problem, default ALiBi slopes remain a surprisingly strong baseline, particularly for needle-in-a-haystack retrieval. Based on these findings we provide concrete recommendations on how to train models with ALiBi.
Community
We identify a previously overlooked failure mode of ALiBi positional encoding: its linear bias scaling underflows floating-point precision, which zeroes out a large fraction of attention weights and renders the affected attention heads partially blind.
We analyze this failure mode, characterize its impact, and examine four mitigation strategies. We further demonstrate its occurrence in state-of-the-art pretrained models based on ALiBi. Comprehensive pretraining experiments with 148M-parameter decoder models help us to disentangle its effects from out-of-context degradation. We find that ALiBi's failure mode can substantially impair token retrieval while having only a minor effect on standard decoder benchmarks. We propose four training-time mitigation strategies and evaluate them individually and in combinations, finding that log-scaled distances yield the most consistent improvements in passkey retrieval.
Despite this problem, default ALiBi slopes remain a surprisingly strong baseline, particularly for needle-in-a-haystack retrieval. Based on these findings we provide concrete recommendations on how to train models with ALiBi.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.03994 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.03994 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.03994 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.