This paper presents attention-state memory, a lookup-based memory of precomputed attention states between prefix and query tokens. Code will be released soon.</p>\n","updatedAt":"2026-05-20T07:36:46.390Z","author":{"_id":"6305d75c5b87d4feaacbb816","avatarUrl":"/avatars/63c8dd5aff1a5ab0b1d3b396d8ddfab1.svg","fullname":"Yasuyuki Okoshi","name":"kusakana","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8956243395805359},"editors":["kusakana"],"editorAvatarUrls":["/avatars/63c8dd5aff1a5ab0b1d3b396d8ddfab1.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.18226","authors":[{"_id":"6a0d63ca0cc88a0d483d36d7","name":"Yasuyuki Okoshi","hidden":false},{"_id":"6a0d63ca0cc88a0d483d36d8","name":"Hao Mark Chen","hidden":false},{"_id":"6a0d63ca0cc88a0d483d36d9","name":"Guanxi Lu","hidden":false},{"_id":"6a0d63ca0cc88a0d483d36da","name":"Hongxiang Fan","hidden":false},{"_id":"6a0d63ca0cc88a0d483d36db","name":"Masato Motomura","hidden":false},{"_id":"6a0d63ca0cc88a0d483d36dc","name":"Daichi Fujiki","hidden":false}],"publishedAt":"2026-05-18T00:00:00.000Z","submittedOnDailyAt":"2026-05-20T00:00:00.000Z","title":"Context Memorization for Efficient Long Context Generation","submittedOnDailyBy":{"_id":"6305d75c5b87d4feaacbb816","avatarUrl":"/avatars/63c8dd5aff1a5ab0b1d3b396d8ddfab1.svg","isPro":false,"fullname":"Yasuyuki Okoshi","user":"kusakana","type":"user","name":"kusakana"},"summary":"Modern large language model (LLM) applications increasingly rely on long conditioning prefixes to control model behavior at inference time. While prefix-augmented inference is effective, it incurs two structural limitations: i) the prefix's influence fades as generation proceeds, and ii) attention computation over the prefix scales linearly with its length. Existing approaches either keep the prefix in attention while compressing it, or internalize it into model parameters through gradient-based training. The former still attends to the prefix at inference, while the latter is training-intensive and ill-suited to prefix updates. To address these issues, we propose attention-state memory, a training-free approach that externalizes the prefix into a lightweight, lookup-based memory of precomputed attention states between prefix and query tokens. On ManyICLBench with LLaMA-3.1-8B, our method improves accuracy over in-context learning at 1K-8K memory budgets while reducing attention latency by 1.36x at 8K, and surpasses full-attention RAG performance on NBA benchmark using only 20% of its memory footprint.","upvotes":2,"discussionId":"6a0d63ca0cc88a0d483d36dd","githubRepo":"https://github.com/yasu0001/AttentionMemory","githubRepoAddedBy":"user","ai_summary":"Attention-state memory enables efficient long-prefix inference by storing precomputed attention states in lightweight memory, improving accuracy and reducing latency compared to traditional methods.","ai_keywords":["prefix-augmented inference","attention computation","attention-state memory","lookup-based memory","precomputed attention states","in-context learning","full-attention RAG","ManyICLBench","LLaMA-3.1-8B","NBA benchmark"],"githubStars":0},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6305d75c5b87d4feaacbb816","avatarUrl":"/avatars/63c8dd5aff1a5ab0b1d3b396d8ddfab1.svg","isPro":false,"fullname":"Yasuyuki Okoshi","user":"kusakana","type":"user"},{"_id":"661ab1f1fa3b144a381fa454","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/661ab1f1fa3b144a381fa454/IlpZBb9NCjo7ntFwMIH53.png","isPro":true,"fullname":"Urro","user":"urroxyz","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.18226.md"}">
Context Memorization for Efficient Long Context Generation
Abstract
Attention-state memory enables efficient long-prefix inference by storing precomputed attention states in lightweight memory, improving accuracy and reducing latency compared to traditional methods.
AI-generated summary
Modern large language model (LLM) applications increasingly rely on long conditioning prefixes to control model behavior at inference time. While prefix-augmented inference is effective, it incurs two structural limitations: i) the prefix's influence fades as generation proceeds, and ii) attention computation over the prefix scales linearly with its length. Existing approaches either keep the prefix in attention while compressing it, or internalize it into model parameters through gradient-based training. The former still attends to the prefix at inference, while the latter is training-intensive and ill-suited to prefix updates. To address these issues, we propose attention-state memory, a training-free approach that externalizes the prefix into a lightweight, lookup-based memory of precomputed attention states between prefix and query tokens. On ManyICLBench with LLaMA-3.1-8B, our method improves accuracy over in-context learning at 1K-8K memory budgets while reducing attention latency by 1.36x at 8K, and surpasses full-attention RAG performance on NBA benchmark using only 20% of its memory footprint.
Community
This paper presents attention-state memory, a lookup-based memory of precomputed attention states between prefix and query tokens. Code will be released soon.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.18226 in a model README.md to link it from this page.
Cite arxiv.org/abs/2605.18226 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2605.18226 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.