WriteSAE extends sparse autoencoders to the matrix-recurrent write site by making decoder atoms rank-1 outer products, matching the native k_t v_t^T cache update. The main results are 92.4% atom-substitution win rate over matched-norm ablation at Qwen3.5-0.8B L9 H4, R^2 = 0.98 closed-form logit-shift prediction, 88.1% transfer to Mamba-2-370M, and a sustained install that moves midrank target-in-continuation from 33.3% to 100%.</p>\n","updatedAt":"2026-05-14T02:17:30.342Z","author":{"_id":"69cc855d54b48932315785e7","avatarUrl":"/avatars/4382c03f336d9930ba863429cad125ea.svg","fullname":"Jack Young","name":"JackYoung27","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8278269171714783},"editors":["JackYoung27"],"editorAvatarUrls":["/avatars/4382c03f336d9930ba863429cad125ea.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.12770","authors":[{"_id":"6a052f0cb1a8cbabc9f086af","user":{"_id":"69cc855d54b48932315785e7","avatarUrl":"/avatars/4382c03f336d9930ba863429cad125ea.svg","isPro":false,"fullname":"Jack Young","user":"JackYoung27","type":"user","name":"JackYoung27"},"name":"Jack Young","status":"claimed_verified","statusLastChangedAt":"2026-05-14T10:55:49.949Z","hidden":false}],"publishedAt":"2026-05-12T00:00:00.000Z","submittedOnDailyAt":"2026-05-14T00:00:00.000Z","title":"WriteSAE: Sparse Autoencoders for Recurrent State","submittedOnDailyBy":{"_id":"69cc855d54b48932315785e7","avatarUrl":"/avatars/4382c03f336d9930ba863429cad125ea.svg","isPro":false,"fullname":"Jack Young","user":"JackYoung27","type":"user","name":"JackYoung27"},"summary":"We introduce WriteSAE, the first sparse autoencoder that decomposes and edits the matrix cache write of state-space and hybrid recurrent language models, where residual SAEs cannot reach. Existing SAEs read residual streams, but Gated DeltaNet, Mamba-2, and RWKV-7 write to a d_k times d_v cache through rank-1 updates k_t v_t^top that no vector atom can replace. WriteSAE factors each decoder atom into the native write shape, exposes a closed form for the per-token logit shift, and trains under matched Frobenius norm so atoms swap one cache slot at a time. Atom substitution beats matched-norm ablation on 92.4% of n=4{,}851 firings at Qwen3.5-0.8B L9 H4, the 87-atom population test holds at 89.8%, the closed form predicts measured effects at R^2=0.98, and Mamba-2-370M substitutes at 88.1% over 2,500 firings. Sustained three-position installs at 3times lift midrank target-in-continuation from 33.3% to 100% under greedy decoding, the first behavioral install at the matrix-recurrent write site.","upvotes":0,"discussionId":"6a052f0cb1a8cbabc9f086b0","projectPage":"https://www.jackyoung.io/research/writesae","githubRepo":"https://github.com/JackYoung27/writesae","githubRepoAddedBy":"user","ai_summary":"WriteSAE enables sparse autoencoder decomposition and editing of matrix cache writes in state-space and hybrid recurrent language models, achieving superior performance in token-level interventions compared to existing methods.","ai_keywords":["sparse autoencoder","matrix cache write","state-space models","hybrid recurrent language models","residual SAEs","Gated DeltaNet","Mamba-2","RWKV-7","rank-1 updates","Frobenius norm","atom substitution","greedy decoding"],"githubStars":0},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[],"acceptLanguages":["en"],"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.12770.md"}">
WriteSAE: Sparse Autoencoders for Recurrent State
Abstract
WriteSAE enables sparse autoencoder decomposition and editing of matrix cache writes in state-space and hybrid recurrent language models, achieving superior performance in token-level interventions compared to existing methods.
AI-generated summary
We introduce WriteSAE, the first sparse autoencoder that decomposes and edits the matrix cache write of state-space and hybrid recurrent language models, where residual SAEs cannot reach. Existing SAEs read residual streams, but Gated DeltaNet, Mamba-2, and RWKV-7 write to a d_k times d_v cache through rank-1 updates k_t v_t^top that no vector atom can replace. WriteSAE factors each decoder atom into the native write shape, exposes a closed form for the per-token logit shift, and trains under matched Frobenius norm so atoms swap one cache slot at a time. Atom substitution beats matched-norm ablation on 92.4% of n=4{,}851 firings at Qwen3.5-0.8B L9 H4, the 87-atom population test holds at 89.8%, the closed form predicts measured effects at R^2=0.98, and Mamba-2-370M substitutes at 88.1% over 2,500 firings. Sustained three-position installs at 3times lift midrank target-in-continuation from 33.3% to 100% under greedy decoding, the first behavioral install at the matrix-recurrent write site.
Community
WriteSAE extends sparse autoencoders to the matrix-recurrent write site by making decoder atoms rank-1 outer products, matching the native k_t v_t^T cache update. The main results are 92.4% atom-substitution win rate over matched-norm ablation at Qwen3.5-0.8B L9 H4, R^2 = 0.98 closed-form logit-shift prediction, 88.1% transfer to Mamba-2-370M, and a sustained install that moves midrank target-in-continuation from 33.3% to 100%.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.12770 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2605.12770 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.