Multimodal expansion of large language models (LLMs) enables new perceptual capabilities but often compromises the language intelligence acquired during pretraining. In this work, we investigate this phenomenon from the perspective of internal adaptation dynamics and discover that neurons in pretrained LLMs exhibit heterogeneous plasticity during multimodal learning: some neurons are critical for preserving language capabilities, while others are more adaptive to multimodal knowledge. Based on this insight, we propose NeuPAT (Neuron-aware Plasticity Allocation Tuning), a lightweight and architecture-agnostic framework that allocates neuron-wise update constraints during multimodal instruction tuning. NeuPAT uses a small-scale probing stage to estimate neuron adaptation patterns and selectively protects language-sensitive neurons while promoting multimodal adaptation through more plastic neurons. Experiments across diverse LLM families demonstrate that NeuPAT recovers 94.5% of the language capability degradation caused by vanilla tuning on 11 language benchmarks while maintaining comparable multimodal performance, providing an effective approach for capability-preserving multimodal expansion.</p>\n","updatedAt":"2026-08-13T03:01:17.090Z","author":{"_id":"691c08008411a45dc9ff4530","avatarUrl":"/avatars/c39c2c9818e96c00825bf12c9dbc912d.svg","fullname":"ltguo","name":"CASIA-IVAer","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8907302021980286},"editors":["CASIA-IVAer"],"editorAvatarUrls":["/avatars/c39c2c9818e96c00825bf12c9dbc912d.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.08107","authors":[{"_id":"6a7d337c0ac8bee77474eea8","name":"Jiayue Jin","hidden":false},{"_id":"6a7d337c0ac8bee77474eea9","name":"Jingwei Zhang","hidden":false},{"_id":"6a7d337c0ac8bee77474eeaa","name":"Chen Wang","hidden":false},{"_id":"6a7d337c0ac8bee77474eeab","name":"Jing Liu","hidden":false},{"_id":"6a7d337c0ac8bee77474eeac","name":"Longteng Guo","hidden":false}],"publishedAt":"2026-08-08T00:00:00.000Z","submittedOnDailyAt":"2026-08-13T00:00:00.000Z","title":"NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMs","submittedOnDailyBy":{"_id":"691c08008411a45dc9ff4530","avatarUrl":"/avatars/c39c2c9818e96c00825bf12c9dbc912d.svg","isPro":false,"fullname":"ltguo","user":"CASIA-IVAer","type":"user","name":"CASIA-IVAer"},"summary":"Multimodal expansion of large language models (LLMs) enables new perceptual capabilities but often compromises the language intelligence acquired during pretraining. In this work, we investigate this phenomenon from the perspective of internal adaptation dynamics and discover that neurons in pretrained LLMs exhibit heterogeneous plasticity during multimodal learning: some neurons are critical for preserving language capabilities, while others are more adaptive to multimodal knowledge. Based on this insight, we propose NeuPAT (Neuron-aware Plasticity Allocation Tuning), a lightweight and architecture-agnostic framework that allocates neuron-wise update constraints during multimodal instruction tuning. NeuPAT uses a small-scale probing stage to estimate neuron adaptation patterns and selectively protects language-sensitive neurons while promoting multimodal adaptation through more plastic neurons. Experiments across diverse LLM families demonstrate that NeuPAT recovers 94.5\\% of the language capability degradation caused by vanilla tuning on 11 language benchmarks while maintaining comparable multimodal performance, providing an effective approach for capability-preserving multimodal expansion.","upvotes":1,"discussionId":"6a7d337c0ac8bee77474eead","ai_summary":"NeuPAT selectively constrains updates to language-sensitive neurons during multimodal tuning to preserve LLM language capabilities while enabling perceptual adaptation.","ai_keywords":["large language models","multimodal expansion","neuron plasticity","NeuPAT","neuron-aware plasticity allocation tuning","instruction tuning","language-sensitive neurons"],"ai_summary_model":"thinkingmachines/Inkling-Small","organization":{"_id":"691c31f046a68e3d1d8dcfac","name":"CASIA-IVA-Lab","fullname":"CASIA-IVA-Lab","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/691c08008411a45dc9ff4530/Tf9QH8MysDlD_FgaJAf_t.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"691c08008411a45dc9ff4530","avatarUrl":"/avatars/c39c2c9818e96c00825bf12c9dbc912d.svg","isPro":false,"fullname":"ltguo","user":"CASIA-IVAer","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"691c31f046a68e3d1d8dcfac","name":"CASIA-IVA-Lab","fullname":"CASIA-IVA-Lab","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/691c08008411a45dc9ff4530/Tf9QH8MysDlD_FgaJAf_t.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.08107.md","query":{}}">
NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMs
Published on Aug 8
· Submitted by ltguo on Aug 13 Abstract
NeuPAT selectively constrains updates to language-sensitive neurons during multimodal tuning to preserve LLM language capabilities while enabling perceptual adaptation.
Multimodal expansion of large language models (LLMs) enables new perceptual capabilities but often compromises the language intelligence acquired during pretraining. In this work, we investigate this phenomenon from the perspective of internal adaptation dynamics and discover that neurons in pretrained LLMs exhibit heterogeneous plasticity during multimodal learning: some neurons are critical for preserving language capabilities, while others are more adaptive to multimodal knowledge. Based on this insight, we propose NeuPAT (Neuron-aware Plasticity Allocation Tuning), a lightweight and architecture-agnostic framework that allocates neuron-wise update constraints during multimodal instruction tuning. NeuPAT uses a small-scale probing stage to estimate neuron adaptation patterns and selectively protects language-sensitive neurons while promoting multimodal adaptation through more plastic neurons. Experiments across diverse LLM families demonstrate that NeuPAT recovers 94.5\% of the language capability degradation caused by vanilla tuning on 11 language benchmarks while maintaining comparable multimodal performance, providing an effective approach for capability-preserving multimodal expansion.
Community
Multimodal expansion of large language models (LLMs) enables new perceptual capabilities but often compromises the language intelligence acquired during pretraining. In this work, we investigate this phenomenon from the perspective of internal adaptation dynamics and discover that neurons in pretrained LLMs exhibit heterogeneous plasticity during multimodal learning: some neurons are critical for preserving language capabilities, while others are more adaptive to multimodal knowledge. Based on this insight, we propose NeuPAT (Neuron-aware Plasticity Allocation Tuning), a lightweight and architecture-agnostic framework that allocates neuron-wise update constraints during multimodal instruction tuning. NeuPAT uses a small-scale probing stage to estimate neuron adaptation patterns and selectively protects language-sensitive neurons while promoting multimodal adaptation through more plastic neurons. Experiments across diverse LLM families demonstrate that NeuPAT recovers 94.5% of the language capability degradation caused by vanilla tuning on 11 language benchmarks while maintaining comparable multimodal performance, providing an effective approach for capability-preserving multimodal expansion.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.08107 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.08107 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.08107 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.