SEATS is a training-free, stage-adaptive token selection method for efficient omni-modal LLM inference. By analyzing layer-wise token dependency, it reveals that visual and audio dependencies follow a block-wise pattern and weaken with depth. SEATS removes spatiotemporal redundancy before the LLM, progressively prunes tokens inside the LLM, and fully removes non-textual tokens in late layers.</p>\n","updatedAt":"2026-05-20T13:41:55.747Z","author":{"_id":"64ae1f92b575c5e272217ea3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64ae1f92b575c5e272217ea3/noAFNdaNW47QedntLO4N4.jpeg","fullname":"Zijie Xin","name":"xxayt","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7944574952125549},"editors":["xxayt"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/64ae1f92b575c5e272217ea3/noAFNdaNW47QedntLO4N4.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.20035","authors":[{"_id":"6a0d699d0cc88a0d483d370f","user":{"_id":"64ae1f92b575c5e272217ea3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64ae1f92b575c5e272217ea3/noAFNdaNW47QedntLO4N4.jpeg","isPro":false,"fullname":"Zijie Xin","user":"xxayt","type":"user","name":"xxayt"},"name":"Zijie Xin","status":"claimed_verified","statusLastChangedAt":"2026-05-20T17:10:02.195Z","hidden":false},{"_id":"6a0d699d0cc88a0d483d3710","name":"Jie Yang","hidden":false},{"_id":"6a0d699d0cc88a0d483d3711","name":"Ruixiang Zhao","hidden":false},{"_id":"6a0d699d0cc88a0d483d3712","name":"Tianyi Wang","hidden":false},{"_id":"6a0d699d0cc88a0d483d3713","name":"Fengyun Rao","hidden":false},{"_id":"6a0d699d0cc88a0d483d3714","name":"Jing Lyu","hidden":false},{"_id":"6a0d699d0cc88a0d483d3715","name":"Xirong Li","hidden":false}],"publishedAt":"2026-05-19T00:00:00.000Z","submittedOnDailyAt":"2026-05-20T00:00:00.000Z","title":"Stage-adaptive Token Selection for Efficient Omni-modal LLMs","submittedOnDailyBy":{"_id":"64ae1f92b575c5e272217ea3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64ae1f92b575c5e272217ea3/noAFNdaNW47QedntLO4N4.jpeg","isPro":false,"fullname":"Zijie Xin","user":"xxayt","type":"user","name":"xxayt"},"summary":"Omni-modal large language models (om-LLMs) achieve unified audio-visual understanding by encoding video and audio into temporally aligned token sequences interleaved at the window level. However, processing these dense non-textual tokens throughout the LLM incurs substantial computational overhead. Although training-free token selection can reduce this cost, existing methods either focus on visual-only inputs or prune om-LLM tokens only before the LLM with fixed per-modality ratios, failing to capture how cross-modal token importance evolves across layers. To address this limitation, we first analyze the layer-wise token dependency of om-LLMs. We find that visual and audio dependencies follow a block-wise pattern and gradually weaken with depth, indicating that many late-layer non-textual tokens become redundant after cross-modal fusion. Motivated by this observation, we propose SEATS, a training-free, stage-adaptive token selection method for efficient om-LLM inference. Before the LLM, SEATS removes spatiotemporal redundancy via attention-weighted diversity selection. Inside the LLM, it progressively prunes tokens across blocks and dynamically allocates the retention budget from temporal windows to modalities using query relevance scores. In late layers, it removes all remaining non-textual tokens once cross-modal fusion is complete. Experiments on Qwen2.5-Omni and Qwen3-Omni demonstrate that SEATS effectively improves inference efficiency. Retaining only 10% of visual and audio tokens, it achieves a 9.3x FLOPs reduction and a 4.8x prefill speedup while preserving 96.3% of the original performance.","upvotes":2,"discussionId":"6a0d699e0cc88a0d483d3716","projectPage":"https://xxayt.github.io/SEATS/","githubRepo":"https://github.com/xxayt/SEATS","githubRepoAddedBy":"user","ai_summary":"SEATS is a training-free, stage-adaptive token selection method that reduces computational overhead in om-LLMs by progressively pruning redundant visual and audio tokens during both pre-LLM and LLM stages.","ai_keywords":["om-LLMs","audio-visual understanding","temporally aligned token sequences","computational overhead","token selection","cross-modal token importance","layer-wise token dependency","attention-weighted diversity selection","query relevance scores","cross-modal fusion","FLOPs reduction","prefill speedup"],"githubStars":2},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64ae1f92b575c5e272217ea3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64ae1f92b575c5e272217ea3/noAFNdaNW47QedntLO4N4.jpeg","isPro":false,"fullname":"Zijie Xin","user":"xxayt","type":"user"},{"_id":"6270324ebecab9e2dcf245de","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6270324ebecab9e2dcf245de/cMbtWSasyNlYc9hvsEEzt.jpeg","isPro":false,"fullname":"Kye Gomez","user":"kye","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.20035.md"}">
Stage-adaptive Token Selection for Efficient Omni-modal LLMs
Abstract
SEATS is a training-free, stage-adaptive token selection method that reduces computational overhead in om-LLMs by progressively pruning redundant visual and audio tokens during both pre-LLM and LLM stages.
AI-generated summary
Omni-modal large language models (om-LLMs) achieve unified audio-visual understanding by encoding video and audio into temporally aligned token sequences interleaved at the window level. However, processing these dense non-textual tokens throughout the LLM incurs substantial computational overhead. Although training-free token selection can reduce this cost, existing methods either focus on visual-only inputs or prune om-LLM tokens only before the LLM with fixed per-modality ratios, failing to capture how cross-modal token importance evolves across layers. To address this limitation, we first analyze the layer-wise token dependency of om-LLMs. We find that visual and audio dependencies follow a block-wise pattern and gradually weaken with depth, indicating that many late-layer non-textual tokens become redundant after cross-modal fusion. Motivated by this observation, we propose SEATS, a training-free, stage-adaptive token selection method for efficient om-LLM inference. Before the LLM, SEATS removes spatiotemporal redundancy via attention-weighted diversity selection. Inside the LLM, it progressively prunes tokens across blocks and dynamically allocates the retention budget from temporal windows to modalities using query relevance scores. In late layers, it removes all remaining non-textual tokens once cross-modal fusion is complete. Experiments on Qwen2.5-Omni and Qwen3-Omni demonstrate that SEATS effectively improves inference efficiency. Retaining only 10% of visual and audio tokens, it achieves a 9.3x FLOPs reduction and a 4.8x prefill speedup while preserving 96.3% of the original performance.
Community
SEATS is a training-free, stage-adaptive token selection method for efficient omni-modal LLM inference. By analyzing layer-wise token dependency, it reveals that visual and audio dependencies follow a block-wise pattern and weaken with depth. SEATS removes spatiotemporal redundancy before the LLM, progressively prunes tokens inside the LLM, and fully removes non-textual tokens in late layers.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.20035 in a model README.md to link it from this page.
Cite arxiv.org/abs/2605.20035 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2605.20035 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.