Existing approaches for improving the efficiency of Large Vision–Language Models (LVLMs) primarily focus on reducing visual tokens or adopting smaller language models, while largely overlooking the vision encoder, which often remains a monolithic and computationally expensive component. In this work, we address this limitation by introducing UltraViT, a vision encoder explicitly designed and optimized for low-latency deployment on resource-constrained edge devices. Specifically, UltraViT uses real, on-device latency measurements to guide the systematic design of a pyramidal architecture that strategically integrates and adapts heterogeneous spatial mixers at the macro-block level. To effectively pre-train this efficient architecture, we further propose a two-stage generative pre-training strategy: dense distillation first captures rich spatial representations, followed by direct generative supervision from a frozen, capacity-mixed LLM to improve high-level semantic grounding. Compared with standard contrastive and self-supervised learning objectives, this strategy better prepares UltraViT for the generative, multimodal alignment required in LVLM training. Extensive experiments demonstrate that combining latency-aware architecture design with tailored generative pre-training establishes a new state of the art among efficient LVLM vision encoders, significantly outperforming existing encoder-centric baselines while achieving nearly 1.7× faster on-device inference.</p>\n","updatedAt":"2026-07-28T16:03:50.075Z","author":{"_id":"61eeb1121e886efb9a8f57e3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1643032839402-noauth.jpeg","fullname":"Alberto Baldrati","name":"ABaldrati","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8693923354148865},"editors":["ABaldrati"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1643032839402-noauth.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.23373","authors":[{"_id":"6a68d2b65dad3683ba934958","name":"Ioannis Maniadis Metaxas","hidden":false},{"_id":"6a68d2b65dad3683ba934959","name":"Adrian Bulat","hidden":false},{"_id":"6a68d2b65dad3683ba93495a","user":{"_id":"61eeb1121e886efb9a8f57e3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1643032839402-noauth.jpeg","isPro":false,"fullname":"Alberto Baldrati","user":"ABaldrati","type":"user","name":"ABaldrati"},"name":"Alberto Baldrati","status":"claimed_verified","statusLastChangedAt":"2026-07-28T16:45:04.695Z","hidden":false},{"_id":"6a68d2b65dad3683ba93495b","name":"Anestis Zaganidis","hidden":false},{"_id":"6a68d2b65dad3683ba93495c","name":"Yassine Ouali","hidden":false},{"_id":"6a68d2b65dad3683ba93495d","name":"Hyeonuk Kim","hidden":false},{"_id":"6a68d2b65dad3683ba93495e","name":"Georgios Tzimiropoulos","hidden":false}],"publishedAt":"2026-07-25T00:00:00.000Z","submittedOnDailyAt":"2026-07-28T00:00:00.000Z","title":"UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models","submittedOnDailyBy":{"_id":"61eeb1121e886efb9a8f57e3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1643032839402-noauth.jpeg","isPro":false,"fullname":"Alberto Baldrati","user":"ABaldrati","type":"user","name":"ABaldrati"},"summary":"Large Vision-Language Models (LVLMs) remain bottlenecked by massive computational footprints, precluding their deployment on resource-constrained edge devices. While efforts to compress LVLMs focus heavily on vision token reduction or smaller language models, the vision encoder is largely overlooked, typically deployed as a monolithic, computationally heavy feature extractor. Moreover, there is no previous effort that designs a vision encoder for LVLMs directly optimized for on-device latency. In this paper, we present UltraViT, a vision encoder for LVLMs, explicitly designed and optimized for on-device performance. Specifically, by taking into account real on-device latencies, we systematically design a pyramidal architecture that strategically integrates and adapts heterogeneous spatial mixers at the macro-block level. Furthermore, to pre-train UltraViT, we propose a novel two-stage generative pre-training strategy: cultivating rich spatial features via dense distillation, followed by direct generative supervision from a capacity-mixed frozen LLM. Compared to standard contrastive and SSL, we show that our pre-training is much more effective for achieving high-level semantic grounding for UltraViT needed for the subsequent generative multimodal alignment of LVLM training. Extensive experiments demonstrate that our on-device latency-informed design combined with our tailored training strategy establishes a new state-of-the-art for efficient LVLM encoding, significantly outperforming existing encoder-centric baselines while operating on-device at nearly 1.7xthe speed.","upvotes":1,"discussionId":"6a68d2b75dad3683ba93495f","organization":{"_id":"686df54910a52f2c2cf03c06","name":"SamsungResearch","fullname":"Samsung Research","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/60ffc3e62403168abcae811d/lBrkzrpjrJ8k-3CGLKRLr.jpeg"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"61eeb1121e886efb9a8f57e3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1643032839402-noauth.jpeg","isPro":false,"fullname":"Alberto Baldrati","user":"ABaldrati","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"686df54910a52f2c2cf03c06","name":"SamsungResearch","fullname":"Samsung Research","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/60ffc3e62403168abcae811d/lBrkzrpjrJ8k-3CGLKRLr.jpeg"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.23373.md","query":{}}">
UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models
Abstract
Large Vision-Language Models (LVLMs) remain bottlenecked by massive computational footprints, precluding their deployment on resource-constrained edge devices. While efforts to compress LVLMs focus heavily on vision token reduction or smaller language models, the vision encoder is largely overlooked, typically deployed as a monolithic, computationally heavy feature extractor. Moreover, there is no previous effort that designs a vision encoder for LVLMs directly optimized for on-device latency. In this paper, we present UltraViT, a vision encoder for LVLMs, explicitly designed and optimized for on-device performance. Specifically, by taking into account real on-device latencies, we systematically design a pyramidal architecture that strategically integrates and adapts heterogeneous spatial mixers at the macro-block level. Furthermore, to pre-train UltraViT, we propose a novel two-stage generative pre-training strategy: cultivating rich spatial features via dense distillation, followed by direct generative supervision from a capacity-mixed frozen LLM. Compared to standard contrastive and SSL, we show that our pre-training is much more effective for achieving high-level semantic grounding for UltraViT needed for the subsequent generative multimodal alignment of LVLM training. Extensive experiments demonstrate that our on-device latency-informed design combined with our tailored training strategy establishes a new state-of-the-art for efficient LVLM encoding, significantly outperforming existing encoder-centric baselines while operating on-device at nearly 1.7xthe speed.
Community
Existing approaches for improving the efficiency of Large Vision–Language Models (LVLMs) primarily focus on reducing visual tokens or adopting smaller language models, while largely overlooking the vision encoder, which often remains a monolithic and computationally expensive component. In this work, we address this limitation by introducing UltraViT, a vision encoder explicitly designed and optimized for low-latency deployment on resource-constrained edge devices. Specifically, UltraViT uses real, on-device latency measurements to guide the systematic design of a pyramidal architecture that strategically integrates and adapts heterogeneous spatial mixers at the macro-block level. To effectively pre-train this efficient architecture, we further propose a two-stage generative pre-training strategy: dense distillation first captures rich spatial representations, followed by direct generative supervision from a frozen, capacity-mixed LLM to improve high-level semantic grounding. Compared with standard contrastive and self-supervised learning objectives, this strategy better prepares UltraViT for the generative, multimodal alignment required in LVLM training. Extensive experiments demonstrate that combining latency-aware architecture design with tailored generative pre-training establishes a new state of the art among efficient LVLM vision encoders, significantly outperforming existing encoder-centric baselines while achieving nearly 1.7× faster on-device inference.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2607.23373 in a model README.md to link it from this page.
Cite arxiv.org/abs/2607.23373 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2607.23373 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.