Zero token video understanding for VLMs!</p>\n<p>We train hypernetworks to metalearn video understanding for VLMs; during inference you can convert a video to a LoRA adapter in a single forward pass, and have efficient video understanding without any visual tokens in the model's context. </p>\n<p>We show:<br>→ Up to 1500× reduction in answer-time visual-token load<br>→ 6–80× faster TTFT during queries<br>→ Statistically equivalent / non-inferior to direct video-in-context inference across most captioning and QA settings<br>→ At scales like 1024px and 1024 frames, where the base model often degenerates into gibberish, Video2LoRA-enabled VLMs still generate meaningfully relevant content</p>\n","updatedAt":"2026-06-05T02:16:58.261Z","author":{"_id":"62daf5ce9852869b34fdaf28","avatarUrl":"/avatars/116fac556c2d129f0699ececc8246084.svg","fullname":"Manan Suri","name":"MananSuri27","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8461301922798157},"editors":["MananSuri27"],"editorAvatarUrls":["/avatars/116fac556c2d129f0699ececc8246084.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.04351","authors":[{"_id":"6a222f0c3490a593e87b1420","name":"Manan Suri","hidden":false},{"_id":"6a222f0c3490a593e87b1421","name":"Sarvesh Baskar","hidden":false},{"_id":"6a222f0c3490a593e87b1422","name":"Dinesh Manocha","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/62daf5ce9852869b34fdaf28/v8vq4hiYloCOgda9wQTUq.gif"],"publishedAt":"2026-06-03T00:00:00.000Z","submittedOnDailyAt":"2026-06-05T00:00:00.000Z","title":"Video2LoRA: Parametric Video Internalization for Vision-Language Models","submittedOnDailyBy":{"_id":"62daf5ce9852869b34fdaf28","avatarUrl":"/avatars/116fac556c2d129f0699ececc8246084.svg","isPro":false,"fullname":"Manan Suri","user":"MananSuri27","type":"user","name":"MananSuri27"},"summary":"Processing video in vision-language models is expensive: each frame occupies hundreds of tokens, and inference cost scales with every frame and every repeated query. We introduce Video2LoRA, a method for parametric video internalization. A perceiver hypernetwork reads the intermediate representations produced layer-by-layer as a frozen VLM encodes a video, and generates a Low-Rank Adaptation (LoRA) adapter in a single forward pass. Unlike standard LoRA fine-tuning, which requires iterative gradient updates, Video2LoRA predicts these weights directly from the video. Trained for SmolVLM2 500M and 2.2B on video summarization and captioning, Video2LoRA enables the same frozen VLM to answer queries from the adapter alone, with zero visual tokens in its context at query time. Video2LoRA is statistically non-inferior and equivalent to direct video-in-context inference across all five captioning benchmarks at both model scales, and across seven of eight video question answering benchmark-scale pairings. Although trained only on 12 frames at 384px, it remains stable up to 1,024 frames and 1024px, where direct video-in-context inference often degenerates. Across this sweep, it reduces answer-time visual-token load by up to 1,500x and query TTFT by 6-80x, while preserving video-faithful outputs. We also find that independently generated adapters for non-overlapping video segments can compose in rank space, suggesting a path toward chunked long-video internalization.","upvotes":1,"discussionId":"6a222f0d3490a593e87b1423","projectPage":"https://video2lora.github.io/","githubRepo":"https://github.com/MananSuri27/video2lora","githubRepoAddedBy":"user","ai_summary":"Video2LoRA enables efficient video processing in vision-language models by predicting Low-Rank Adaptation weights from video representations, reducing computational costs while maintaining video-faithful outputs.","ai_keywords":["perceiver hypernetwork","intermediate representations","Low-Rank Adaptation","LoRA","video summarization","captioning","video question answering","visual tokens","query TTFT","rank space"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":1},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"615c4fc9ce94868478737e38","avatarUrl":"/avatars/66af6ea2113fe070cba6e3b48c153c3b.svg","isPro":false,"fullname":"Charlie","user":"Charliezyl","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.04351.md"}">
Video2LoRA: Parametric Video Internalization for Vision-Language Models
Abstract
Video2LoRA enables efficient video processing in vision-language models by predicting Low-Rank Adaptation weights from video representations, reducing computational costs while maintaining video-faithful outputs.
Processing video in vision-language models is expensive: each frame occupies hundreds of tokens, and inference cost scales with every frame and every repeated query. We introduce Video2LoRA, a method for parametric video internalization. A perceiver hypernetwork reads the intermediate representations produced layer-by-layer as a frozen VLM encodes a video, and generates a Low-Rank Adaptation (LoRA) adapter in a single forward pass. Unlike standard LoRA fine-tuning, which requires iterative gradient updates, Video2LoRA predicts these weights directly from the video. Trained for SmolVLM2 500M and 2.2B on video summarization and captioning, Video2LoRA enables the same frozen VLM to answer queries from the adapter alone, with zero visual tokens in its context at query time. Video2LoRA is statistically non-inferior and equivalent to direct video-in-context inference across all five captioning benchmarks at both model scales, and across seven of eight video question answering benchmark-scale pairings. Although trained only on 12 frames at 384px, it remains stable up to 1,024 frames and 1024px, where direct video-in-context inference often degenerates. Across this sweep, it reduces answer-time visual-token load by up to 1,500x and query TTFT by 6-80x, while preserving video-faithful outputs. We also find that independently generated adapters for non-overlapping video segments can compose in rank space, suggesting a path toward chunked long-video internalization.
Community
Zero token video understanding for VLMs!
We train hypernetworks to metalearn video understanding for VLMs; during inference you can convert a video to a LoRA adapter in a single forward pass, and have efficient video understanding without any visual tokens in the model's context.
We show:
→ Up to 1500× reduction in answer-time visual-token load
→ 6–80× faster TTFT during queries
→ Statistically equivalent / non-inferior to direct video-in-context inference across most captioning and QA settings
→ At scales like 1024px and 1024 frames, where the base model often degenerates into gibberish, Video2LoRA-enabled VLMs still generate meaningfully relevant content
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.04351 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.04351 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.