DRIFT combines a base predictor, which provides a coarse estimate of the target output, with a generative refinement module based on flow matching that iteratively improves the prediction. This residual formulation transforms the generative modeling problem from learning a global output distribution to modeling a localized residual distribution around a strong prior, substantially simplifying optimization. </p>\n<p>We evaluate DRIFT on both perception and planning tasks, including visual grounding and robotic control. Across multiple tasks and architectures spanning Multi-modal Large Language Models, Vision-Language-Action Models, and World Action Models, DRIFT consistently outperforms a strong set of regression- and generative-based solutions.</p>\n","updatedAt":"2026-06-11T15:28:15.608Z","author":{"_id":"666f2534a119281ee0e39783","avatarUrl":"/avatars/1de737da5cdf68d9bb1f98db8d5ad8b9.svg","fullname":"Zhuoming Liu","name":"zhuomingliu","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8416462540626526},"editors":["zhuomingliu"],"editorAvatarUrls":["/avatars/1de737da5cdf68d9bb1f98db8d5ad8b9.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.05758","authors":[{"_id":"6a28beefe7d78ea7587e5278","user":{"_id":"666f2534a119281ee0e39783","avatarUrl":"/avatars/1de737da5cdf68d9bb1f98db8d5ad8b9.svg","isPro":false,"fullname":"Zhuoming Liu","user":"zhuomingliu","type":"user","name":"zhuomingliu"},"name":"Zhuoming Liu","status":"claimed_verified","statusLastChangedAt":"2026-06-11T08:44:25.691Z","hidden":false},{"_id":"6a28beefe7d78ea7587e5279","name":"Jinhong Lin","hidden":false},{"_id":"6a28beefe7d78ea7587e527a","name":"Kwan Man Cheng","hidden":false},{"_id":"6a28beefe7d78ea7587e527b","name":"Lin Zhang","hidden":false},{"_id":"6a28beefe7d78ea7587e527c","name":"Shayok Bagchi","hidden":false},{"_id":"6a28beefe7d78ea7587e527d","name":"Yin Li","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/666f2534a119281ee0e39783/_LbOFURW_GPIhWRKams8a.png","https://cdn-uploads.huggingface.co/production/uploads/666f2534a119281ee0e39783/Df6ZqGZAcdYFcWIMgalH2.mp4","https://cdn-uploads.huggingface.co/production/uploads/666f2534a119281ee0e39783/Hnr8EWn9amiWlbjx2GDtD.mp4","https://cdn-uploads.huggingface.co/production/uploads/666f2534a119281ee0e39783/F2aoaiiSQMkxvCiBhC-NJ.mp4","https://cdn-uploads.huggingface.co/production/uploads/666f2534a119281ee0e39783/wZNR3M-aieEkoA61jrYYV.mp4","https://cdn-uploads.huggingface.co/production/uploads/666f2534a119281ee0e39783/TQvpC3p4I7GVu6Ye0jBcm.png"],"publishedAt":"2026-06-04T00:00:00.000Z","submittedOnDailyAt":"2026-06-11T00:00:00.000Z","title":"DRIFT: A Residual Flow Adapter for Decoding Continuous Outputs in Vision-Language Models","submittedOnDailyBy":{"_id":"666f2534a119281ee0e39783","avatarUrl":"/avatars/1de737da5cdf68d9bb1f98db8d5ad8b9.svg","isPro":false,"fullname":"Zhuoming Liu","user":"zhuomingliu","type":"user","name":"zhuomingliu"},"summary":"Many modern vision-language models (VLMs) build on autoregressive decoding of discrete tokens. While text-based output interfaces enable scalable pretraining and strong zero-shot generalization across diverse tasks, they are poorly suited for problems that require precise continuous outputs, such as localizing temporal boundaries of events or generating robotic control actions. To address this challenge, we propose DRIFT, a general framework for adapting pretrained VLMs to continuous decoding tasks. DRIFT combines a base predictor, which provides a coarse estimate of the target output, with a generative refinement module based on flow matching that iteratively improves the prediction. This residual formulation transforms the generative modeling problem from learning a global output distribution to modeling a localized residual distribution around a strong prior, substantially simplifying optimization. We evaluate DRIFT on both perception and planning tasks, including visual grounding and robotic control. Across multiple tasks and architectures spanning MLLMs, VLAs, and WAMs, DRIFT consistently outperforms a strong set of regression- and generative-based solutions.","upvotes":2,"discussionId":"6a28bef0e7d78ea7587e527e","projectPage":"https://dragonlzm.github.io/DRIFT/","githubRepo":"https://github.com/dragonlzm/DRIFT","githubRepoAddedBy":"user","ai_summary":"DRIFT is a framework that adapts pretrained vision-language models for continuous decoding tasks by combining coarse prediction with iterative refinement through flow matching, improving performance across perception and planning tasks.","ai_keywords":["vision-language models","autoregressive decoding","discrete tokens","continuous outputs","visual grounding","robotic control","base predictor","generative refinement module","flow matching","residual formulation","global output distribution","localized residual distribution"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":0,"organization":{"_id":"61d090ec03bc10eb8e1c2970","name":"uw-madison","fullname":"University of Wisconsin - Madison","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/68e396f2b5bb631e9b2fac9a/IYmUaLUc_rDVNC6F7-k8M.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"666f2534a119281ee0e39783","avatarUrl":"/avatars/1de737da5cdf68d9bb1f98db8d5ad8b9.svg","isPro":false,"fullname":"Zhuoming Liu","user":"zhuomingliu","type":"user"},{"_id":"6a2ae6c2e36bc84d91b6e7cc","avatarUrl":"/avatars/abf4b4c0020f9332b6827952cc53163e.svg","isPro":false,"fullname":"mmgood","user":"mmgood","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"61d090ec03bc10eb8e1c2970","name":"uw-madison","fullname":"University of Wisconsin - Madison","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/68e396f2b5bb631e9b2fac9a/IYmUaLUc_rDVNC6F7-k8M.png"}}">
DRIFT: A Residual Flow Adapter for Decoding Continuous Outputs in Vision-Language Models
Abstract
DRIFT is a framework that adapts pretrained vision-language models for continuous decoding tasks by combining coarse prediction with iterative refinement through flow matching, improving performance across perception and planning tasks.
Many modern vision-language models (VLMs) build on autoregressive decoding of discrete tokens. While text-based output interfaces enable scalable pretraining and strong zero-shot generalization across diverse tasks, they are poorly suited for problems that require precise continuous outputs, such as localizing temporal boundaries of events or generating robotic control actions. To address this challenge, we propose DRIFT, a general framework for adapting pretrained VLMs to continuous decoding tasks. DRIFT combines a base predictor, which provides a coarse estimate of the target output, with a generative refinement module based on flow matching that iteratively improves the prediction. This residual formulation transforms the generative modeling problem from learning a global output distribution to modeling a localized residual distribution around a strong prior, substantially simplifying optimization. We evaluate DRIFT on both perception and planning tasks, including visual grounding and robotic control. Across multiple tasks and architectures spanning MLLMs, VLAs, and WAMs, DRIFT consistently outperforms a strong set of regression- and generative-based solutions.
Community
DRIFT combines a base predictor, which provides a coarse estimate of the target output, with a generative refinement module based on flow matching that iteratively improves the prediction. This residual formulation transforms the generative modeling problem from learning a global output distribution to modeling a localized residual distribution around a strong prior, substantially simplifying optimization.
We evaluate DRIFT on both perception and planning tasks, including visual grounding and robotic control. Across multiple tasks and architectures spanning Multi-modal Large Language Models, Vision-Language-Action Models, and World Action Models, DRIFT consistently outperforms a strong set of regression- and generative-based solutions.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.05758 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.05758 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.05758 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.