🚀 PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models</p>\n<p>We introduce PerceptionDLM, a multimodal diffusion VLM built for efficient parallel region perception. Instead of describing image regions one-by-one like autoregressive captioners, it exploits the parallel decoding nature of dLLMs: given an image and multiple region masks, it generates descriptions for all regions in a single denoising pass — avoiding the linear latency growth of AR region captioners.</p>\n<p>Highlights:</p>\n<p>🏆 Strong open diffusion VLM baseline. PerceptionDLM-Base outperforms LLaDA-V on 15/16 multimodal benchmarks and stays competitive with leading AR VLMs (Qwen2.5-VL, InternVL3) at the same scale.<br>🧩 Parallel region captioning for a much better accuracy–efficiency trade-off on multi-region dense captioning.<br>📊 New benchmark — ParaDLC-Bench, jointly evaluating caption quality and inference efficiency.<br>🔁 Fully open: code, model weights, training data, and the full evaluation suite.<br>📄 Paper: <a href=\"https://arxiv.org/abs/2606.19534\" rel=\"nofollow\">https://arxiv.org/abs/2606.19534</a><br>💻 Code: <a href=\"https://github.com/MSALab-PKU/PerceptionDLM\" rel=\"nofollow\">https://github.com/MSALab-PKU/PerceptionDLM</a><br>🤗 Models & Data: <a href=\"https://huggingface.co/collections/MSALab/perceptiondlm-model-zoo\">https://huggingface.co/collections/MSALab/perceptiondlm-model-zoo</a></p>\n<p>Feedback and discussion welcome! 🙌</p>\n","updatedAt":"2026-06-22T02:21:57.380Z","author":{"_id":"67344c08ba8aa259654bd819","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67344c08ba8aa259654bd819/OzFuQ9ebdOlX8hCq0Yrm1.jpeg","fullname":"Yueyi Sun","name":"bitersun","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.760411262512207},"editors":["bitersun"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/67344c08ba8aa259654bd819/OzFuQ9ebdOlX8hCq0Yrm1.jpeg"],"reactions":[{"reaction":"🔥","users":["taesiri"],"count":1}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.19534","authors":[{"_id":"6a34bcb94c5c5e0d69bf1ccc","user":{"_id":"67344c08ba8aa259654bd819","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67344c08ba8aa259654bd819/OzFuQ9ebdOlX8hCq0Yrm1.jpeg","isPro":false,"fullname":"Yueyi Sun","user":"bitersun","type":"user","name":"bitersun"},"name":"Yueyi Sun","status":"claimed_verified","statusLastChangedAt":"2026-06-19T14:19:34.909Z","hidden":false},{"_id":"6a34bcb94c5c5e0d69bf1ccd","name":"Yuhao Wang","hidden":false},{"_id":"6a34bcb94c5c5e0d69bf1cce","name":"Jason Li","hidden":false},{"_id":"6a34bcb94c5c5e0d69bf1ccf","name":"Ye Tian","hidden":false},{"_id":"6a34bcb94c5c5e0d69bf1cd0","name":"Tao Zhang","hidden":false},{"_id":"6a34bcb94c5c5e0d69bf1cd1","name":"Jacky Mai","hidden":false},{"_id":"6a34bcb94c5c5e0d69bf1cd2","name":"Yihan Wang","hidden":false},{"_id":"6a34bcb94c5c5e0d69bf1cd3","name":"Haochen Wang","hidden":false},{"_id":"6a34bcb94c5c5e0d69bf1cd4","name":"Jinbin Bai","hidden":false},{"_id":"6a34bcb94c5c5e0d69bf1cd5","name":"Ling Yang","hidden":false},{"_id":"6a34bcb94c5c5e0d69bf1cd6","name":"Yunhai Tong","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/67344c08ba8aa259654bd819/TwOAWZhIBUUnAMG2aJdPS.mp4","https://cdn-uploads.huggingface.co/production/uploads/67344c08ba8aa259654bd819/BTPbkQGG4USDOSQ56hDO4.mp4"],"publishedAt":"2026-06-17T00:00:00.000Z","submittedOnDailyAt":"2026-06-22T00:00:00.000Z","title":"PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models","submittedOnDailyBy":{"_id":"67344c08ba8aa259654bd819","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67344c08ba8aa259654bd819/OzFuQ9ebdOlX8hCq0Yrm1.jpeg","isPro":false,"fullname":"Yueyi Sun","user":"bitersun","type":"user","name":"bitersun"},"summary":"Multimodal large language models (MLLMs) have achieved remarkable progress in visual understanding tasks. However, most existing MLLMs rely on autoregressive generation, which limits their efficiency for perception tasks that require captioning multiple regions. In this work, we propose PerceptionDLM, a multimodal diffusion language model optimized for efficient parallel region perception. Built upon PerceptionDLM-Base, a strong foundational baseline that achieves state-of-the-art performance among open-source diffusion MLLMs, our architecture fully leverages the parallel decoding nature of DLMs. Specifically, we introduce efficient prompting and structured attention masking to enable simultaneous perception of multiple masked regions, allowing the model to generate region descriptions in parallel at both the sequence and token levels. This design significantly improves inference efficiency compared with existing approaches that process regions sequentially. To systematically evaluate the parallelism property of visual perception capability for DLMs, we construct a new Parallel Detailed Localized Captioning Benchmark (ParaDLC-Bench) by scaling the DLC-Bench to include multiple region masks per image, enabling joint evaluation of both caption quality and inference efficiency. Experiments demonstrate that PerceptionDLM maintains competitive performance in region captioning while achieving substantial speed improvements for multi-region perception tasks. Our results highlight the potential of multimodal diffusion language models for efficient, parallel visual perception. To the best of our knowledge, we are the first to achieve parallel region caption and perception by leveraging the advantages of diffusion language models. Code, models, and datasets are released.","upvotes":40,"discussionId":"6a34bcba4c5c5e0d69bf1cd7","projectPage":"https://msalab-pku.github.io/projects/PerceptionDLM/index.html","githubRepo":"https://github.com/MSALab-PKU/PerceptionDLM","githubRepoAddedBy":"user","ai_summary":"PerceptionDLM enables efficient parallel region perception in multimodal diffusion language models through structured attention masking and efficient prompting, achieving faster inference without sacrificing caption quality.","ai_keywords":["multimodal large language models","diffusion language models","parallel decoding","structured attention masking","region captioning","visual perception","inference efficiency","PerceptionDLM","PerceptionDLM-Base"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":24,"organization":{"_id":"653b817d32c97d0655575872","name":"ByteDance","fullname":"ByteDance","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6535c9e88bde2fae19b6fb25/0clr54wj5Ly-RkYU9OXPp.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6a1e4becfaa0b23a50071f9f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a1e4becfaa0b23a50071f9f/UuG_Sq92u68IufbxZ_7_s.png","isPro":false,"fullname":"MSALab@PKU","user":"MSALab","type":"user"},{"_id":"67344c08ba8aa259654bd819","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67344c08ba8aa259654bd819/OzFuQ9ebdOlX8hCq0Yrm1.jpeg","isPro":false,"fullname":"Yueyi Sun","user":"bitersun","type":"user"},{"_id":"6902c0711cbf187eb72ff30b","avatarUrl":"/avatars/bc35ca66abf84b08fb3ab43fe201605c.svg","isPro":true,"fullname":"MLLM","user":"Anran-MLLM","type":"user"},{"_id":"6662a23c2f86097c6d828b96","avatarUrl":"/avatars/2aa31ab30874257529861f2e4024acc2.svg","isPro":false,"fullname":"liu","user":"miao6","type":"user"},{"_id":"660d6d02e4990549abb4a2e3","avatarUrl":"/avatars/5bb5ac03f3160fc8997a5b64f6c3868b.svg","isPro":false,"fullname":"Harry Jhou","user":"myourr","type":"user"},{"_id":"63958b4414513eaf9029ebf1","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png","isPro":false,"fullname":"Xiangtai Li","user":"LXT","type":"user"},{"_id":"65a28e129acab19980226731","avatarUrl":"/avatars/abc3828f807efc4e03837b0eae063f98.svg","isPro":false,"fullname":"Jiahao Meng","user":"marinero4972","type":"user"},{"_id":"65bce64b8467e2a3d6a450af","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65bce64b8467e2a3d6a450af/IFjoy4GYMA6oUgYyJfZ1F.jpeg","isPro":false,"fullname":"Chao Tang","user":"Tangc03","type":"user"},{"_id":"656724074f6ec72017754d33","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zLIgt4xvfbYPLZh0E3WWF.png","isPro":false,"fullname":"QingyuShi","user":"QingyuShi","type":"user"},{"_id":"63fccdac93b993a4ebd7789a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63fccdac93b993a4ebd7789a/KRx8vpdoDjsZBRw0j8Vg8.jpeg","isPro":false,"fullname":"Jinbin Bai","user":"BryanW","type":"user"},{"_id":"65025370b6595dc45c397340","avatarUrl":"/avatars/9469599b176034548042922c0afa7051.svg","isPro":false,"fullname":"J C","user":"dark-pen","type":"user"},{"_id":"6463554dd2044cd1d7c6e0bf","avatarUrl":"/avatars/d7653623117268c545a7063fec69664b.svg","isPro":false,"fullname":"Bingzheng Wei","user":"Bingzheng","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":1,"organization":{"_id":"653b817d32c97d0655575872","name":"ByteDance","fullname":"ByteDance","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6535c9e88bde2fae19b6fb25/0clr54wj5Ly-RkYU9OXPp.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.19534.md","query":{}}">
PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models
Authors: ,
,
,
,
,
,
,
,
,
Abstract
PerceptionDLM enables efficient parallel region perception in multimodal diffusion language models through structured attention masking and efficient prompting, achieving faster inference without sacrificing caption quality.
Multimodal large language models (MLLMs) have achieved remarkable progress in visual understanding tasks. However, most existing MLLMs rely on autoregressive generation, which limits their efficiency for perception tasks that require captioning multiple regions. In this work, we propose PerceptionDLM, a multimodal diffusion language model optimized for efficient parallel region perception. Built upon PerceptionDLM-Base, a strong foundational baseline that achieves state-of-the-art performance among open-source diffusion MLLMs, our architecture fully leverages the parallel decoding nature of DLMs. Specifically, we introduce efficient prompting and structured attention masking to enable simultaneous perception of multiple masked regions, allowing the model to generate region descriptions in parallel at both the sequence and token levels. This design significantly improves inference efficiency compared with existing approaches that process regions sequentially. To systematically evaluate the parallelism property of visual perception capability for DLMs, we construct a new Parallel Detailed Localized Captioning Benchmark (ParaDLC-Bench) by scaling the DLC-Bench to include multiple region masks per image, enabling joint evaluation of both caption quality and inference efficiency. Experiments demonstrate that PerceptionDLM maintains competitive performance in region captioning while achieving substantial speed improvements for multi-region perception tasks. Our results highlight the potential of multimodal diffusion language models for efficient, parallel visual perception. To the best of our knowledge, we are the first to achieve parallel region caption and perception by leveraging the advantages of diffusion language models. Code, models, and datasets are released.
Community
This comment has been hidden (marked as Resolved) This comment has been hidden This comment has been hidden (marked as Resolved) 🚀 PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models
We introduce PerceptionDLM, a multimodal diffusion VLM built for efficient parallel region perception. Instead of describing image regions one-by-one like autoregressive captioners, it exploits the parallel decoding nature of dLLMs: given an image and multiple region masks, it generates descriptions for all regions in a single denoising pass — avoiding the linear latency growth of AR region captioners.
Highlights:
🏆 Strong open diffusion VLM baseline. PerceptionDLM-Base outperforms LLaDA-V on 15/16 multimodal benchmarks and stays competitive with leading AR VLMs (Qwen2.5-VL, InternVL3) at the same scale.
🧩 Parallel region captioning for a much better accuracy–efficiency trade-off on multi-region dense captioning.
📊 New benchmark — ParaDLC-Bench, jointly evaluating caption quality and inference efficiency.
🔁 Fully open: code, model weights, training data, and the full evaluation suite.
📄 Paper: https://arxiv.org/abs/2606.19534
💻 Code: https://github.com/MSALab-PKU/PerceptionDLM
🤗 Models & Data: https://huggingface.co/collections/MSALab/perceptiondlm-model-zoo
Feedback and discussion welcome! 🙌
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.19534 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.