We propose Omni-DuplexEval, a benchmark for systematically evaluating real-time duplex interaction. And we further introduce an automatic evaluation framework based on LLM-as-a-Judge.</p>\n<p>Resource：</p>\n<ul>\n<li>Paper: <a href=\"https://arxiv.org/pdf/2605.17360\" rel=\"nofollow\">https://arxiv.org/pdf/2605.17360</a></li>\n<li>Github Project: <a href=\"https://github.com/OpenBMB/Omni-DuplexEval\" rel=\"nofollow\">https://github.com/OpenBMB/Omni-DuplexEval</a></li>\n<li>Data: <a href=\"https://huggingface.co/datasets/Hothan/Omni-DuplexEval\">https://huggingface.co/datasets/Hothan/Omni-DuplexEval</a></li>\n</ul>\n","updatedAt":"2026-05-20T06:03:57.364Z","author":{"_id":"64ca3bec2e592905f9c66385","avatarUrl":"/avatars/02f7e38072e37d4a882c22dc6aac139c.svg","fullname":"Chaoqun He","name":"Hothan","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.6468110084533691},"editors":["Hothan"],"editorAvatarUrls":["/avatars/02f7e38072e37d4a882c22dc6aac139c.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.17360","authors":[{"_id":"6a0d4cba65eb30f20d962d64","name":"Chaoqun He","hidden":false},{"_id":"6a0d4cba65eb30f20d962d65","name":"Mingyang Xiang","hidden":false},{"_id":"6a0d4cba65eb30f20d962d66","name":"Yingjing Xu","hidden":false},{"_id":"6a0d4cba65eb30f20d962d67","name":"Bokai Xu","hidden":false},{"_id":"6a0d4cba65eb30f20d962d68","name":"Junbo Cui","hidden":false},{"_id":"6a0d4cba65eb30f20d962d69","name":"Jie Zhou","hidden":false},{"_id":"6a0d4cba65eb30f20d962d6a","name":"Yuan Yao","hidden":false},{"_id":"6a0d4cba65eb30f20d962d6b","name":"Lijie Wen","hidden":false}],"publishedAt":"2026-05-17T00:00:00.000Z","submittedOnDailyAt":"2026-05-20T00:00:00.000Z","title":"Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interaction","submittedOnDailyBy":{"_id":"64ca3bec2e592905f9c66385","avatarUrl":"/avatars/02f7e38072e37d4a882c22dc6aac139c.svg","isPro":false,"fullname":"Chaoqun He","user":"Hothan","type":"user","name":"Hothan"},"summary":"Real-time duplex interaction is essential for multimodal AI systems operating in real-world scenarios, where models must continuously process streaming inputs and respond at appropriate moments. However, most existing multimodal large language models (MLLMs) are evaluated in offline settings, where the entire video input is processed before any response is generated. While recent work has started to explore real-time duplex MLLMs, there is still no comprehensive benchmark or automatic evaluation method for this setting. To address this gap, we propose Omni-DuplexEval, a benchmark for systematically evaluating real-time duplex interaction. The benchmark consists of two complementary scenarios: (1) Real-Time Description, which evaluates the ability to generate continuous, time-aligned responses that track evolving multimodal inputs, and (2) Proactive Reminder, which evaluates the ability to identify salient events and respond at appropriate moments. Omni-DuplexEval contains 660 videos with fine-grained, human-annotated labels and precise temporal metadata, spanning 9 tasks grounded in real-world scenarios, where all questions are formulated as open-ended queries. We further introduce an automatic evaluation framework based on LLM-as-a-Judge, which enables systematic assessment by jointly evaluating response-content alignment and response timing through timestamp-aware and sequential reasoning, achieving strong alignment with human judgments. Experiments on state-of-the-art duplex MLLMs reveal substantial limitations. The best-performing model achieves only 39.6% overall, while scoring only 20.0% on Proactive Reminder. Our analysis identifies two key challenges: models struggle to balance timely responses with coherent, holistic content generation, and they often fail to determine both when to respond and what to produce. We hope our work facilitates further progress in MLLMs.","upvotes":1,"discussionId":"6a0d4cbb65eb30f20d962d6c","githubRepo":"https://github.com/OpenBMB/Omni-DuplexEval","githubRepoAddedBy":"user","ai_summary":"A benchmark and evaluation framework for real-time duplex interaction in multimodal large language models, assessing continuous response generation and proactive event detection in streaming scenarios.","ai_keywords":["multimodal large language models","real-time duplex interaction","benchmark","automatic evaluation","LLM-as-a-Judge","response-content alignment","response timing","timestamp-aware reasoning","sequential reasoning"],"githubStars":2},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64ca3bec2e592905f9c66385","avatarUrl":"/avatars/02f7e38072e37d4a882c22dc6aac139c.svg","isPro":false,"fullname":"Chaoqun He","user":"Hothan","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.17360.md"}">
Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interaction
Abstract
A benchmark and evaluation framework for real-time duplex interaction in multimodal large language models, assessing continuous response generation and proactive event detection in streaming scenarios.
AI-generated summary
Real-time duplex interaction is essential for multimodal AI systems operating in real-world scenarios, where models must continuously process streaming inputs and respond at appropriate moments. However, most existing multimodal large language models (MLLMs) are evaluated in offline settings, where the entire video input is processed before any response is generated. While recent work has started to explore real-time duplex MLLMs, there is still no comprehensive benchmark or automatic evaluation method for this setting. To address this gap, we propose Omni-DuplexEval, a benchmark for systematically evaluating real-time duplex interaction. The benchmark consists of two complementary scenarios: (1) Real-Time Description, which evaluates the ability to generate continuous, time-aligned responses that track evolving multimodal inputs, and (2) Proactive Reminder, which evaluates the ability to identify salient events and respond at appropriate moments. Omni-DuplexEval contains 660 videos with fine-grained, human-annotated labels and precise temporal metadata, spanning 9 tasks grounded in real-world scenarios, where all questions are formulated as open-ended queries. We further introduce an automatic evaluation framework based on LLM-as-a-Judge, which enables systematic assessment by jointly evaluating response-content alignment and response timing through timestamp-aware and sequential reasoning, achieving strong alignment with human judgments. Experiments on state-of-the-art duplex MLLMs reveal substantial limitations. The best-performing model achieves only 39.6% overall, while scoring only 20.0% on Proactive Reminder. Our analysis identifies two key challenges: models struggle to balance timely responses with coherent, holistic content generation, and they often fail to determine both when to respond and what to produce. We hope our work facilitates further progress in MLLMs.
Community
We propose Omni-DuplexEval, a benchmark for systematically evaluating real-time duplex interaction. And we further introduce an automatic evaluation framework based on LLM-as-a-Judge.
Resource：
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.17360 in a model README.md to link it from this page.
Cite arxiv.org/abs/2605.17360 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.