<strong>TL;DR.</strong> Video-Oasis rethinks the current benchmark landscape by examining whether proliferating video benchmarks truly satisfy shared criteria for genuine video understanding.</p>\n","updatedAt":"2026-07-10T04:11:33.520Z","author":{"_id":"682c33e0d2dc6b34d4198042","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/IBo9E-6_ErEL9rYSiNkrb.png","fullname":"Geuntaek Lim","name":"geuntaek","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7084177732467651},"editors":["geuntaek"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/IBo9E-6_ErEL9rYSiNkrb.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2603.29616","authors":[{"_id":"6a473cda6ee372f6920de3b2","user":{"_id":"682c33e0d2dc6b34d4198042","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/IBo9E-6_ErEL9rYSiNkrb.png","isPro":false,"fullname":"Geuntaek Lim","user":"geuntaek","type":"user","name":"geuntaek"},"name":"Geuntaek Lim","status":"claimed_verified","statusLastChangedAt":"2026-07-05T21:09:59.432Z","hidden":false},{"_id":"6a473cda6ee372f6920de3b3","name":"Sungjune Park","hidden":false},{"_id":"6a473cda6ee372f6920de3b4","name":"Jaeyun Lee","hidden":false},{"_id":"6a473cda6ee372f6920de3b5","name":"Inwoong Lee","hidden":false},{"_id":"6a473cda6ee372f6920de3b6","name":"Taeoh Kim","hidden":false},{"_id":"6a473cda6ee372f6920de3b7","name":"Dongyoon Wee","hidden":false},{"_id":"6a473cda6ee372f6920de3b8","name":"Minho Shim","hidden":false},{"_id":"6a473cda6ee372f6920de3b9","name":"Yukyung Choi","hidden":false}],"publishedAt":"2026-07-02T00:00:00.000Z","submittedOnDailyAt":"2026-07-10T00:00:00.000Z","title":"Video-Oasis: Rethinking Evaluation of Video Understanding","submittedOnDailyBy":{"_id":"682c33e0d2dc6b34d4198042","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/IBo9E-6_ErEL9rYSiNkrb.png","isPro":false,"fullname":"Geuntaek Lim","user":"geuntaek","type":"user","name":"geuntaek"},"summary":"The inherent complexity of video understanding makes it difficult to determine whether Video-LLM benchmark performance stems from visual perception, linguistic reasoning, or knowledge priors. While many benchmarks have emerged to assess high-level reasoning, shared criteria for evaluating video understanding remain largely overlooked. Instead of introducing yet another benchmark, we take a step back to re-examine the criteria for evaluating video understanding. In this work, we introduce Video-Oasis, a sustainable diagnostic suite for systematically auditing existing video understanding benchmarks. This audit reveals that 55\\% of existing benchmark samples are solvable without visual input or temporal context. After filtering these shortcuts, the remaining video-native challenges expose a substantial capability gap: state-of-the-art models perform only marginally above random guessing. Building on these findings, we use the distilled challenges as a testbed to investigate which algorithmic design choices contribute to robust video understanding. We hope our work provides a practical foundation for constructing rigorous video benchmarks and evaluating future Video-LLMs. Code is available at https://github.com/sejong-rcv/Video-Oasis.","upvotes":43,"discussionId":"6a473cda6ee372f6920de3ba","projectPage":"https://limgeuntaekk.github.io/Video-Oasis/","githubRepo":"https://github.com/sejong-rcv/Video-Oasis","githubRepoAddedBy":"user","ai_summary":"Video-Oasis diagnostics reveal that half of existing video benchmarks can be solved without visual input, exposing significant capability gaps in current video understanding models.","ai_keywords":["Video-LLM","video understanding","benchmark evaluation","diagnostic suite","video-native challenges","algorithmic design choices","visual perception","linguistic reasoning","knowledge priors"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":21,"organization":{"_id":"6172b71e14c201e6e67e7eb1","name":"naver","fullname":"NAVER","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a3e2b51e54bee5f593f1a2f/IZeHJnuX0CIJFanqQpeV4.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"682c33e0d2dc6b34d4198042","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/IBo9E-6_ErEL9rYSiNkrb.png","isPro":false,"fullname":"Geuntaek Lim","user":"geuntaek","type":"user"},{"_id":"68d140b16aeac63ef5787947","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/09m86Xm5pa-KYrTGEkVuD.png","isPro":false,"fullname":"KangHeeseung","user":"kHeedong","type":"user"},{"_id":"64fb26df491fad09631dd03e","avatarUrl":"/avatars/e3fc6979e02563aae130762644d08b9f.svg","isPro":false,"fullname":"JungwooKim","user":"JungwooKim","type":"user"},{"_id":"67f37f8ccef233be93fbdf94","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/6yt9UsKuV835hH-Q5kTMB.png","isPro":false,"fullname":"Yujin Hwang","user":"jinwgo","type":"user"},{"_id":"652fcd7ce13118b6fbc9d60e","avatarUrl":"/avatars/5b508977cbb24fb1b2f399e864ec4119.svg","isPro":false,"fullname":"Taejoo Kim","user":"Arururu12","type":"user"},{"_id":"64ea23eb18d79efd5336505b","avatarUrl":"/avatars/2ac8562880f865e84c9485e524ceb840.svg","isPro":false,"fullname":"Geo Ahn","user":"ahnGeo","type":"user"},{"_id":"69fbfa3a43ea865d29951040","avatarUrl":"/avatars/15c648b5b78aa81a78a7896add1aec9b.svg","isPro":false,"fullname":"juyoung hong","user":"youngeeeee","type":"user"},{"_id":"64a0b18608f544767ce094c8","avatarUrl":"/avatars/e26c747697417f613c9eaaa089b8bbd3.svg","isPro":false,"fullname":"feasfeef","user":"feasfeef","type":"user"},{"_id":"64a614710c2ce52c64e9d7dd","avatarUrl":"/avatars/09d23a39906e708f67a23ed743f89158.svg","isPro":true,"fullname":"Dongheon Lee","user":"dslisleedh","type":"user"},{"_id":"69c1f3367f7dd6e7c1168c2e","avatarUrl":"/avatars/55addb4b04a634493629b7aefa21c62a.svg","isPro":false,"fullname":"Son","user":"woojinny","type":"user"},{"_id":"697b5938c46c4743ab515875","avatarUrl":"/avatars/c044baec85c433b071ca3fc077938e68.svg","isPro":false,"fullname":"Lee Jaeyun","user":"rcvjylee","type":"user"},{"_id":"64e44a0424809d7fa01527a9","avatarUrl":"/avatars/86f2f4678a24c8aa898026eb1ab95c28.svg","isPro":false,"fullname":"Inwoong Lee","user":"Inwoong","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":2,"organization":{"_id":"6172b71e14c201e6e67e7eb1","name":"naver","fullname":"NAVER","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a3e2b51e54bee5f593f1a2f/IZeHJnuX0CIJFanqQpeV4.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2603/2603.29616.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2603.29616 in a model README.md to link it from this page.

Cite arxiv.org/abs/2603.29616 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2603.29616 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.