Recent video generative models have greatly improved the realism of AI-generated videos, yet their outputs still exhibit artifacts such as temporal inconsistencies, structural distortions, and semantic incoherence. While Multimodal Large Language Models (MLLMs) show strong visual understanding capabilities, their ability to perceive and reason about such artifacts remains unclear. Existing benchmarks often lack systematic evaluation of artifact-aware perception and fine-grained diagnostic reasoning, especially across diverse AI-generated video domains beyond photorealistic content. To address this gap, we introduce Artifact-Bench, a comprehensive benchmark for evaluating MLLMs on AI-generated video artifact detection and analysis. We first establish a three-level hierarchical taxonomy of realism artifacts, covering photorealistic, animated, and CG-style videos. Based on this taxonomy, Artifact-Bench defines three complementary tasks: real vs. AI-generated video classification, pairwise realism comparison, and fine-grained artifact identification. Experiments on 19 leading MLLMs reveal substantial limitations in artifact perception and reasoning, with many models approaching random or even below-random performance in challenging settings. We further observe significant misalignment between MLLM judgments and human perceptual preferences, highlighting their limited reliability as general evaluators for AI-generated video realism.</p>\n","updatedAt":"2026-05-20T01:41:17.579Z","author":{"_id":"673c7319d11b1c2e246ead9c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/673c7319d11b1c2e246ead9c/IjFIO--N7Hm_BOEafhEQv.jpeg","fullname":"Yang Shi","name":"DogNeverSleep","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":11,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.894311249256134},"editors":["DogNeverSleep"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/673c7319d11b1c2e246ead9c/IjFIO--N7Hm_BOEafhEQv.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.18984","authors":[{"_id":"6a0d112365eb30f20d962b3c","name":"Yuqi Tang","hidden":false},{"_id":"6a0d112365eb30f20d962b3d","name":"Yang Shi","hidden":false},{"_id":"6a0d112365eb30f20d962b3e","name":"Zhuoran Zhang","hidden":false},{"_id":"6a0d112365eb30f20d962b3f","name":"Qixun Wang","hidden":false},{"_id":"6a0d112365eb30f20d962b40","name":"Xuehai Bai","hidden":false},{"_id":"6a0d112365eb30f20d962b41","name":"Yue Ding","hidden":false},{"_id":"6a0d112365eb30f20d962b42","name":"Ruizhe Chen","hidden":false},{"_id":"6a0d112365eb30f20d962b43","name":"Bohan Zeng","hidden":false},{"_id":"6a0d112365eb30f20d962b44","name":"Xinlong Chen","hidden":false},{"_id":"6a0d112365eb30f20d962b45","name":"Xuanyu Zhu","hidden":false},{"_id":"6a0d112365eb30f20d962b46","name":"Bozhou Li","hidden":false},{"_id":"6a0d112365eb30f20d962b47","name":"Yuran Wang","hidden":false},{"_id":"6a0d112365eb30f20d962b48","name":"Yifan Dai","hidden":false},{"_id":"6a0d112365eb30f20d962b49","name":"Chengzhuo Tong","hidden":false},{"_id":"6a0d112365eb30f20d962b4a","name":"Xinyu Liu","hidden":false},{"_id":"6a0d112365eb30f20d962b4b","name":"Yiyan Ji","hidden":false},{"_id":"6a0d112365eb30f20d962b4c","name":"Yujie Wei","hidden":false},{"_id":"6a0d112365eb30f20d962b4d","name":"Yuhao Dong","hidden":false},{"_id":"6a0d112365eb30f20d962b4e","name":"Shilin Yan","hidden":false},{"_id":"6a0d112365eb30f20d962b4f","name":"Fengxiang Wang","hidden":false},{"_id":"6a0d112365eb30f20d962b50","name":"Yi-Fan Zhang","hidden":false},{"_id":"6a0d112365eb30f20d962b51","name":"Haotian Wang","hidden":false},{"_id":"6a0d112365eb30f20d962b52","name":"Yuanxing Zhang","hidden":false},{"_id":"6a0d112365eb30f20d962b53","name":"Pengfei Wan","hidden":false}],"publishedAt":"2026-05-18T00:00:00.000Z","submittedOnDailyAt":"2026-05-20T00:00:00.000Z","title":"Artifact-Bench: Evaluating MLLMs on Detecting and Assessing the Artifacts of AI-Generated Videos","submittedOnDailyBy":{"_id":"673c7319d11b1c2e246ead9c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/673c7319d11b1c2e246ead9c/IjFIO--N7Hm_BOEafhEQv.jpeg","isPro":false,"fullname":"Yang Shi","user":"DogNeverSleep","type":"user","name":"DogNeverSleep"},"summary":"Recent video generative models have greatly improved the realism of AI-generated videos, yet their outputs still exhibit artifacts such as temporal inconsistencies, structural distortions, and semantic incoherence. While Multimodal Large Language Models (MLLMs) show strong visual understanding capabilities, their ability to perceive and reason about such artifacts remains unclear. Existing benchmarks often lack systematic evaluation of artifact-aware perception and fine-grained diagnostic reasoning, especially across diverse AI-generated video domains beyond photorealistic content. To address this gap, we introduce Artifact-Bench, a comprehensive benchmark for evaluating MLLMs on AI-generated video artifact detection and analysis. We first establish a three-level hierarchical taxonomy of realism artifacts, covering photorealistic, animated, and CG-style videos. Based on this taxonomy, Artifact-Bench defines three complementary tasks: real vs. AI-generated video classification, pairwise realism comparison, and fine-grained artifact identification. Experiments on 19 leading MLLMs reveal substantial limitations in artifact perception and reasoning, with many models approaching random or even below-random performance in challenging settings. We further observe significant misalignment between MLLM judgments and human perceptual preferences, highlighting their limited reliability as general evaluators for AI-generated video realism.","upvotes":21,"discussionId":"6a0d112365eb30f20d962b54","githubRepo":"https://github.com/FrankYang-17/Artifact-Bench","githubRepoAddedBy":"user","ai_summary":"Artifact-Bench evaluates multimodal large language models' capability to detect and analyze artifacts in AI-generated videos, revealing significant limitations in artifact perception and reasoning.","ai_keywords":["Multimodal Large Language Models","AI-generated video","artifact detection","fine-grained artifact identification","realism artifacts","visual understanding","diagnostic reasoning"],"githubStars":3},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"662f93942510ef5735d7ad00","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png","isPro":false,"fullname":"magicwpf","user":"magicwpf","type":"user"},{"_id":"673c7319d11b1c2e246ead9c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/673c7319d11b1c2e246ead9c/IjFIO--N7Hm_BOEafhEQv.jpeg","isPro":false,"fullname":"Yang Shi","user":"DogNeverSleep","type":"user"},{"_id":"644d2532d185572dd1e48f90","avatarUrl":"/avatars/5831acebb02d8bc8f80f56b7b11c7c69.svg","isPro":false,"fullname":"Zhu","user":"zzzhu","type":"user"},{"_id":"6700b2b6bff0e8b51d07fa00","avatarUrl":"/avatars/6cd7e243b7bc37ae9d308c175cbe6f05.svg","isPro":false,"fullname":"asdasd","user":"asdjghh","type":"user"},{"_id":"66a5f512c0c2ce1db1869144","avatarUrl":"/avatars/7f2c80a7fc587ef8e4a35bfe4c3de426.svg","isPro":false,"fullname":"bxh520","user":"bxh520","type":"user"},{"_id":"658a9872c4b2004663db8159","avatarUrl":"/avatars/a0bac905b3182e1abef32bf2a987d7fa.svg","isPro":false,"fullname":"crz","user":"RuizheChen","type":"user"},{"_id":"661e62c6bac5d981f886f77b","avatarUrl":"/avatars/f1eb51ed4499ca434c8939573dfbd5e2.svg","isPro":false,"fullname":"Bozhou Li","user":"zooblastlbz","type":"user"},{"_id":"66650d38b52f0890724f3b07","avatarUrl":"/avatars/c25a365bff4985ebb71c96dd097b804f.svg","isPro":false,"fullname":"Xinlong Chen","user":"XinlongChen","type":"user"},{"_id":"66100bacac50abb8d56dece6","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66100bacac50abb8d56dece6/fd-4VMpb_1nl903yAIK4K.jpeg","isPro":false,"fullname":"Ding Yue","user":"dingyue1011","type":"user"},{"_id":"637f70d6fab5db9101c3dfc8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/637f70d6fab5db9101c3dfc8/NgkYNXWLDavLbrnCby2Fl.jpeg","isPro":false,"fullname":"Yujie Wei","user":"weilllllls","type":"user"},{"_id":"674e77fa59a127e4eacf5dba","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/674e77fa59a127e4eacf5dba/W7qr94Buvvaio8zhKrEha.jpeg","isPro":false,"fullname":"Yifan Dai","user":"Moonwines","type":"user"},{"_id":"652965773a416e1f2173443b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg","isPro":true,"fullname":"Yuhao Dong","user":"THUdyh","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.18984.md"}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2605.18984 in a model README.md to link it from this page.

Cite arxiv.org/abs/2605.18984 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2605.18984 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.