<video src=\"https://cdn-uploads.huggingface.co/production/uploads/667b8de7a68bf81afe668afe/NhK4K9aGlvH-YUC_3CfpU.mp4\" controls=\"\" class=\"max-w-full!\"></video></p>\n","updatedAt":"2026-07-21T07:08:54.607Z","author":{"_id":"667b8de7a68bf81afe668afe","avatarUrl":"/avatars/aeff10805ff858332e6f6a58735dbbd9.svg","fullname":"leoli","name":"lifuguan","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.2650795876979828},"editors":["lifuguan"],"editorAvatarUrls":["/avatars/aeff10805ff858332e6f6a58735dbbd9.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.16401","authors":[{"_id":"6a5ed3fc4fe5d1d13e84aacb","name":"Runmao Yao","hidden":false},{"_id":"6a5ed3fc4fe5d1d13e84aacc","name":"Kairui Hu","hidden":false},{"_id":"6a5ed3fc4fe5d1d13e84aacd","name":"Yukang Cao","hidden":false},{"_id":"6a5ed3fc4fe5d1d13e84aace","name":"Ruisi Wang","hidden":false},{"_id":"6a5ed3fc4fe5d1d13e84aacf","name":"Shulin Tian","hidden":false},{"_id":"6a5ed3fc4fe5d1d13e84aad0","name":"Ziang Cao","hidden":false},{"_id":"6a5ed3fc4fe5d1d13e84aad1","name":"Weichen Fan","hidden":false},{"_id":"6a5ed3fc4fe5d1d13e84aad2","name":"Ziqi Huang","hidden":false},{"_id":"6a5ed3fc4fe5d1d13e84aad3","name":"Yuhao Dong","hidden":false},{"_id":"6a5ed3fc4fe5d1d13e84aad4","name":"Hao Li","hidden":false},{"_id":"6a5ed3fc4fe5d1d13e84aad5","name":"Zhaoxi Chen","hidden":false},{"_id":"6a5ed3fc4fe5d1d13e84aad6","name":"Zhongang Cai","hidden":false},{"_id":"6a5ed3fc4fe5d1d13e84aad7","name":"Lei Yang","hidden":false},{"_id":"6a5ed3fc4fe5d1d13e84aad8","name":"Ziwei Liu","hidden":false}],"publishedAt":"2026-07-17T00:00:00.000Z","submittedOnDailyAt":"2026-07-21T00:00:00.000Z","title":"Apple-π: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence","submittedOnDailyBy":{"_id":"667b8de7a68bf81afe668afe","avatarUrl":"/avatars/aeff10805ff858332e6f6a58735dbbd9.svg","isPro":false,"fullname":"leoli","user":"lifuguan","type":"user","name":"lifuguan"},"summary":"Modern video generation models are increasingly hailed as emerging world models with an internalized grasp of physical law. Yet existing benchmarks largely evaluate physical plausibility only at the output level, without verifying whether the model arrives there through a faithful, law-grounded reasoning process. We introduce Apple-PI, the first benchmark that anchors video-model evaluation explicitly in physical laws. Apple-PI comprises three components. 1) Orchard: a dataset of 400 videos covering ten canonical tasks in classical mechanics. It separates single-law tasks for confounder-free diagnosis from multi-law tasks for probing generalization. 2) Benchmark Protocol: a three-stage protocol based on scientific reasoning, including Perception, Formulation, and Deduction. It uses chain-of-frames prompting on infographic-annotated first frames, treating the generated video as the model's visible reasoning trace. 3) Evaluation Suite: a hybrid evaluation suite that combines MLLM-based subjective scoring with physics-law-grounded objective measures. This enables stage-resolved diagnosis of not only whether a model fails, but where it fails. Benchmarking 11 models shows that current video models remain far from reliable law-grounded world simulators, with the best video model scoring only 0.473. Our stage-, pillar-, and source-resolved analyses further expose a Perception-to-Formulation-to-Deduction bottleneck, weak multi-law state transfer, and a persistent Sim-to-Real gap. These findings position Apple-PI as a diagnostic foundation for guiding future video models toward world models with law-grounded physical intelligence.","upvotes":25,"discussionId":"6a5ed3fc4fe5d1d13e84aad9","projectPage":"https://21yrm.github.io/Apple-PI-homepage","githubRepo":"https://github.com/21yrm/Apple-PI","githubRepoAddedBy":"user","githubStars":10,"organization":{"_id":"62d55f243bf5e059f7ca25ba","name":"mmlab-ntu","fullname":"MMLab@NTU","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1658151991971-62b5777f593a2c49da69dc02.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"66aa94cbd59743aa4a65646f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66aa94cbd59743aa4a65646f/Z6BKgT_oYcfJgGHQNm9K3.png","isPro":false,"fullname":"Runmao Yao","user":"yaorunmao","type":"user"},{"_id":"65f3ee92c2d214f8848d687c","avatarUrl":"/avatars/10c7eac57273e8dcb4b2a08fb00270f6.svg","isPro":false,"fullname":"Juntong Fang","user":"hellexf","type":"user"},{"_id":"65fd418fbe8494c6be4f55d4","avatarUrl":"/avatars/3c4a2c45e0d714dd726b923a74582fd1.svg","isPro":false,"fullname":"Jiawei Guo","user":"Kaku666","type":"user"},{"_id":"67ee368b0f6ee9ad85bce4b7","avatarUrl":"/avatars/3f2e3003470ded76f5fd4f9e071890f3.svg","isPro":false,"fullname":"Eric Ward","user":"Eric-21","type":"user"},{"_id":"6573e182c13a2c747809f49d","avatarUrl":"/avatars/0c76b8800e425d141e7f2afb060e4912.svg","isPro":false,"fullname":"Lucas","user":"hydrogen03","type":"user"},{"_id":"66ba98c899e022a2572e9799","avatarUrl":"/avatars/4e7edcc7e00400daec0f54f0e2c61d46.svg","isPro":false,"fullname":"Runsheng Bai","user":"Brssss","type":"user"},{"_id":"661b894a9fbab39d0e4dda40","avatarUrl":"/avatars/7d28e358249f30a388cf705516201ef1.svg","isPro":false,"fullname":"Hex-WIllAppear","user":"WillAppear","type":"user"},{"_id":"6970748d8761801ce94260ba","avatarUrl":"/avatars/b70b0d17fa9a28ee378e1c707f21bd46.svg","isPro":false,"fullname":"Haiheng","user":"HaihengLiu","type":"user"},{"_id":"68e4a33ef26746d2511f2ce4","avatarUrl":"/avatars/1f8c79b9400cb661becf537620888575.svg","isPro":false,"fullname":"Haowen Wang","user":"haowenw","type":"user"},{"_id":"6517f727f7724d9aee55b67c","avatarUrl":"/avatars/18049305b41d103c599367144d96594d.svg","isPro":false,"fullname":"Zhangguowei","user":"MalcomF1","type":"user"},{"_id":"62ab1ac1d48b4d8b048a3473","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png","isPro":false,"fullname":"Ziwei Liu","user":"liuziwei7","type":"user"},{"_id":"69fae2f570585ad496262741","avatarUrl":"/avatars/f4c0b4a39cf28f35ad1b030248ee6fdc.svg","isPro":false,"fullname":"AnonymousSubmitt","user":"AnonymousSubmitt","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"62d55f243bf5e059f7ca25ba","name":"mmlab-ntu","fullname":"MMLab@NTU","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1658151991971-62b5777f593a2c49da69dc02.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.16401.md","query":{}}">
Apple-π: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence
Published on Jul 17
· Submitted by leoli on Jul 21 Abstract
Modern video generation models are increasingly hailed as emerging world models with an internalized grasp of physical law. Yet existing benchmarks largely evaluate physical plausibility only at the output level, without verifying whether the model arrives there through a faithful, law-grounded reasoning process. We introduce Apple-PI, the first benchmark that anchors video-model evaluation explicitly in physical laws. Apple-PI comprises three components. 1) Orchard: a dataset of 400 videos covering ten canonical tasks in classical mechanics. It separates single-law tasks for confounder-free diagnosis from multi-law tasks for probing generalization. 2) Benchmark Protocol: a three-stage protocol based on scientific reasoning, including Perception, Formulation, and Deduction. It uses chain-of-frames prompting on infographic-annotated first frames, treating the generated video as the model's visible reasoning trace. 3) Evaluation Suite: a hybrid evaluation suite that combines MLLM-based subjective scoring with physics-law-grounded objective measures. This enables stage-resolved diagnosis of not only whether a model fails, but where it fails. Benchmarking 11 models shows that current video models remain far from reliable law-grounded world simulators, with the best video model scoring only 0.473. Our stage-, pillar-, and source-resolved analyses further expose a Perception-to-Formulation-to-Deduction bottleneck, weak multi-law state transfer, and a persistent Sim-to-Real gap. These findings position Apple-PI as a diagnostic foundation for guiding future video models toward world models with law-grounded physical intelligence.
Community
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2607.16401 in a model README.md to link it from this page.
Cite arxiv.org/abs/2607.16401 in a dataset README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.