<a href=\"https://huggingface.co/datasets/Junjie-Ye/SPIEval\">https://huggingface.co/datasets/Junjie-Ye/SPIEval</a></p>\n","updatedAt":"2026-08-12T02:57:18.721Z","author":{"_id":"66384be673c2c55f2ded89fa","avatarUrl":"/avatars/1d8721074f0f51fab405f81474f2035f.svg","fullname":"Junjie Ye","name":"Junjie-Ye","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":6,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.5615724325180054},"editors":["Junjie-Ye"],"editorAvatarUrls":["/avatars/1d8721074f0f51fab405f81474f2035f.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.10692","authors":[{"_id":"6a7be00d1653ef87c6af1bff","user":{"_id":"66384be673c2c55f2ded89fa","avatarUrl":"/avatars/1d8721074f0f51fab405f81474f2035f.svg","isPro":false,"fullname":"Junjie Ye","user":"Junjie-Ye","type":"user","name":"Junjie-Ye"},"name":"Junjie Ye","status":"claimed_verified","statusLastChangedAt":"2026-08-12T08:45:05.067Z","hidden":false},{"_id":"6a7be00d1653ef87c6af1c00","name":"Zhuohui Sheng","hidden":false},{"_id":"6a7be00d1653ef87c6af1c01","name":"Shaofan Liu","hidden":false},{"_id":"6a7be00d1653ef87c6af1c02","name":"Yulun Zhu","hidden":false},{"_id":"6a7be00d1653ef87c6af1c03","name":"Wenjie Fu","hidden":false},{"_id":"6a7be00d1653ef87c6af1c04","name":"Dingwei Zhu","hidden":false},{"_id":"6a7be00d1653ef87c6af1c05","name":"Ming Zhang","hidden":false},{"_id":"6a7be00d1653ef87c6af1c06","name":"Yujiong Shen","hidden":false},{"_id":"6a7be00d1653ef87c6af1c07","name":"Weichao Wang","hidden":false},{"_id":"6a7be00d1653ef87c6af1c08","name":"Xin Zhao","hidden":false},{"_id":"6a7be00d1653ef87c6af1c09","name":"Shihan Dou","hidden":false},{"_id":"6a7be00d1653ef87c6af1c0a","name":"Tao Gui","hidden":false},{"_id":"6a7be00d1653ef87c6af1c0b","name":"Qi Zhang","hidden":false},{"_id":"6a7be00d1653ef87c6af1c0c","name":"Xuanjing Huang","hidden":false},{"_id":"6a7be00d1653ef87c6af1c0d","name":"Pluto Zhou","hidden":false}],"publishedAt":"2026-08-11T00:00:00.000Z","submittedOnDailyAt":"2026-08-12T00:00:00.000Z","title":"SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information","submittedOnDailyBy":{"_id":"66384be673c2c55f2ded89fa","avatarUrl":"/avatars/1d8721074f0f51fab405f81474f2035f.svg","isPro":false,"fullname":"Junjie Ye","user":"Junjie-Ye","type":"user","name":"Junjie-Ye"},"summary":"Large language models (LLMs) are increasingly deployed as mobile assistants, where a key challenge is leveraging personal information scattered across multiple applications (apps) to complete user instructions. However, due to the lack of dedicated benchmarks, their capabilities remain poorly understood. To address this gap, we introduce SPIEval, a human-curated benchmark grounded in five cognitive capabilities (i.e., reasoning, disambiguation, integration, preference inference, and multi-intent decomposition). SPIEval comprises 250 tasks spanning 4,335 personal records distributed across 10 apps and supports multi-turn interaction through 21 tools. Analysis shows that the benchmark exhibits diverse scenarios, challenging tasks, scattered information, controllable environments, and verifiable outcomes. We evaluate nine representative LLMs and find substantial room for improvement. The best-performing model, GPT-5.5 (xhigh), achieves only 57.3% accuracy, while the weakest achieves just 16.4%. Further analysis reveals that 79% of failures stem from inaccurate information localization, as LLMs often commit to plausible but incorrect information instead of continuing retrieval for verification. We also find that fewer than 2% of retrieval actions employ advanced search methods and observe substantial variation in search efficiency across models. These findings expose fundamental limitations of current LLM-based mobile assistants and motivate future research in this direction. Data and code are available at https://huggingface.co/datasets/Junjie-Ye/SPIEval.","upvotes":5,"discussionId":"6a7be00d1653ef87c6af1c0e","ai_summary":"SPIEval benchmarks mobile assistant LLMs on scattered personal data tasks, revealing major gaps in information retrieval and verification.","ai_keywords":["large language models","mobile assistants","SPIEval","reasoning","disambiguation","integration","preference inference","multi-intent decomposition","information localization","retrieval","search efficiency"],"ai_summary_model":"thinkingmachines/Inkling-Small"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"66384be673c2c55f2ded89fa","avatarUrl":"/avatars/1d8721074f0f51fab405f81474f2035f.svg","isPro":false,"fullname":"Junjie Ye","user":"Junjie-Ye","type":"user"},{"_id":"69bcdbc39044732b67dde7b6","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/q7hG0SuIjxC5jR3CC08va.png","isPro":false,"fullname":"Sofia Gonzalez","user":"thohall2023","type":"user"},{"_id":"64b0a5037a475fba70a7260d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64b0a5037a475fba70a7260d/MauBbb6raMA23yrR1Zq21.jpeg","isPro":false,"fullname":"Zhen Fang","user":"CostaliyA","type":"user"},{"_id":"6a5d9bcef38757cb4aaff991","avatarUrl":"/avatars/849eed88386f78e0640f14ff75496bc7.svg","isPro":false,"fullname":"Julio Pastor","user":"sarabarnes802","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.10692.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2608.10692 in a model README.md to link it from this page.

Cite arxiv.org/abs/2608.10692 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.