<strong>Can a data agent do more than answer a question—can it navigate an entire workspace and deliver a result you can actually verify?</strong></p>\n<p>Real-world analytics rarely begins with a single clean table. The evidence is often scattered across databases, structured files, long documents, and videos—sometimes even across languages.</p>\n<p>We are excited to introduce <strong>DataSpace</strong>, a benchmark that challenges data agents to become true <em>workspace solvers</em>. Given only a natural-language question and a task-local heterogeneous workspace, an agent must autonomously discover the relevant evidence, connect information across formats, perform multi-step analysis, and return the <strong>complete, verifiable tabular result</strong>.</p>\n<p>📊 <strong>DataSpace at a glance:</strong></p>\n<ul>\n<li><strong>410 cross-language analytical tasks</strong></li>\n<li><strong>7,439 artifacts totaling 15.01 GB</strong></li>\n<li><strong>6 formats:</strong> CSV, JSON, SQLite, Markdown, PDF, and video</li>\n<li>Financial, macroeconomic, and healthcare scenarios</li>\n<li>Execution-grounded construction and review by <strong>11 domain experts</strong></li>\n<li>Deterministic, model-free evaluation of complete tabular results</li>\n</ul>\n<p>DataSpace also served as the official evaluation benchmark for the <strong>KDD Cup 2026: Data Agents for Complex Data Analysis</strong>, bringing the challenge to <strong>703 valid teams and 1,307 participants worldwide</strong>.</p>\n<p>Our evaluation across <strong>6 frontier multimodal models</strong> and <strong>5 widely used agent harnesses</strong> reveals how much remains unsolved. The strongest controlled setting reaches only <strong>66.34% Task Accuracy</strong>, while <strong>76 tasks are missed by every tested model</strong>. Even with the backbone fixed, changing the agent harness creates a striking <strong>15.36-point accuracy gap</strong>. Multimodal evidence integration and cross-source joins remain persistent bottlenecks.</p>\n<p>The message is clear: today’s data agents are promising, but reliable workspace-scale analytics is far from solved.</p>\n<p>We are releasing the benchmark, dataset, evaluator, baselines, and leaderboard to support the next generation of <strong>reliable and verifiable data agents</strong>. We warmly invite the community to test new models, harnesses, and agent designs—and push beyond today’s frontier. 🚀</p>\n<p>📄 <strong>Paper:</strong> <a href=\"https://arxiv.org/abs/2608.03451\" rel=\"nofollow\">https://arxiv.org/abs/2608.03451</a><br>🌐 <strong>Project &amp; KDD Cup:</strong> <a href=\"https://dataagent.top/\" rel=\"nofollow\">https://dataagent.top/</a><br>💻 <strong>Code &amp; Evaluator:</strong> <a href=\"https://github.com/HKUSTDial/DataSpace\" rel=\"nofollow\">https://github.com/HKUSTDial/DataSpace</a><br>🤗 <strong>Dataset:</strong> <a href=\"https://huggingface.co/datasets/HKUSTDial/DataSpace\">https://huggingface.co/datasets/HKUSTDial/DataSpace</a><br>🏆 <strong>Leaderboard:</strong> <a href=\"https://dataspace-bench.github.io/\" rel=\"nofollow\">https://dataspace-bench.github.io/</a></p>\n<p>Questions, feedback, results, and contributions are all highly welcome!</p>\n","updatedAt":"2026-08-07T12:25:37.523Z","author":{"_id":"643ab52ac5f633a7fa7f4238","avatarUrl":"/avatars/24596155f5c80df57fa868f6fd517822.svg","fullname":"Boyan Li","name":"BugMaker-Boyan","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8147358894348145},"editors":["BugMaker-Boyan"],"editorAvatarUrls":["/avatars/24596155f5c80df57fa868f6fd517822.svg"],"reactions":[{"reaction":"🚀","users":["derrickzhu"],"count":1}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.03451","authors":[{"_id":"6a72bf741a375f948521c4c6","user":{"_id":"643ab52ac5f633a7fa7f4238","avatarUrl":"/avatars/24596155f5c80df57fa868f6fd517822.svg","isPro":false,"fullname":"Boyan Li","user":"BugMaker-Boyan","type":"user","name":"BugMaker-Boyan"},"name":"Boyan Li","status":"claimed_verified","statusLastChangedAt":"2026-08-05T08:45:04.564Z","hidden":false},{"_id":"6a72bf741a375f948521c4c7","name":"Zhuowen Liang","hidden":false},{"_id":"6a72bf741a375f948521c4c8","name":"Yupeng Xie","hidden":false},{"_id":"6a72bf741a375f948521c4c9","name":"Xiaotian Lin","hidden":false},{"_id":"6a72bf741a375f948521c4ca","name":"Tianqi Luo","hidden":false},{"_id":"6a72bf741a375f948521c4cb","name":"Xinyu Liu","hidden":false},{"_id":"6a72bf741a375f948521c4cc","name":"Yizhang Zhu","hidden":false},{"_id":"6a72bf741a375f948521c4cd","name":"Zhangyang Peng","hidden":false},{"_id":"6a72bf741a375f948521c4ce","name":"Yuan Li","hidden":false},{"_id":"6a72bf741a375f948521c4cf","name":"Zhengxuan Zhang","hidden":false},{"_id":"6a72bf741a375f948521c4d0","name":"Jiayi Zhang","hidden":false},{"_id":"6a72bf741a375f948521c4d1","name":"Nan Tang","hidden":false},{"_id":"6a72bf741a375f948521c4d2","name":"Guoliang Li","hidden":false},{"_id":"6a72bf741a375f948521c4d3","name":"Yuyu Luo","hidden":false}],"publishedAt":"2026-08-04T00:00:00.000Z","submittedOnDailyAt":"2026-08-07T00:00:00.000Z","title":"DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces","submittedOnDailyBy":{"_id":"643ab52ac5f633a7fa7f4238","avatarUrl":"/avatars/24596155f5c80df57fa868f6fd517822.svg","isPro":false,"fullname":"Boyan Li","user":"BugMaker-Boyan","type":"user","name":"BugMaker-Boyan"},"summary":"Data agents enable natural-language analytics over organizational workspaces, where relevant evidence may be scattered across databases, structured files, long documents, and multimedia. Existing benchmarks largely isolate structured querying, retrieval, or open-ended analysis, leaving heterogeneous evidence discovery, complete tabular outputs, and deterministic evaluation insufficiently unified. We introduce DataSpace, a benchmark in which data agents produce verifiable tabular results from task-local heterogeneous workspaces. It contains 410 cross-language tasks and 7,439 artifacts totaling 15.01 GB across CSV, JSON, SQLite, Markdown, PDF, and video. DataSpace also served as the official evaluation benchmark for the KDD Cup 2026 Data Agents for Complex Data Analysis competition. Each agent receives only a question and workspace and returns the complete requested tabular result. We construct DataSpace with DataSpace-Builder, an execution-grounded framework comprising cross-language transformation, constraint-aware relational sampling, modality routing and artifact rendering, and human review and task repair by 11 domain experts. A deterministic evaluator performs header-invariant column alignment, type- and precision-aware normalization, and order-aware row comparison. Across six recently released frontier multimodal models and five widely used agent harnesses, the best accuracy reaches 66.34%, while harness choice creates a 15.36-point spread with the backbone fixed. Multimodal evidence integration and joins consistently reduce accuracy across all six backbones. These results show that DataSpace remains unsaturated and identify key challenges for improving data-agent reliability.","upvotes":19,"discussionId":"6a72bf741a375f948521c4d4","projectPage":"https://dataspace-bench.github.io","githubRepo":"https://github.com/HKUSTDial/DataSpace","githubRepoAddedBy":"user","githubStars":11,"organization":{"_id":"69b9da9e9e3e77cac79506d3","name":"HKUSTDial","fullname":"HKUSTDial","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/643ab52ac5f633a7fa7f4238/ZFOnRThWVc9G1OSN6Fz8r.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"643ab52ac5f633a7fa7f4238","avatarUrl":"/avatars/24596155f5c80df57fa868f6fd517822.svg","isPro":false,"fullname":"Boyan Li","user":"BugMaker-Boyan","type":"user"},{"_id":"683ebfd683a130f817d95ce8","avatarUrl":"/avatars/74e8453f24b2e803a604412a32def111.svg","isPro":false,"fullname":"xypkent","user":"xypkent","type":"user"},{"_id":"688dcfa2cd8913f87d6ef096","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/aPl_FTwI_fit4Ahtij29i.png","isPro":false,"fullname":"Zhangyang PENG","user":"August-GKD","type":"user"},{"_id":"65dd77bfcb021a4a9ebdc62f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65dd77bfcb021a4a9ebdc62f/o58j3T670xByIjJhnNLj9.png","isPro":false,"fullname":"Derrick ZHU","user":"derrickzhu","type":"user"},{"_id":"66a4afec0c86556c158aee69","avatarUrl":"/avatars/1037edea5c3459d324d71c3388d5967d.svg","isPro":false,"fullname":"Jose","user":"chx7514","type":"user"},{"_id":"66ea643899af9ac3463639b1","avatarUrl":"/avatars/252d470e761a57834dee3dbc60dfefed.svg","isPro":false,"fullname":"Disen Lan","user":"landisen","type":"user"},{"_id":"6387676c23da90491eb9fb16","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1669818175965-noauth.jpeg","isPro":true,"fullname":"Kai Li","user":"JusperLee","type":"user"},{"_id":"65f40e83653c231cbaf7defe","avatarUrl":"/avatars/afa5ce72324112739e539865c9aee26b.svg","isPro":false,"fullname":"Jiayi Zhang","user":"didiforhugface","type":"user"},{"_id":"67679b5cfeac1e9f62571cf9","avatarUrl":"/avatars/4b0a0348dd0bf871aa40f8ff37703efa.svg","isPro":false,"fullname":"Zhuowen Liang","user":"SetonLiang2","type":"user"},{"_id":"672a2f88fd7653daca87c285","avatarUrl":"/avatars/bd8026cddc9237a3d24bb69c867138ea.svg","isPro":false,"fullname":"Xiaotian LIN","user":"lxtlxt","type":"user"},{"_id":"65ffff82bda47e9bcf6373c5","avatarUrl":"/avatars/ea65e245bb7cc4549a5a97f1650a10cc.svg","isPro":false,"fullname":"LIU yu","user":"MeteorFeather","type":"user"},{"_id":"6a75da21ba5d8103db9d6180","avatarUrl":"/avatars/c96937f032222f1339cad572fd705cbb.svg","isPro":false,"fullname":"敖明澍","user":"MingshuAo","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"69b9da9e9e3e77cac79506d3","name":"HKUSTDial","fullname":"HKUSTDial","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/643ab52ac5f633a7fa7f4238/ZFOnRThWVc9G1OSN6Fz8r.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.03451.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2608.03451 in a model README.md to link it from this page.

Cite arxiv.org/abs/2608.03451 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.