We introduce <strong>HakushoBench</strong>, a challenging Japanese chart and table visual question answering (VQA) benchmark for evaluating vision-language models (VLMs). To construct HakushoBench, we leverage Japanese government white papers, which contain a large and diverse collection of chart and table images. The benchmark is built from <strong>33 white papers</strong>, covering <strong>more than 10 image types</strong> and comprising <strong>2,053 VQA instances</strong>. Experimental results reveal a substantial performance gap between open-weight and proprietary models, highlighting significant room for improvement in open-weight VLMs.</p>\n","updatedAt":"2026-06-02T02:54:04.378Z","author":{"_id":"630b39e8910e17bbfea8436d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/630b39e8910e17bbfea8436d/JvxwqDb4MuGKDXGxsmm2I.jpeg","fullname":"Issa Sugiura","name":"speed","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8478043675422668},"editors":["speed"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/630b39e8910e17bbfea8436d/JvxwqDb4MuGKDXGxsmm2I.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.01132","authors":[{"_id":"6a1e43b9808ddbc3c7d43c3d","name":"Issa Sugiura","hidden":false},{"_id":"6a1e43b9808ddbc3c7d43c3e","name":"Shuhei Kurita","hidden":false},{"_id":"6a1e43b9808ddbc3c7d43c3f","name":"Yusuke Oda","hidden":false},{"_id":"6a1e43b9808ddbc3c7d43c40","name":"Naoaki Okazaki","hidden":false}],"publishedAt":"2026-05-31T00:00:00.000Z","submittedOnDailyAt":"2026-06-02T00:00:00.000Z","title":"HakushoBench: A Japanese Chart and Table VQA Benchmark from Governmental White Papers","submittedOnDailyBy":{"_id":"630b39e8910e17bbfea8436d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/630b39e8910e17bbfea8436d/JvxwqDb4MuGKDXGxsmm2I.jpeg","isPro":false,"fullname":"Issa Sugiura","user":"speed","type":"user","name":"speed"},"summary":"Understanding chart and table images is essential for applying vision-language models (VLMs) to real-world document understanding. While English benchmarks have advanced rapidly, non-English counterparts remain scarce, leaving it unclear whether this progress generalizes across languages. A key obstacle is the difficulty of collecting realistic and diverse non-English chart and table images at scale. To address this, we leverage governmental white papers as a scalable source for benchmark construction beyond English, as they contain naturally occurring charts and tables across diverse formats and domains and are freely accessible in many countries. As a first instantiation, we introduce HakushoBench, a challenging Japanese chart and table VQA benchmark built from 33 governmental white papers. HakushoBench contains 2,053 images spanning over 10 image types, with manually annotated QA pairs, designed to assess deep and holistic understanding of charts and tables, rather than local visual cues alone. Experiments across a broad range of VLMs demonstrate that HakushoBench remains challenging for open-weight models: the best open-weight model achieves only 58.6% accuracy, and a 34.9-point gap between open-weight and proprietary models highlights substantial room for improvement in complex chart and table understanding. We release our dataset and code.","upvotes":3,"discussionId":"6a1e43ba808ddbc3c7d43c41","projectPage":"https://huggingface.co/datasets/llm-jp/HakushoBench","ai_summary":"Researchers created HakushoBench, a Japanese chart and table visual question answering benchmark derived from governmental documents, to evaluate vision-language models' ability to understand complex visual data beyond English-language datasets.","ai_keywords":["vision-language models","visual question answering","benchmark","governmental white papers","chart understanding","table understanding"],"organization":{"_id":"64925b450d0d95337114f17e","name":"llm-jp","fullname":"LLM-jp","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/649259b1b2bcb8358cf77958/WEl_gpGpAWQte_podstYD.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"69bce31a9237fd29e759f932","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/jwSo3-Ekpk6bpNA52UUDn.png","isPro":false,"fullname":"Haoran Zheng","user":"levi-harris5","type":"user"},{"_id":"630b39e8910e17bbfea8436d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/630b39e8910e17bbfea8436d/JvxwqDb4MuGKDXGxsmm2I.jpeg","isPro":false,"fullname":"Issa Sugiura","user":"speed","type":"user"},{"_id":"63171caf1cc81c5e95ed7b92","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63171caf1cc81c5e95ed7b92/29I5Lr0vLRcQR7AfCZcYj.jpeg","isPro":false,"fullname":"Akim Mousterou","user":"AkimfromParis","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"64925b450d0d95337114f17e","name":"llm-jp","fullname":"LLM-jp","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/649259b1b2bcb8358cf77958/WEl_gpGpAWQte_podstYD.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.01132.md"}">
HakushoBench: A Japanese Chart and Table VQA Benchmark from Governmental White Papers
Abstract
Researchers created HakushoBench, a Japanese chart and table visual question answering benchmark derived from governmental documents, to evaluate vision-language models' ability to understand complex visual data beyond English-language datasets.
AI-generated summary
Understanding chart and table images is essential for applying vision-language models (VLMs) to real-world document understanding. While English benchmarks have advanced rapidly, non-English counterparts remain scarce, leaving it unclear whether this progress generalizes across languages. A key obstacle is the difficulty of collecting realistic and diverse non-English chart and table images at scale. To address this, we leverage governmental white papers as a scalable source for benchmark construction beyond English, as they contain naturally occurring charts and tables across diverse formats and domains and are freely accessible in many countries. As a first instantiation, we introduce HakushoBench, a challenging Japanese chart and table VQA benchmark built from 33 governmental white papers. HakushoBench contains 2,053 images spanning over 10 image types, with manually annotated QA pairs, designed to assess deep and holistic understanding of charts and tables, rather than local visual cues alone. Experiments across a broad range of VLMs demonstrate that HakushoBench remains challenging for open-weight models: the best open-weight model achieves only 58.6% accuracy, and a 34.9-point gap between open-weight and proprietary models highlights substantial room for improvement in complex chart and table understanding. We release our dataset and code.
Community
We introduce HakushoBench, a challenging Japanese chart and table visual question answering (VQA) benchmark for evaluating vision-language models (VLMs). To construct HakushoBench, we leverage Japanese government white papers, which contain a large and diverse collection of chart and table images. The benchmark is built from 33 white papers, covering more than 10 image types and comprising 2,053 VQA instances. Experimental results reveal a substantial performance gap between open-weight and proprietary models, highlighting significant room for improvement in open-weight VLMs.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.01132 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.01132 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.