How do vision models learn to understand text directly from pixels? How can models pretrained on synthetic text rendered at ≤224×224 resolution generalize all the way to real-world 4K documents at test time?</p>\n<p>Pixel Linguist II answers these questions through controlled studies of pixel-text representation learning, identifying four key ingredients: spatial proxies for resolution generalization, multimodal grounding with natural images, layout-aware rendering to avoid visual shortcuts, and a two-stage multilingual curriculum. Building on these findings, Pixel Linguist II achieves state-of-the-art results on English, cross-lingual, and multilingual Visual STS and ViDoRe (Visual Document Retrieval), while also improving downstream MLLM performance by 2.75% on average over the Qwen2.5-ViT. Pixel Linguist II further demonstrates strong optical context compression capability: its representations remain robust under up to 80% visual-token compression, and with only 40% of visual tokens retained, it still outperforms the full-budget Qwen2.5-ViT on MLLM downstream tasks.</p>\n","updatedAt":"2026-09-03T02:34:43.751Z","author":{"_id":"63108cc834c7d77420b0fd68","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63108cc834c7d77420b0fd68/taDnqEmcI9Rhe3uzcPEE3.jpeg","fullname":"Chenghao Xiao","name":"gowitheflow","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":28,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8337590098381042},"editors":["gowitheflow"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/63108cc834c7d77420b0fd68/taDnqEmcI9Rhe3uzcPEE3.jpeg"],"reactions":[{"reaction":"🔥","users":["kenchan0226"],"count":1}],"isReport":false}},{"id":"6a98dda2291909ea5261dbc0","author":{"_id":"604f67ef0fe8ff3ec13d71ef","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/604f67ef0fe8ff3ec13d71ef/KhUwWvZ3OJ9nEee3B-SXO.png","fullname":"Hou Pong (Ken) Chan","name":"kenchan0226","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":17,"isUserFollowing":false},"createdAt":"2026-09-03T02:38:26.000Z","type":"comment","data":{"edited":true,"hidden":false,"latest":{"raw":"\n","html":"<p><a href=\"https://cdn-uploads.huggingface.co/production/uploads/604f67ef0fe8ff3ec13d71ef/f8A0Yzr19aybCxZ3cKiWb.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/604f67ef0fe8ff3ec13d71ef/f8A0Yzr19aybCxZ3cKiWb.png\" alt=\"image\"></a></p>\n","updatedAt":"2026-09-03T02:47:40.491Z","author":{"_id":"604f67ef0fe8ff3ec13d71ef","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/604f67ef0fe8ff3ec13d71ef/KhUwWvZ3OJ9nEee3B-SXO.png","fullname":"Hou Pong (Ken) Chan","name":"kenchan0226","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":17,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.4239020347595215},"editors":["kenchan0226"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/604f67ef0fe8ff3ec13d71ef/KhUwWvZ3OJ9nEee3B-SXO.png"],"reactions":[{"reaction":"🔥","users":["gowitheflow"],"count":1}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2609.01147","authors":[{"_id":"6a98d334fea818274321fe3f","user":{"_id":"6628f9de341f589246164776","avatarUrl":"/avatars/b8b8665cce7b3aea23bf74156a8869d7.svg","isPro":false,"fullname":"Chaohao Yuan","user":"ychaohao","type":"user","name":"ychaohao"},"name":"Chaohao Yuan","status":"claimed_verified","statusLastChangedAt":"2026-09-03T08:45:04.364Z","hidden":false},{"_id":"6a98d334fea818274321fe40","name":"Ruifeng Yuan","hidden":false},{"_id":"6a98d334fea818274321fe41","name":"Zhuoxu Huang","hidden":false},{"_id":"6a98d334fea818274321fe42","name":"Yu Rong","hidden":false},{"_id":"6a98d334fea818274321fe43","name":"Hong Cheng","hidden":false},{"_id":"6a98d334fea818274321fe44","user":{"_id":"604f67ef0fe8ff3ec13d71ef","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/604f67ef0fe8ff3ec13d71ef/KhUwWvZ3OJ9nEee3B-SXO.png","isPro":false,"fullname":"Hou Pong (Ken) Chan","user":"kenchan0226","type":"user","name":"kenchan0226"},"name":"Hou Pong Chan","status":"claimed_verified","statusLastChangedAt":"2026-09-03T08:17:08.711Z","hidden":false},{"_id":"6a98d334fea818274321fe45","user":{"_id":"63108cc834c7d77420b0fd68","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63108cc834c7d77420b0fd68/taDnqEmcI9Rhe3uzcPEE3.jpeg","isPro":false,"fullname":"Chenghao Xiao","user":"gowitheflow","type":"user","name":"gowitheflow"},"name":"Chenghao Xiao","status":"claimed_verified","statusLastChangedAt":"2026-09-03T08:17:07.011Z","hidden":false}],"publishedAt":"2026-09-01T00:00:00.000Z","submittedOnDailyAt":"2026-09-03T00:00:00.000Z","title":"On the Design Fundamentals of Pixel Text Representation Learning","submittedOnDailyBy":{"_id":"63108cc834c7d77420b0fd68","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63108cc834c7d77420b0fd68/taDnqEmcI9Rhe3uzcPEE3.jpeg","isPro":false,"fullname":"Chenghao Xiao","user":"gowitheflow","type":"user","name":"gowitheflow"},"summary":"Text-rich visual inputs require models that can read, retrieve, and compress language directly in pixel space, yet existing pixel-text encoders struggle with fixed resolution pretraining, visual shortcut learning, weak visual grounding, and multilingual visual text understanding. In this work, we investigate the fundamental design principles required for robust visual text representation learning. Through systematic controlled ablations, we identify four critical components: variable image resolutions and rendered font sizes provide spatial proxies for high-resolution document generalization; natural image-text pairs are indispensable for grounding and prevent text-only collapse; layout-aware rendering helps prevent pixel-level shortcuts; and a two-stage multilingual curriculum enables effective cross-lingual alignment. By integrating these principles into a scalable training recipe, we train Pixel Linguist II, a native-resolution vision encoder trained with on-the-fly rendering, unified contrastive grounding, and a multilingual curriculum over 280M training examples. Pixel Linguist II sets new state-of-the-art results on English, cross-lingual, and multilingual Visual STS and ViDoRe, while also enabling better MLLM downstream evaluation. Notably, Pixel Linguist II remains robust under 80\\% visual token compression, showing great promise for optical context compression. Our code and resources are available at https://github.com/Pixel-Linguist/Pixel-Linguist-II.","upvotes":25,"discussionId":"6a98d335fea818274321fe46","projectPage":"https://huggingface.co/collections/Pixel-Linguist/pixel-linguist-ii","ai_summary":"Pixel Linguist II improves visual text encoding through variable resolution training, natural image-text grounding, layout-aware rendering, and multilingual curricula, achieving state-of-the-art results and strong compression robustness.","ai_keywords":["visual text representation learning","variable image resolutions","layout-aware rendering","two-stage multilingual curriculum","contrastive grounding","visual token compression","Pixel Linguist II"],"ai_summary_model":"thinkingmachines/Inkling-Small","organization":{"_id":"68eb82481dfe9ec7bd1cc16b","name":"LCO-Embedding","fullname":"LCO-Embedding","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/63108cc834c7d77420b0fd68/kOt7TD2Ge6hgUsz8CF6Ro.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"63108cc834c7d77420b0fd68","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63108cc834c7d77420b0fd68/taDnqEmcI9Rhe3uzcPEE3.jpeg","isPro":false,"fullname":"Chenghao Xiao","user":"gowitheflow","type":"user"},{"_id":"604f67ef0fe8ff3ec13d71ef","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/604f67ef0fe8ff3ec13d71ef/KhUwWvZ3OJ9nEee3B-SXO.png","isPro":false,"fullname":"Hou Pong (Ken) Chan","user":"kenchan0226","type":"user"},{"_id":"6628f9de341f589246164776","avatarUrl":"/avatars/b8b8665cce7b3aea23bf74156a8869d7.svg","isPro":false,"fullname":"Chaohao Yuan","user":"ychaohao","type":"user"},{"_id":"60f313f4adf471cbdf8bb66a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60f313f4adf471cbdf8bb66a/wurLk6JH8Duo17isf34pq.png","isPro":false,"fullname":"Yang Wang","user":"yangwang825","type":"user"},{"_id":"66fa54b9076c1a309d563a41","avatarUrl":"/avatars/2daaa10c4c5bfcde74c7f995d15be1e0.svg","isPro":false,"fullname":"Ruifeng Yuan","user":"csyrf","type":"user"},{"_id":"66fb03d6b505f1a04c39d935","avatarUrl":"/avatars/e9b830c460ec02037758c9b3469bb8ad.svg","isPro":false,"fullname":"Xuanlang Dai","user":"XuanlangDai","type":"user"},{"_id":"6814533922645cf8cb16a56a","avatarUrl":"/avatars/6a9f113a7be60b9ba63663fb9d58b867.svg","isPro":false,"fullname":"Li","user":"Tonylmz","type":"user"},{"_id":"69ccc72354b48932315c34a2","avatarUrl":"/avatars/1fcf956e86bcd4038da8a939aa331f25.svg","isPro":false,"fullname":"Dong Zihan","user":"yangmuyu6","type":"user"},{"_id":"6a98e97003edf75fa01322f9","avatarUrl":"/avatars/113680183349bbdce3a0a6569eeefa89.svg","isPro":false,"fullname":"jiangzhiyou","user":"jzy-sc30","type":"user"},{"_id":"68410e4e5de370509548f0ed","avatarUrl":"/avatars/ec53b49b3de495003a0d05e137586326.svg","isPro":false,"fullname":"Jiacheng Cen","user":"Chewxq","type":"user"},{"_id":"6a98eb85d0145c3ec3ef6f03","avatarUrl":"/avatars/2b4607a96a8e4c43468e57d2938bfae0.svg","isPro":false,"fullname":"Jiachen Hu","user":"NickHclos","type":"user"},{"_id":"67d92d6061f369297a4a225a","avatarUrl":"/avatars/af71c847d8122c827b22ce52d4c5af71.svg","isPro":false,"fullname":"Songyou Li","user":"Wthinker","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"68eb82481dfe9ec7bd1cc16b","name":"LCO-Embedding","fullname":"LCO-Embedding","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/63108cc834c7d77420b0fd68/kOt7TD2Ge6hgUsz8CF6Ro.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2609/2609.01147.md","query":{}}">
On the Design Fundamentals of Pixel Text Representation Learning
Abstract
Pixel Linguist II improves visual text encoding through variable resolution training, natural image-text grounding, layout-aware rendering, and multilingual curricula, achieving state-of-the-art results and strong compression robustness.
Text-rich visual inputs require models that can read, retrieve, and compress language directly in pixel space, yet existing pixel-text encoders struggle with fixed resolution pretraining, visual shortcut learning, weak visual grounding, and multilingual visual text understanding. In this work, we investigate the fundamental design principles required for robust visual text representation learning. Through systematic controlled ablations, we identify four critical components: variable image resolutions and rendered font sizes provide spatial proxies for high-resolution document generalization; natural image-text pairs are indispensable for grounding and prevent text-only collapse; layout-aware rendering helps prevent pixel-level shortcuts; and a two-stage multilingual curriculum enables effective cross-lingual alignment. By integrating these principles into a scalable training recipe, we train Pixel Linguist II, a native-resolution vision encoder trained with on-the-fly rendering, unified contrastive grounding, and a multilingual curriculum over 280M training examples. Pixel Linguist II sets new state-of-the-art results on English, cross-lingual, and multilingual Visual STS and ViDoRe, while also enabling better MLLM downstream evaluation. Notably, Pixel Linguist II remains robust under 80\% visual token compression, showing great promise for optical context compression. Our code and resources are available at https://github.com/Pixel-Linguist/Pixel-Linguist-II.
Community
How do vision models learn to understand text directly from pixels? How can models pretrained on synthetic text rendered at ≤224×224 resolution generalize all the way to real-world 4K documents at test time?
Pixel Linguist II answers these questions through controlled studies of pixel-text representation learning, identifying four key ingredients: spatial proxies for resolution generalization, multimodal grounding with natural images, layout-aware rendering to avoid visual shortcuts, and a two-stage multilingual curriculum. Building on these findings, Pixel Linguist II achieves state-of-the-art results on English, cross-lingual, and multilingual Visual STS and ViDoRe (Visual Document Retrieval), while also improving downstream MLLM performance by 2.75% on average over the Qwen2.5-ViT. Pixel Linguist II further demonstrates strong optical context compression capability: its representations remain robust under up to 80% visual-token compression, and with only 40% of visual tokens retained, it still outperforms the full-budget Qwen2.5-ViT on MLLM downstream tasks.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2609.01147 in a model README.md to link it from this page.
Cite arxiv.org/abs/2609.01147 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2609.01147 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.